Method of processing image and elecronic apparatus performing the same

Abstract

The present disclosure relates to a technology for an electronic device to process images. According to one embodiment of the present disclosure, as a camera module located below a display module is driven, the electronic device identifies the state of target pixels included in an area of ​​a display module corresponding to the camera module, and when the target pixels are turned on, acquires display information of the target pixels, determines offset information of pixels included in a captured image acquired from the camera module based on the display information of the target pixels, and corrects the captured image based on the determined offset information.

Description

Technology Field

[0001] The embodiments disclosed in this document relate to a method for processing images and an electronic device for performing the same. Background Technology

[0002] Manufacturers, who had been steadily increasing the size of smartphones, began researching ways to reduce bezel thickness as a method to increase screen size while maintaining the device dimensions. One such method developed was the notch design, which minimizes the area of ​​the top bezel excluding the display. However, because the front camera area could not be removed from the top bezel, it was difficult to utilize the entire front of the top bezel as a display area.

[0003] To address this, under-display camera (UDC) technology, which involves installing the camera at the bottom of the display, is being actively researched. When UDC technology is applied, the entire top of the smartphone consists of the display area, which not only enhances the design's completeness but also provides an environment where users can be more immersed in the information displayed. However, due to the structure in which the camera is positioned at the bottom of the display, video captured using UDC may experience various optical problems. The problem to be solved

[0004] The disclosed embodiments aim to provide an image processing technique for correcting images captured using an under-display camera (UDC). means of solving the problem

[0005] A method for processing an image in an electronic device according to one embodiment may include: identifying the state of target pixels included in an area of ​​a display module corresponding to a camera module as a camera module located below a display module in the electronic device is driven; obtaining display information of the target pixels when the target pixels are turned on; determining offset information of pixels included in a captured image obtained from a camera module based on the display information of the target pixels; and correcting the captured image based on the determined offset information.

[0006] An electronic device for processing an image according to one embodiment includes a memory for storing one or more instructions; a display module; a camera module located below the display module; and at least one processor for executing one or more instructions stored in the memory. The at least one processor identifies the state of target pixels included in an area of ​​the display module corresponding to the camera module as the camera module located below the display module is driven, and when the target pixels are turned on, acquires display information of the target pixels, determines offset information of pixels included in a captured image acquired from the camera module based on the display information of the target pixels, and corrects the captured image based on the determined offset information.

[0007] A computer program product comprising a recording medium storing a program that enables an electronic device according to one embodiment to perform a method of processing an image can perform the following operations as a camera module located below a display module in the electronic device is driven: identifying the state of target pixels included in an area of ​​a display module corresponding to the camera module; acquiring display information of target pixels when the target pixels are turned on; determining offset information of pixels included in a captured image acquired from a camera module based on the display information of the target pixels; and correcting the captured image based on the determined offset information. Effects of the invention

[0008] An electronic device according to one embodiment can more effectively improve the quality of a captured image by correcting a captured image obtained from a camera module by taking into account the light generated from target pixels of a display module. Brief explanation of the drawing

[0009] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments. FIG. 2 is a block diagram illustrating a camera module according to various embodiments. FIG. 3 is a drawing for explaining a display module and a camera module according to one embodiment. FIG. 4 is a drawing illustrating the screen of an electronic device in which a camera module is located at the bottom of a display module according to one embodiment. FIG. 5 is a flowchart illustrating a method for an electronic device to correct a captured image according to one embodiment. FIG. 6 is a diagram illustrating an offset image used for correction of a captured image according to one embodiment. FIG. 7 is a diagram illustrating an offset image used for correction of a captured image according to another embodiment. FIG. 8 is a flowchart illustrating a method for correcting a captured image when flicker occurs in the captured image using an electronic device according to one embodiment. FIG. 9 is a diagram illustrating a method for correcting a captured image when flicker occurs in an electronic device according to one embodiment. FIG. 10 is a diagram illustrating a method for correcting a captured image when flicker occurs in an electronic device including a plurality of camera modules according to one embodiment. FIG. 11 is a diagram illustrating a method for an electronic device according to one embodiment to correct a captured image using an artificial intelligence model. FIG. 12 is a diagram illustrating a method for an electronic device according to another embodiment to correct a captured image using an artificial intelligence model. FIG. 13 is a block diagram of an electronic device according to one embodiment. Specific details for implementing the invention

[0010] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments.

[0011] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network).

[0012] According to one embodiment, the electronic device (101) can communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0013] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0014] The auxiliary processor (123) can control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state.

[0015] According to one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as part of other functionally related components (e.g., a camera module (180) or a communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model.

[0016] An artificial intelligence model can be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers. The artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, or a combination of two or more of the above, but is not limited to the examples described above. Artificial intelligence models may include software structures, either additionally or as a substitute, in addition to hardware structures.

[0017] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0018] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0019] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0020] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0021] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0022] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0023] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0024] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0025] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0026] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0027] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0028] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0029] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0030] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0031] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0032] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197). According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0033] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0034] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0035] FIG. 2 is a block diagram illustrating a camera module according to various embodiments.

[0036] Referring to FIG. 2, the camera module (180) may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a memory (250) (e.g., a buffer memory), or an image signal processor (260). The lens assembly (210) may collect light emitted from a subject that is the target of image capture. The lens assembly (210) may include one or more lenses. According to one embodiment, the camera module (180) may include a plurality of lens assemblies (210). In this case, the camera module (180) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties different from the lens properties of other lens assemblies. The lens assemblies (210) may include, for example, a wide-angle lens or a telephoto lens.

[0037] The flash (220) may emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash (220) may include one or more light-emitting diodes (e.g., RGB (red-green-blue) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp.

[0038] The image sensor (230) can acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly (210) into an electrical signal. According to one embodiment, the image sensor (230) may include, for example, one image sensor selected from image sensors with different attributes such as an RGB sensor, a BW (black and white) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same attribute, or a plurality of image sensors having different attributes. Each image sensor included in the image sensor (230) may be implemented using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.

[0039] The image stabilizer (240) may move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction or control the operational characteristics of the image sensor (230) (e.g., adjusting read-out timing, etc.) in response to the movement of the camera module (180) or the electronic device (101) containing it. This allows for compensating for at least some of the negative effects caused by the movement on the image being captured. According to one embodiment, the image stabilizer (240) may detect such movement of the camera module (180) or the electronic device (101) using a gyroscope sensor (not shown) or an accelerometer sensor (not shown) placed inside or outside the camera module (180). According to one embodiment, the image stabilizer (240) may be implemented as, for example, an optical image stabilizer.

[0040] The memory (250) may temporarily store at least a portion of the image acquired through the image sensor (230) for the next image processing operation. For example, if image acquisition by the shutter is delayed or multiple images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in the memory (250), and the corresponding copy image (e.g., a low-resolution image) can be previewed through the display module (160). Subsequently, when a specified condition is satisfied (e.g., user input or system command), at least a portion of the original image stored in the memory (250) may be acquired and processed by, for example, an image signal processor (260). According to one embodiment, the memory (250) may be configured as at least a portion of the memory (130) or as a separate memory that operates independently thereof.

[0041] The image signal processor (260) can perform one or more image processing operations on an image obtained through the image sensor (230) or an image stored in memory (250). One or more of the above image processing methods may include, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softing). Additionally or generally, the image signal processor (260) may perform control (e.g., exposure time control, or readout timing control, etc.) over at least one of the components included in the camera module (180) (e.g., image sensor (230)). The image processed by the image signal processor (260) may be stored back in memory (250) for further processing or provided to an external component of the camera module (180) (e.g., memory (130), display module (160), electronic device (102), electronic device (104), or server (108)). According to one embodiment, the image signal processor (260) is at least part of the processor (120). It may be configured as a separate processor that operates independently of the processor (120). If the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display module (160) as is or after additional image processing by the processor (120).

[0042] According to one embodiment, the electronic device (101) may include a plurality of camera modules (180) each having different attributes or functions. In this case, for example, at least one of the plurality of camera modules (180) may be a wide-angle camera and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera modules (180) may be a front camera and at least another may be a rear camera.

[0043] FIG. 3 is a drawing for explaining a display module and a camera module according to one embodiment.

[0044] Referring to FIG. 3, the display module (160) may include a plurality of glass panels (310, 360), an organic layer (320), a pixel layer (330), a wiring layer (340), and a mask layer (350). However, not all of the illustrated components are essential components. The display module (160) may be implemented with more components than those illustrated, or with fewer components.

[0045] In the display module (160), an organic layer (320), a pixel layer (330), a wiring layer (340), and a mask layer (350) may be laminated between the first glass panel (310) and the second glass panel (360).

[0046] The organic layer (320) may include a light-emitting material, and when electricity is applied to the organic layer (320), light may be generated by electroluminescence. Additionally, the organic layer (320) may be composed of several auxiliary layers to assist in the flow of electrons and holes that generate electroluminescence.

[0047] The pixel layer (330) may include multiple pixels, and each of the multiple pixels may be composed of subpixels included in RGB (red green blue) channels. Additionally, the pixel layer (330) may be divided into an A1 region and an A2 region depending on whether pixels are removed. In the A1 region, some pixels may be removed for image capture by the camera module (180). Accordingly, the A1 region may contain both a portion where pixels are removed (332) and a portion where pixels are not removed (334). The A2 region represents an area where pixels are not removed.

[0048] The wiring layer (340) may include wirings for supplying power to the display module (160), and the wirings may connect the power and at least one component included in the display module (160).

[0049] The mask layer (350) serves to block light transmission and may be located between the wiring layer (340) and the second glass panel (360). In the mask layer (350), the mask may be removed in at least a portion of the VA area corresponding to the camera's field of view. For example, in the VA area, the mask may be removed in the area corresponding to the portion where the pixels of the aforementioned pixel layer (330) have been removed.

[0050] A camera module (180) may be located at the bottom of the display module (160). The camera module (180) may include one or more lenses (e.g., 370). However, this is merely an example, and the camera module (180) may include various components other than the lens (370), as described above in FIG. 2. Additionally, in this disclosure, the camera module (180) may be referred to as an under-display camera (UDC).

[0051] The camera module (180) can capture an image by detecting light transmitted through the portion of the display module (160) from which pixels and masks have been removed. Meanwhile, if the camera module (180) is located at the bottom of the display module (160), light generated from the display module (160) may affect the captured image. Accordingly, in an electronic device according to one embodiment, the captured image obtained by the camera module (180) can be corrected by taking into account the light generated from the display module (160). Other embodiments of correcting the captured image in the electronic device will be described in more detail later with reference to FIGS. 4 to 13.

[0052] FIG. 4 is a drawing illustrating the screen of an electronic device in which a camera module is located at the bottom of a display module according to one embodiment.

[0053] When a camera module is located at the bottom of a display module, light emitted from pixels within the display module can affect the captured image. However, if some pixels within the display module are left turned off to prevent this, there is a problem in that the screen is not displayed properly.

[0054] For example, referring to FIG. 4, when the pixel at the position corresponding to the camera module is turned off (420a), it can be seen that part of the screen (410) is lost. On the other hand, when the pixel at the position corresponding to the camera module is turned on (420b), the screen (410) can be displayed completely.

[0055] According to one embodiment, an electronic device (400) can determine the state of a pixel of a display module corresponding to a camera module based on at least one preset criterion when capturing an image using a camera module. The at least one preset criterion may include user input, the surrounding environment of the electronic device (400) when using the camera module, display setting information, or shooting setting information. For example, when the electronic device (400) receives user input requesting that the pixel of the display module corresponding to the camera module be kept in a turned-on state, the pixel may be kept in a turned-on state. According to another example, the electronic device (400) may keep the pixel of the display module corresponding to the camera module in a turned-on state when the brightness of the surrounding environment of the electronic device (400) is greater than or equal to a preset value. According to yet another example, when high-resolution or high-frame-rate content is displayed on the display module, the electronic device (400) may keep the pixel of the display module corresponding to the camera module in a turned-on state. According to another example, the electronic device (400) may keep the pixels of the display module corresponding to the camera module in an turned-on state when the shooting mode is set to portrait shooting mode. However, this is merely an example, and the criteria for determining the state of the pixels of the display module corresponding to the camera module in the electronic device (400) are not limited to the example described above.

[0056] An electronic device (400) according to one embodiment provides a technique for correcting a captured image obtained with the pixels of a display module turned on, in order to display the screen (410) completely while preventing degradation of the quality of the captured image. Meanwhile, if the electronic device (400) determines, based on the above-mentioned at least one criterion, that the pixels of the display module corresponding to the camera module should be kept in an off state, it may not perform the technique for correcting the captured image according to one embodiment.

[0057] An electronic device (400) according to one embodiment can obtain display information of target pixels located in an area corresponding to the camera module in a display module when a shooting mode in which the camera module is driven is selected. The area corresponding to the camera module in the display module may be determined based on the angle of view of the lens included in the camera module, but this is merely an example, and the area corresponding to the camera module may be changed according to the settings. Hereinafter, pixels located in an area corresponding to the camera module will be described as target pixels. The electronic device (400) can obtain at least one of the pixel values ​​by color of the target pixels, the degree of light emitted by the display elements of the target pixels, or the color correction matrix of the target pixels as display information of the target pixels. However, this is merely an example, and the display information of the target pixels is not limited to the example described above.

[0058] The electronic device (400) can determine offset information of pixels included in a captured image obtained from a camera module based on display information of target pixels. Offset information is information used to remove light elements of target pixels reflected in the captured image, and can be determined based on at least one of the pixel values ​​by color of the target pixels, the degree of light emitted by the display elements of the target pixels, or the color correction matrix of the target pixels. The electronic device (400) can correct the captured image based on the determined offset information. For example, the electronic device (400) can correct the captured image by subtracting the determined offset information from the pixel values ​​constituting the captured image.

[0059] FIG. 5 is a flowchart illustrating a method for an electronic device to correct a captured image according to one embodiment.

[0060] In step 510, the electronic device can identify the state of target pixels included in the area of ​​the display module corresponding to the camera module as the camera module located below the display module is driven.

[0061] The electronic device may include at least one camera module. At least some of the at least one camera module included in the electronic device may be located below the display module. Hereinafter, the camera module located below the display module among the at least one camera module included in the electronic device will be described.

[0062] The camera module can be driven as a user selects a shooting mode on the electronic device. For example, a shooting mode may be selected when the execution of a camera application is requested on the electronic device. In another example, a shooting mode may be selected when the video recording function is executed in an application other than a camera application.

[0063] The electronic device can identify the state of target pixels included in the area of ​​the display module corresponding to the camera module as the camera module is driven. The area of ​​the display module corresponding to the camera module may be determined based on whether light generated from pixels included in the display module affects the image captured by the camera. For example, the area of ​​the display module corresponding to the camera module may be determined based on the angle of view of the lens included in the camera module. In the present disclosure, pixels of the display module that affect the image captured by the camera are described as target pixels.

[0064] An electronic device can identify whether target pixels are turned on or off. For example, the electronic device can identify whether target pixels are turned on or off by checking whether power is supplied to each of the target pixels. According to another embodiment, the electronic device can manage target pixels in blocks. A block refers to a group of a predetermined number of target pixels, and the number of target pixels included in each block may be set differently. When target pixels are managed in blocks, the electronic device can determine whether the target pixels included in a block are turned on or off by checking whether power is supplied to only one target pixel belonging to each block.

[0065] In step 520, the electronic device can obtain display information of the target pixels displayed on the display module when the target pixels are turned on.

[0066] In the display information of target pixels according to one embodiment, at least one of the pixel values ​​by color of the target pixels, the degree of light emission by the display elements of the target pixels, or the color correction matrix of the target pixels may be obtained. However, this is merely an example, and the display information of the target pixels is not limited to the example described above.

[0067] In step 530, the electronic device can determine offset information of pixels included in a captured image obtained from a camera module based on display information of target pixels.

[0068] An electronic device according to one embodiment can determine offset information by combining display information of target pixels based on preset weights. For example, if the electronic device obtains pixel values ​​of target pixels by color using display information, it can determine the average of the pixel values ​​by color as offset information. However, this is merely an example, and the electronic device can combine the pixel values ​​of target pixels by color by setting different weights depending on the position of the target pixels.

[0069] According to another embodiment, the electronic device may determine offset information by grouping target pixels into blocks. For example, a preset number of target pixels may be grouped into a single block, and the number of target pixels per block may be the same or different depending on the setting. The electronic device may pre-store offset information for each block when the target pixels included in the block are turned on. For example, the electronic device may store the average of the pixel values ​​by color when the target pixels included in the block are turned on as offset information. According to another example, the electronic device may store the offset information of the blocks in the form of an image, which will be described in more detail later with reference to FIGS. 6 and 7. Additionally, the electronic device may store the offset information of the blocks by RGB channel.

[0070] According to another embodiment, the electronic device may determine offset information using a pre-generated artificial intelligence model based on the display information of target pixels. For example, the electronic device may store an artificial intelligence model trained based on the pixel values ​​of target pixels by color, the degree of light emitted by the display elements of target pixels, or the color correction matrix of target pixels. The electronic device may input the display information of target pixels into the stored artificial intelligence model to obtain offset information as an output value for use in correcting a captured image. However, this is merely an example, and according to other examples, the artificial intelligence model may be stored in an external device. If the artificial intelligence model is stored in an external device, the electronic device may transmit the display information of target pixels to the external device and receive the offset information output from the artificial intelligence model in response.

[0071] In step 540, the electronic device can correct the captured image based on the determined offset information. In one embodiment, the electronic device can correct the captured image by subtracting the determined offset information from the pixel values ​​of the captured image. However, this is merely an example, and the method of correcting the captured image based on the offset information is not limited to the example described above. According to another example, when flicker or the like occurs during image capture by the camera module, the electronic device can correct the image by considering image quality distortion such as flicker together with the offset information. This will be described in detail later with reference to FIG. 8.

[0072] An electronic device according to one embodiment may store a corrected captured image. Meanwhile, this is merely an example, and according to another embodiment, the electronic device may store a captured image acquired from a camera module and offset information. The offset information may be stored as one of the correction parameters for the captured image. For example, if the captured image is stored as a data file in JPEG format, the offset information may be stored in the tail area of ​​the JPEG. However, this is merely an example, and the method of storing offset information is not limited to the example described above. According to yet another embodiment, the electronic device may store a captured image acquired from a camera module and a corrected captured image.

[0073] Meanwhile, the image processing method according to the present disclosure may be performed based on display information of target pixels in a raw domain where linearity is guaranteed. However, this is merely one embodiment, and as long as linearity is guaranteed, the display information of target pixels is not necessarily limited to information in the raw domain.

[0074] FIG. 6 is a diagram illustrating an offset image used for correction of a captured image according to one embodiment.

[0075] Referring to FIG. 6, the electronic device may divide the area of ​​the display module corresponding to the camera module (hereinafter, target display area, 610) into a plurality of blocks. For example, the electronic device may divide the target display area (610) into 6x5 blocks. Each of the blocks may include a plurality of target pixels.

[0076] For each of the blocks, the electronic device can store offset information that occurs when the target pixels included in the block are turned on in the form of an image. According to one embodiment, the offset information may be stored for each average pixel value of the block for a specific color channel. For example, when the target pixels included in the 14th block are turned on, a light component may be diffused to the surrounding blocks centered on the 14th block. To determine the value of the diffused light component, the electronic device may store the offset information as offset image 14 (620) when the average pixel value for the R (red) color of the target display area (610) is I while the 14th block is turned on. Additionally, the electronic device may store the offset information as offset image 15 (630) when the average pixel value for the R color of the target display area (610) is I while the 15th block is turned on.

[0077] However, this is merely one example; depending on another example, offset images may be stored according to the color correction matrix of the pixels in the block. In yet another example, offset images may be stored according to the average value of the degree of light emitted by the display elements connected to the pixels in the block.

[0078] The electronic device can obtain offset information by summing offset images according to weights. For example, if offset images are stored for each color channel for 6x5 blocks, a total of 3x6x5 offset images may be stored. The electronic device can sum at least some of the 3x6x5 offset images according to weights. For example, if n of the 3x6x5 offset images are selected, offset information can be determined based on the following mathematical formula 1.

[0079] [Mathematical Formula 1]

[0080]

[0081] In mathematical formula 1, represents the offset image n, and represents the weight for the offset image n, and represents offset information.

[0082] Weights may be determined, for example, in proportion to the average pixel value of the block. However, this is merely one example, and the method of determining weights is not limited to the aforementioned example. In other examples, the same weight may be applied to all offset images.

[0083] FIG. 7 is a diagram illustrating an offset image used for correction of a captured image according to another embodiment.

[0084] Referring to FIG. 7, the electronic device may divide the area of ​​a display module corresponding to a camera module (hereinafter referred to as the target display area, 710) into a plurality of blocks. The electronic device according to one embodiment may set the number of target pixels included in each block differently. For example, target pixels located at the edge of the target display area (710) may have less influence on the captured image compared to target pixels located inside the target display area (710). Accordingly, the electronic device may group the target pixels located at the edge of the target display area (710) and the target pixels located inside the target display area (710) into different numbers. For example, the electronic device may group x target pixels into one block in the case of target pixels located at the edge of the target display area (710). Additionally, the electronic device may group y target pixels into one block in the case of target pixels located inside the target display area (710).

[0085] For each of the blocks, the electronic device can store offset information in the form of an image that occurs when the target pixels included in the block are turned on. According to one embodiment, the offset information may be stored for each average pixel value of the block for a specific color channel. For example, to determine the value of the diffused light component when the target pixels included in the first block are turned on, the electronic device may store the offset information as offset image 1 (720) when the average pixel value for the R (red) color of the target display area (710) is 'a' while the first block is turned on. Additionally, the electronic device may store the offset information as offset image 2 (730) when the average pixel value for the R color of the target display area (710) is 'b' while the second block is turned on.

[0086] However, this is merely one example; depending on another example, offset images may be stored according to the color correction matrix of the pixels in the block. In yet another example, offset images may be stored according to the average value of the degree of light emitted by the display elements connected to the pixels in the block.

[0087] The electronic device can obtain offset information by summing offset images according to weights. For example, if offset images are stored for each color channel for 16 blocks, a total of 3x16 offset images may be stored. The electronic device can sum at least some of the 3x16 offset images according to weights.

[0088] Weights may be determined, for example, in proportion to the average pixel value of the block. However, this is merely one example, and the method of determining weights is not limited to the aforementioned example. In other examples, the same weight may be applied to all offset images.

[0089] FIG. 8 is a flowchart illustrating a method for correcting a captured image when flicker occurs in the captured image using an electronic device according to one embodiment.

[0090] In step 810, the electronic device can identify whether flicker occurs in a captured image acquired from a camera module. In one embodiment, flicker may refer to a phenomenon in which horizontal or vertical stripes are displayed in a captured image when the shutter speed of the camera module and the frequency of the light source are different. An electronic device according to one embodiment can identify that flicker has occurred if there is an area in the captured image where the difference between pixel values ​​exceeds a threshold value. However, this is merely an example, and the method by which the electronic device identifies whether flicker has occurred is not limited to the example described above. According to another example, the electronic device can identify whether flicker has occurred in a captured image by using a previously generated artificial intelligence model for flicker identification.

[0091] In step 820, if flicker occurs in the captured image, the electronic device can identify the area where flicker occurs in the captured image. For example, the electronic device can identify an area in the captured image where the difference between pixel values ​​exceeds a threshold value as a flicker area. Additionally, the electronic device can detect flicker components that need to be removed in the flicker area, and the flicker components to be removed can be quantified as flicker values.

[0092] In step 830, the electronic device can correct the captured image based on the identified flicker area and offset information.

[0093] In this embodiment, it is assumed that the offset information is determined based on the display information of the target pixels, as described above with reference to FIGS. 5 to 7. For example, the electronic device may obtain offset information by combining previously stored offset images according to weights. The offset information may include offset values ​​representing the light components of the target pixels that need to be removed for each of the pixels constituting the captured image.

[0094] In addition, the electronic device can correct the captured images using offset values ​​and flicker values ​​corresponding to each pixel of the captured images. For example, the electronic device can correct the captured images by subtracting the product of the offset value and the flicker value from the pixel value of the captured image.

[0095] FIG. 9 is a diagram illustrating a method for correcting a captured image when flicker occurs in an electronic device according to one embodiment.

[0096] Referring to FIG. 9, as the electronic device acquires a captured image (910) from a camera module, it can identify whether flicker has occurred in the captured image (910). If flicker has occurred, the electronic device can acquire flicker information (920) by analyzing the pixel values ​​of the captured image (910). The flicker information (920) may consist of flicker values ​​that quantify the flicker components to be removed from the captured image (910).

[0097] Meanwhile, when the electronic device acquires a captured image (910) from the camera module, if the target pixels included in the area of ​​the display module corresponding to the camera module are turned on, the electronic device can acquire display information of the target pixels. Based on the display information of the target pixels, the electronic device can determine offset information (930) of the pixels included in the captured image. For example, the electronic device can determine the offset information (930) by combining the offset images of each block composed of target pixels according to weights. However, this is merely an example, and the method of determining the offset information (930) is not limited to the example described above.

[0098] An electronic device according to one embodiment can obtain correction information for a captured image (910) by multiplying the flicker values ​​included in the flicker information (920) and the offset values ​​included in the offset information (930). The electronic device can correct the captured image (910) based on the obtained correction information. For example, the electronic device can obtain a corrected captured image (940) by subtracting the value obtained by multiplying the flicker values ​​and the offset values ​​from the captured image (910).

[0099] FIG. 10 is a diagram illustrating a method for correcting a captured image when flicker occurs in an electronic device including a plurality of camera modules according to one embodiment.

[0100] Referring to FIG. 10, the electronic device (1000) may include a plurality of camera modules (1010, 1020). In this embodiment, it is assumed that the plurality of camera modules (1010, 1020) are all located below the display module.

[0101] The electronic device (1000) can determine correction information based on captured images (1015, 1025) obtained from each of the plurality of camera modules (1010, 1020). For example, the electronic device (1000) can identify whether flicker has occurred in the first captured image (1015) as the first captured image (1015) is obtained from the first camera module (1010).

[0102] When flicker occurs in the first captured image (1015), the electronic device (1000) can obtain first flicker information (1017) by analyzing the pixel values ​​of the first captured image (1015). The first flicker information (1017) may consist of flicker values ​​that quantify the flicker components to be removed from the first captured image (1015).

[0103] Additionally, the electronic device (1000) can acquire display information of target pixels when target pixels included in the area of ​​a display module corresponding to the first camera module (1010) are turned on when the first captured image (1015) is acquired from the first camera module (1010). The electronic device (1000) can determine first offset information (1019) of pixels included in the captured image based on the display information of the target pixels. For example, the electronic device (1000) can determine the first offset information (1019) by combining the offset images of each block composed of target pixels according to weights.

[0104] Additionally, the electronic device (1000) can identify whether flicker has occurred in the second captured image (1025) as the second captured image (1025) is acquired from the second camera module (1020). If flicker has occurred in the second captured image (1025), the electronic device (1000) can obtain second flicker information (1027) by analyzing the pixel values ​​of the second captured image (1025). The second flicker information (1027) may consist of flicker values ​​that quantify the flicker components to be removed from the second captured image (1025).

[0105] Additionally, the electronic device (1000) can acquire display information of target pixels when target pixels included in the area of ​​a display module corresponding to the second camera module (1020) are turned on when acquiring a second captured image (1025) from the second camera module (1020). The electronic device (1000) can determine second offset information (1029) of pixels included in the captured image based on the display information of the target pixels. For example, the electronic device (1000) can determine the second offset information (1029) by combining the offset images of each block composed of target pixels according to weights.

[0106] An electronic device (1000) according to one embodiment can compare first flicker information (1017) and second flicker information (1027) to identify whether there is a cause of image quality degradation other than flicker. For example, if the electronic device (1000) identifies that there is a cause of image quality degradation other than flicker, such as diffraction, as a result of comparing the first flicker information (1017) and the second flicker information (1027), it can separate the flicker component and the diffraction component.

[0107] When the flicker component and the diffraction component are separated, the electronic device (1000) can obtain pixel values ​​that need to be removed from the captured image due to each cause of image quality degradation as flicker values ​​and diffraction values. The electronic device (1000) can obtain a third captured image (1030) as a result of correcting the first captured image (1015) or the second captured image (1025) using the obtained flicker values ​​and diffraction values ​​and offset values ​​included in the first offset information (1019) and the second offset information (1029).

[0108] FIG. 11 is a diagram illustrating a method for an electronic device according to one embodiment to correct a captured image using an artificial intelligence model.

[0109] Referring to FIG. 11, the electronic device can acquire a corrected captured image (1130) by inputting captured image (1110) and display information into an artificial intelligence model composed of a neural network (1120). The layers and nodes constituting the neural network (1120) can determine offset information used to correct the captured image based on the captured image and display information, and can be trained to correct the captured image based on the offset information. The display information may include at least one of the color-specific pixel values ​​of target pixels included in the area of ​​a display module corresponding to the camera module, the degree of light emitted by the display element corresponding to the target pixels, and a color correction matrix.

[0110] An artificial intelligence model including a neural network (1120) according to one embodiment may be stored within an electronic device or stored in an external electronic device. When the artificial intelligence model is stored in an external electronic device, the electronic device may transmit a captured image (1110) and display information (1115) to the external electronic device. As the correction of the captured image (1110) is completed based on the captured image (1110) and display information (1115) using the artificial intelligence model in the external electronic device, the electronic device may receive the corrected captured image (1130) from the external electronic device.

[0111] Meanwhile, according to another embodiment, the electronic device may input a captured image that has undergone a pre-processing step, such as flicker removal, into an artificial intelligence model to eliminate causes of image quality degradation other than light generated from the target pixels of the display module. According to yet another embodiment, the electronic device may also eliminate causes of image quality degradation other than light generated from the target pixels of the display module by using an artificial intelligence model. This will be described in more detail later with reference to FIG. 12.

[0112] FIG. 12 is a diagram illustrating a method for an electronic device according to another embodiment to correct a captured image using an artificial intelligence model.

[0113] Referring to FIG. 12, the electronic device can obtain a second captured image (1230) as a result of performing correction by inputting a first captured image (1210) and shooting setting information into an artificial intelligence model composed of a first neural network (1220). The layers and nodes constituting the first neural network (1220) can be trained to determine correction information used to eliminate the cause of image quality degradation caused by the shooting environment, based on the captured image and shooting setting information. The shooting setting information may include setting information (e.g., exposure time, read-out timing, shutter speed, etc.) for at least one of the components included in the camera module described above with reference to FIG. 2. An artificial intelligence model including the first neural network (1220) according to one embodiment may be stored within the electronic device or may be stored in an external electronic device.

[0114] For example, the electronic device may input a first captured image (1210) and shooting setting information (1215) into an artificial intelligence model including a first neural network (1220). The electronic device may obtain a second captured image (1230) from the artificial intelligence model including the first neural network (1220) in which flicker has been removed from the first captured image (1210).

[0115] Additionally, the electronic device can obtain a third captured image (1250) as a result of inputting the second captured image (1230) and display information (1235) into an artificial intelligence model composed of a second neural network (1240) to perform correction. The layers and nodes constituting the second neural network (1240) can determine offset information used to correct the captured image based on the captured image and display information, and can be trained to correct the captured image based on the offset information. An artificial intelligence model including the second neural network (1240) according to one embodiment may be stored within the electronic device or may be stored in an external electronic device.

[0116] FIG. 13 is a block diagram of an electronic device according to one embodiment.

[0117] Referring to FIG. 13, the electronic device (1300) may include a display module (1310), a camera module (1320), a memory (1330), and at least one processor (1340). However, not all of the illustrated components are essential components. The electronic device (1300) may be implemented with more components than illustrated, or with fewer components.

[0118] The display module (1310) can display information generated by the electronic device (1300) or information received by the electronic device (1300) from another electronic device. The display module (1310) may be composed of a plurality of pixels including the aforementioned target pixels for displaying information, and information may be displayed by light generated from the pixels.

[0119] The camera module (1320) may be located below the display module (1310). An image may be captured through the camera module (1320). The camera module (1320) according to one embodiment may correspond to the one described above in FIGS. 2 and FIGS. 3.

[0120] The memory (1330) can store one or more instructions for correcting a captured image according to one embodiment in the electronic device (1300). Additionally, the memory (1330) can store an image captured through the camera module (1320).

[0121] The memory (1330) may store offset images of blocks composed of target pixels. Additionally, the memory (1330) may store an artificial intelligence model trained for correcting the captured image. The memory (1330) may also store the corrected captured image. Meanwhile, this is merely one example, and according to other embodiments, the memory (1330) may store the captured image and offset information obtained from the camera module. For example, if the captured image is stored as a data file in JPEG format, the offset information may be stored in the tail area of ​​the JPEG. According to yet another embodiment, the memory (1330) may store the captured image obtained from the camera module and the corrected captured image.

[0122] The processor (1340) can correct the captured image by executing one or more instructions stored in memory (1330).

[0123] The processor (1340) can identify the state of target pixels included in the area of ​​the display module (1310) corresponding to the camera module (1320) as the camera module (1320) is driven. If the target pixels are turned on, the processor (1340) can obtain display information of the target pixels.

[0124] The processor (1340) can determine offset information of pixels included in the captured image obtained from the camera module (1320) based on the display information of the target pixels.

[0125] A processor (1340) according to one embodiment can determine offset information by combining display information of target pixels according to preset weights.

[0126] According to another embodiment, the processor (1340) may determine offset information for correcting a captured image using offset information stored in advance for each block of target pixels. To this end, the processor (1340) may group the target pixels into blocks of a preset size. For each of the blocks, the processor (1340) may obtain offset information included in the captured image of the camera taken with the target pixels included in the block turned on, and may store the obtained offset information. The processor (1340) may determine offset information used for correcting the captured image based on at least one of the offset information of each of the stored blocks.

[0127] A processor (1340) according to another embodiment can obtain offset information from the display information of captured images and target pixels using an artificial intelligence model.

[0128] Meanwhile, a processor (1340) according to one embodiment can identify whether flicker occurs in a captured image obtained from a camera. If flicker occurs in the captured image, the processor (1340) can identify a flicker area in the captured image. Based on the identified flicker area and offset information, the processor (1340) can correct the captured image.

[0129] A processor (1340) according to one embodiment can determine offset information of pixels included in a captured image based on the values ​​of target pixels corresponding to each of the plurality of camera modules (1320) when a plurality of camera modules (1320) exist below a display module (1310).

[0130] The processor (1340) can correct the captured image based on the determined offset information. Meanwhile, if the processor (1340) determines, based on at least one criterion described above with reference to FIG. 4, that the pixels of the display module corresponding to the camera module are left in an off state, the processor may not perform the technique of correcting the captured image according to one embodiment.

[0131] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0132] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0133] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0134] Various embodiments of this document may be implemented as software (e.g., program (#40)) comprising one or more instructions stored in a storage medium (e.g., internal memory (#36) or external memory (#38)) readable by a machine (e.g., electronic device (#01)). For example, a processor (e.g., processor (#20)) of the machine (e.g., electronic device (#01)) may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0135] According to one embodiment, the method according to the various embodiments disclosed herein may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0136] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

Claim 1 A method for processing an image in an electronic device, comprising: identifying that target pixels included in an area of ​​the display module corresponding to the camera module are turned on when a camera module located below a display module in the electronic device is driven; obtaining display information of the target pixels; determining offset information of pixels included in a captured image obtained from the camera module based on the display information of the target pixels and offset information of at least one block among offset information of each of the blocks generated by grouping the target pixels; and correcting the captured image based on the determined offset information of the pixels. Claim 2 delete Claim 3 A method according to claim 1, wherein offset information of each of the blocks is determined when the target pixels included in each of the blocks are turned on. Claim 4 A method according to claim 1, further comprising the step of storing offset information for each of the blocks. Claim 5 A method according to claim 1, wherein the offset information of each of the blocks includes offset information for each RGB channel determined in a state where the target pixels included in each of the blocks are turned on for each RGB channel. Claim 6 A method according to claim 1, wherein the display information of the target pixels comprises at least one of the color-specific pixel values ​​of the target pixels, the degree of light emitted by the display element corresponding to the target pixels, and a color correction matrix. Claim 7 delete Claim 8 The method of claim 1 further comprises: a step of identifying whether flicker occurs in a captured image obtained from the camera module; and a step of identifying a flicker region in the captured image when flicker occurs in the captured image, wherein the step of correcting the captured image is to correct the captured image based on the identified flicker region and the offset information of the determined pixels. Claim 9 In claim 8, the step of correcting the captured image is a method of correcting at least one pixel included in the identified flicker region among the pixels included in the captured image based on the offset value and flicker value of the at least one pixel. Claim 10 In claim 1, the step of determining offset information of the pixels is a method of determining offset information for each region of the captured image based on the values ​​of target pixels included in each region of the display module corresponding to each of the plurality of camera modules when a plurality of camera modules exist below the display module. Claim 11 An electronic device for processing images, comprising: a memory for storing one or more instructions; a display module; a camera module located below the display module; and at least one processor for executing the one or more instructions stored in the memory, wherein when the camera module located below the display module is driven, the at least one processor identifies that target pixels included in an area of ​​the display module corresponding to the camera module are turned on, obtains display information of the target pixels, determines offset information of pixels included in a captured image obtained from the camera module based on the display information of the target pixels and offset information of at least one block among the offset information of each of the blocks generated by grouping the target pixels, and corrects the captured image based on the determined offset information of the pixels. Claim 12 delete Claim 13 An electronic device according to claim 11, wherein offset information of each of the blocks is determined when the target pixels included in each of the blocks are turned on. Claim 14 delete Claim 15 An electronic device according to claim 11, wherein the offset information of each of the blocks includes offset information for each RGB channel determined with the target pixels included in each of the blocks turned on for each RGB channel. Claim 16 An electronic device according to claim 11, wherein the display information of the target pixels comprises at least one of the color-specific pixel values ​​of the target pixels, the degree of light emitted by the display element corresponding to the target pixels, and a color correction matrix. Claim 17 delete Claim 18 An electronic device according to claim 11, wherein at least one processor identifies whether flicker occurs in a captured image acquired from the camera module, identifies a flicker area in the captured image if flicker occurs in the captured image, and corrects the captured image based on the identified flicker area and the offset information. Claim 19 An electronic device according to claim 18, wherein the at least one processor corrects at least one pixel included in the identified flicker region among the pixels included in the captured image based on the offset value and flicker value of the at least one pixel. Claim 20 An electronic device according to claim 11, wherein the at least one processor determines offset information for each region of the captured image based on the values ​​of target pixels included in each region of the display module corresponding to each of the plurality of camera modules when a plurality of camera modules exist below the display module. Claim 21 A recording medium storing a program that enables an electronic device to perform a method of processing an image, wherein, when a camera module located below a display module in the electronic device is driven, the recording medium stores a program that enables the following operations: identifying that target pixels included in an area of ​​the display module corresponding to the camera module are turned on; obtaining display information of the target pixels; determining offset information of pixels included in a captured image obtained from the camera module based on the display information of the target pixels and offset information of at least one block among the offset information of each of the blocks generated by grouping the target pixels; and correcting the captured image based on the determined offset information of the pixels.

Images