Electronic device and method for displaying image or video, and computer-readable medium

Abstract

In embodiments, an electronic device is provided. The electronic device may comprise: a memory for storing instructions; one or more processors; and a display. When executed by the one or more processors, the instructions can cause the electronic device to: acquire a second image on the basis of image processing on a first image; acquire one or more regions of interest (ROIs) on the basis of one or more objects included in the first image; and display a first portion of the first image and a second portion of the second image through the display on the basis of a first ROI from among the one or more ROIs.

Description

Electronic device, method, and computer-readable medium for displaying images or moving images

[0001] The present disclosure relates to electronic devices, methods, and computer-readable media for displaying images or videos.

[0002] Electronic devices can capture images through a camera. The electronic device can capture images based on its own factors, such as sensors, displays, or communication. The image quality may deteriorate due to internal and / or external factors, such as camera movement, lighting, humidity, sensor temperature, and data compression. The electronic device can process the captured images to display images of improved quality. The electronic device can then display the processed images through a display.

[0003] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0004] In embodiments, an electronic device is provided. The electronic device may include a memory storing instructions, at least one processor, and a display. The instructions, when executed by the at least one processor, may cause the electronic device to acquire a second image based on image processing of a first image. The instructions, when executed by the at least one processor, may cause the electronic device to acquire a plurality of regions of interest (ROIs) of the first image. The instructions, when executed by the at least one processor, may cause the electronic device to identify a first region of interest among the plurality of ROIs based on alignments of the plurality of ROIs. The instructions, when executed by the at least one processor, may cause the electronic device to display a first portion of the first image and a second portion of the second image through the display based on the first region of interest among the plurality of ROIs.

[0005] In embodiments, a non-transitory computer-readable medium is provided. The non-transitory computer-readable medium may include a memory storing instructions. The instructions, when executed by at least one processor, may cause an electronic device to acquire a second image based on image processing of a first image, acquire a plurality of regions of interest based on the first image, identify a first region of interest among the plurality of regions of interest based on alignment of the plurality of regions of interest, and display a first portion of the first image and a second portion of the second image based on the first region of interest among the plurality of regions of interest through a display.

[0006] In embodiments, a method performed by an electronic device is provided. The method may include an operation of obtaining a second image based on image processing of a first image. The method may include an operation of obtaining a plurality of regions of interest (ROIs) based on one or more objects included in the first image. The method may include an operation of identifying a first region of interest among the plurality of ROIs based on alignment of the plurality of ROIs. The method may include an operation of displaying a first portion of the first image and a second portion of the second image based on the first region of interest among the plurality of ROIs through a display.

[0007] Figure 1 is a block diagram of an electronic device within a network environment.

[0008] Figure 2 shows the components of an electronic device.

[0009] Figure 3 shows an example of image processing.

[0010] Figure 4 illustrates an operational flow of an electronic device for displaying the results of image processing based on a region of interest.

[0011] Figure 5 shows examples of object extraction.

[0012] Figures 6a to 6e show examples of acquisition of a region of interest.

[0013] Figure 7a shows the operational flow of an electronic device for displaying areas of interest.

[0014] Figures 7b and 7c illustrate examples of a user interface of an electronic device for displaying areas of interest.

[0015] Figures 8a to 8d show examples of alignment of regions of interest.

[0016] Figure 9 shows examples of changes in the display area.

[0017] Figures 10a and 10b show examples of filtering of a region of interest.

[0018] Figures 11a to 11d show examples of combinations of regions of interest.

[0019] The terms used in this disclosure are used only to describe specific embodiments and may not be intended to limit the scope of other embodiments. The singular expression may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by those of ordinary skill in the art described in this disclosure. Terms defined in general dictionaries among the terms used in this disclosure may be interpreted as having the same or similar meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined in this disclosure. In some cases, even if a term is defined in this disclosure, it cannot be interpreted to exclude embodiments of the present disclosure.

[0020] The various embodiments of the present disclosure described below illustrate a hardware-based approach as an example. However, since the various embodiments of the present disclosure include techniques utilizing both hardware and software, the various embodiments of the present disclosure do not exclude a software-based approach.

[0021] In the following description, terms referring to signals (e.g., signal, information, message, signaling), terms referring to areas within an image (e.g., portion, image portion, image area, region of interest), terms referring to image processing (e.g., image remaster, remastering, image conversion, resolution conversion, image enhancement, color correction, color correction, brightness correction, contrast adjustment, noise removal, lens distortion correction, face correction, moire removal, face correction), terms for operation states (e.g., step, operation, procedure), terms referring to network entities, terms referring to components of a device, etc., are examples for convenience of explanation. Therefore, the present disclosure is not limited to the terms described below, and other terms having equivalent technical meanings may be used.

[0022] In addition, in the present disclosure, expressions such as "more than" or "less than" may be used to determine whether a specific condition is satisfied or fulfilled, but this is merely a description for expressing an example and does not exclude descriptions such as "more than" or "less than." A condition described as "more than" may be replaced with "more than," a condition described as "less than" may be replaced with "less than," and a condition described as "more than and less than" may be replaced with "more than and less than." In addition, hereinafter, "A" to "B" mean at least one of elements from A (including A) to B (including B). hereinafter, "C" and / or "D" mean at least one of "C" or "D," that is, including {"C", "D", "C" and "D"}.

[0023] FIG. 1 is a block diagram of an electronic device within a network environment according to embodiments.

[0024] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0025] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculation, the processor (120) may store a command or data received from another component (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the command or data stored in the volatile memory (132), and store the resulting data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor)) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0026] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0027] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0028] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0029] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0030] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0031] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0032] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0033] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0034] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0035] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0036] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0037] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0038] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0039] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0040] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Of these communication modules, the corresponding communication module may be a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, 5G (5 th The wireless communication module (192) can communicate with an external electronic device (104) via a long-distance communication network such as a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can use subscriber information (e.g., international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196) to verify or authenticate the electronic device (101) within a communication network, such as the first network (198) or the second network (199).

[0041] The wireless communication module (192) is 4G (4 th The wireless communication module (192) can support 5G networks and next-generation communication technologies after the 5G network, such as new radio access technology (NR). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0042] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0043] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0044] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0045] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0046] FIG. 2 illustrates components of an electronic device (e.g., the electronic device (101) of FIG. 1).

[0047] Referring to FIG. 2, the electronic device (101) may include a memory (210) (e.g., the memory (130) of FIG. 1), a processor (220) (e.g., the processor (120) of FIG. 1), a display (230) (e.g., the display module (160) of FIG. 1), and a transceiver (e.g., the communication module (190) of FIG. 1). According to one embodiment, the memory (210) may store data such as instructions for the operation of the electronic device (101), a basic program, an application program, and setting information. The memory (210) may be configured as a volatile memory, a nonvolatile memory, or a combination of a volatile memory and a nonvolatile memory. The memory (210) may store at least one image. The memory (210) may provide the stored image upon a request of the processor (220). The memory (210) may include information about an application for managing the at least one image.

[0048] According to one embodiment, the processor (220) may control the overall operations of the electronic device (101). The processor (220) may obtain an image from the memory (210). The processor (220) may perform image processing. The processor (220) may extract a region of interest from the image. The processor (220) may perform alignment of the regions of interest. The processor (220) may control the display (230) to display a portion of the image of the region of interest. The processor (220) may control the transceiver (240) to transmit or receive information about the image (e.g., an image, a result of image processing, information about the region of interest) to or from an external electronic device. According to one embodiment, the processor (220) may include an image supply unit (221) for providing an image, an image conversion unit (223) for image processing, a region of interest determination unit (225) for obtaining a region of interest from the image, and a region of interest alignment unit (227) for determining an order for the regions of interest. According to one embodiment, the image supply unit (221), the image conversion unit (223), the region of interest determination unit (225), and the region of interest alignment unit (227) may be stored as a set of instructions or codes, which may be instructions / codes or a storage space storing instructions / codes that are at least temporarily resided in the processor (220), or may be part of the circuitry constituting the control unit 240. According to one embodiment of the present disclosure, at least one of the image supply unit (221), the region of interest determination unit (225), the image conversion unit (223), and the region of interest alignment unit (227) may be logically separated and operated within the processor (220), or may be generated and operated as at least one separate processor that is physically separated from the processor (220).

[0049] According to one embodiment, the display (230) can visually provide an image to an external party (e.g., a user) under the control of the processor (220). For example, the image data can include at least one pixel-specific information (e.g., color, brightness). By driving the pixels based at least in part on voltage values ​​or current values, visual information (e.g., text, pictures, or icons) corresponding to the image data can be displayed through the display (230). The display (230) can display an image according to the result of the image processing. For example, the display (230) can change the saturation, brightness, and / or color according to the image processing and display the image. According to one embodiment, the display (230) can display a region of interest. The display (230) can include a user interface (UI) for displaying the result of the image processing based on the region of interest.

[0050] According to one embodiment, the transceiver (240) may perform functions for transmitting and receiving signals via a wireless channel. For example, when transmitting data, the transceiver (240) encodes and modulates a transmission bit stream to generate transmission symbols having a magnitude and phase. Additionally, when receiving data, the transceiver (240) restores the reception bit stream by demodulating and decoding a baseband signal. The transceiver (240) may communicate with an external electronic device (e.g., the electronic device (102), the electronic device (104), and the server (108) of FIG. 1 ). According to one embodiment, the transceiver (240) may include a communication module for supporting a wireless access technology. For example, the wireless access technology may include Bluetooth low energy (BLE), Wireless Fidelity (Wi-Fi), WiFi Gigabyte (WiGig), or a cellular network (e.g., long term evolution (LTE), 5G NR (new radio), etc.). The transceiver (240) may be referred to as a “transmitter,” a “receiver,” or a “transmitter-receiver” because it transmits and receives signals. In addition, in the following description, transmission and reception performed through a wireless channel may be used to mean that the processing described above is performed by the transceiver (240). The transceiver (240) may transmit the region of interest and the result of the image processing to the external electronic device. The external electronic device may display an image and / or the region of interest according to the result of the image processing.

[0051] In one embodiment, image processing technology can enhance original images based on various algorithms (e.g., artificial intelligence algorithms). For example, image processing technology utilizing artificial intelligence (AI) technology performs image processing based on machine learning and deep learning algorithms. These algorithms learn how computers understand and analyze images and identify features and patterns in the images, thereby enhancing the quality of the images. Image processing technology utilizing the results of this learning enhances details in the image, removes noise, optimizes color and contrast to create realistic and beautiful images, and enhances image resolution and allows for the clear expression of fine details. Furthermore, for example, the image processing technology is widely used in the film and video game industries to enhance actually filmed scenes and enhance realism and immersion. Furthermore, for example, the image processing technology can also help doctors make accurate diagnoses in the field of medical imaging. Below, the present disclosure describes a technique for effectively displaying an image enhanced through the image processing technology described above to a user.

[0052] FIG. 3 illustrates an example of image processing. The image processing may be performed by an electronic device (e.g., the electronic device 101 of FIG. 1). Through the image processing, a first image may be converted into a second image. In one embodiment, the image processing includes a function of correcting the first image to provide a better image to a user. For example, the image processing may increase the resolution of the first image. The image processing may increase the resolution of the first image, and values ​​for the added pixels of the changed resolution may be set based on an AI algorithm. For example, the image processing may include brightness and color adjustment to increase the clarity of the first image. For example, the image processing may include removing reflected light, removing foreign matter, and / or correcting shake within the first image. For example, the image processing may include removing shadows. For example, the image processing may be referred to as image remastering, image conversion, image enhancement, image correction, upscaling, resolution conversion, color correction, and / or terms having equivalent technical meanings.

[0053] Referring to FIG. 3, the electronic device (101) may perform image processing on a first image. According to one embodiment, the electronic device (101) may display a result (300) of the image processing. For example, the electronic device (101) may obtain a second image based on the image processing on the first image. The electronic device (101) may display a first portion (310) of the first image and a second portion (320) of the second image. According to one embodiment, the electronic device (101) may identify a first area (331) and a second area (332) among an image area (330). The image area (330) refers to the entire area represented by the first image and the second image. According to one embodiment, the image area (330) may include the entire area (e.g., an absolute area) displayed through the display. For example, the image area (330) for the first image may include the entire area (e.g., absolute area) where the first image is actually displayed on the display. The image area (330) for the second image may include the entire area (e.g., absolute area) where the second image is actually displayed on the display. For example, the image area (330) for the first image is the same as the image area (330) for the second image. The electronic device (101) may identify a first area (331) where a first portion (310) of the first image is to be displayed. For example, the first area (331) may correspond to the left side of a dividing line (e.g., dividing line (341)) in the image area (330). The electronic device (101) may identify a second area (332) where a second portion (320) of the second image is to be displayed. For example, the second region (332) may correspond to the right of a dividing line (e.g., dividing line (341)) in the image region (330). In one embodiment, the first region (331) and the second region (332) may not overlap.

[0054] According to one embodiment, the electronic device (101) may display a dividing line (341) and a visual object (343) for the dividing line (341) so that the user can more easily identify the improvement effect of the image processing. The electronic device (101) may display the first part (310) of the first image and the second part (320) of the second image together with the dividing line (341). For example, the electronic device (101) may display a first identifier (345a) for identification (e.g., 'Before' text) on the first part (310) of the first image, and may display a second identifier (345b) (e.g., 'After' text) on the second part (320) of the second image, in order to indicate the result of the image processing. In FIG. 3, text is illustrated as an example of the first identifier (345a) and the second identifier (345b), but embodiments of the present disclosure are not limited thereto. According to one embodiment, the first identifier (345a) and / or the second identifier (345b) are not limited to text and may be created as objects of various types. For example, the first identifier (345a) and / or the second identifier (345b) may be implemented in various forms such as images, icons, and symbols.

[0055] According to one embodiment, a dividing line (341) may be arranged between a first portion (310) of the first image and a second portion (320) of the second image. When the electronic device (101) receives a user input (e.g., a zoom-in input) for magnifying the first image and the second image, the electronic device (101) may display an enlarged specific area for at least a portion of the image area (330). The electronic device (101) may divide the enlarged specific area into a first enlarged area and a second enlarged area based on the dividing line (341). The electronic device (101) may display a portion of the first image corresponding to the first enlarged area. The electronic device (101) may display a portion of the second image corresponding to the second enlarged area. The user may effectively check the result of the image processing by magnifying the image.

[0056] According to one embodiment, the electronic device (101) may further display a visual object (343) on the dividing line (341). According to one embodiment, the visual object (343) may be used to adjust the position of the dividing line (341). For example, the electronic device (101) may receive a user input (e.g., a drag input) for moving the visual object (343) in one direction (e.g., the (-) x-axis direction). Based on the user input, the electronic device (101) may move the position of the dividing line (341) in the one direction (e.g., the (-) x-axis direction). Although not illustrated in FIG. 3, as the dividing line (341) moves, the first region (331) may become narrower and the second region (332) may become wider. The first portion (310) of the first image may become smaller, and the second portion (320) of the second image may become larger. According to the user input, the second image improved by the image processing may be displayed more widely. For example, the electronic device (101) may receive a user input (e.g., a drag input) for moving the visual object (343) in another direction (e.g., the (+) x-axis direction). Based on the user input, the electronic device (101) may move the position of the dividing line (341) in the other direction (e.g., the (+) x-axis direction). Although not illustrated in FIG. 3, as the dividing line (341) moves, the first region (331) may widen and the second region (332) may narrow. The first part (310) of the first image may become larger, and the second part (320) of the second image may become narrower. According to the user input, the first image, which is an original image, may be displayed more widely.

[0057] According to one embodiment, a number of methods may be used to display the results of image processing. For example, as illustrated in FIG. 3, the electronic device (101) may display a first image before image processing and a second image after image processing together based on a dividing line (341). By moving the dividing line (341), the user may easily compare the first image and the second image. However, if image processing is performed locally rather than on the entire tone, it is generally difficult for the user to determine in which area the image processing was performed. For example, in frequency-variant corrections such as noise removal or upscaling, it is difficult for the user to visually determine whether or how much magnification is required. To more effectively display the results of the image processing, the electronic device (101) according to embodiments of the present disclosure may extract at least one region of interest (ROI) to automatically perform or guide magnification and movement. In addition, the electronic device (101) according to embodiments of the present disclosure may perform alignment on the extracted ROIs.

[0058] FIG. 4 illustrates an operation flow of an electronic device (e.g., an electronic device (101) of FIG. 1) for displaying the results of image processing based on a region of interest.

[0059] Referring to FIG. 4, in operation (401), the electronic device (101) (e.g., the processor (220) of FIG. 2) may obtain a second image based on image processing on a first image. The first image may be an original image. For example, the first image may be an image obtained through a camera of the electronic device (101) (e.g., the camera module (180) of FIG. 1). For example, the first image may be an image obtained from an external electronic device (e.g., the electronic device (102), the electronic device (104), or the server (108) of FIG. 1). According to one embodiment, the first image may include a single image, a GIF (graphics interchange format) including a plurality of images, and / or a video. According to one embodiment, the second image may be a result of image processing performed on the first image. The second image may include a single image, a GIF, and / or a video. In one embodiment, the image processing includes a function for correcting the first image to provide a better image to the user. For example, the image processing may include improving the image quality of the first image, the image quality of the original image, correction of distortion, shadows, reflection areas, and red-eye areas, enhancing details of the moon or a person, saturation and color conversion, and / or style conversion.

[0060] In operation (403), the electronic device (101) (e.g., the processor (220) of FIG. 2) may acquire at least one region of interest based on at least one object included in a first image. The electronic device (101) may identify at least one object based on the first image. The electronic device (101) may perform object extraction on the first image. For example, the electronic device (101) may identify at least one object included in the first image based on object detection. For example, the electronic device (101) may identify at least one object included in the first image based on image recognition. For example, the electronic device (101) may identify at least one object included in the first image based on semantic segmentation. For example, the electronic device (101) may identify at least one object included in the first image based on instance segmentation. The electronic device (101) can acquire at least one region of interest based on the at least one object. For example, the electronic device (101) can set an area including an object as a region of interest. For example, the electronic device (101) can set an area including both an area including an object and an area where a pixel difference between the first image and the second image is greater than a threshold value as a region of interest.

[0061] Although an example of obtaining a region of interest based on an object is described in FIG. 4, embodiments of the present disclosure are not limited thereto. The electronic device (101) may obtain a region of interest based on a factor other than at least one object. According to one embodiment, the electronic device (101) may set a region in which image processing of operation (401) is performed as a region of interest. According to one embodiment, the electronic device (101) may set a region other than a background region as a region of interest. According to one embodiment, the electronic device (101) may obtain a region of interest based on a user's ROI selection history. According to one embodiment, the electronic device (101) may identify a time section in a video in which a difference according to the result of the image processing is greater than a threshold value, and may set a region changed by the image processing within frames of the time section as a region of interest. According to one embodiment, the electronic device (101) may acquire a region of interest based on personalized information about a user within the electronic device (101) (e.g., preferences preset in a user account, frequency of each object among images stored within the electronic device (101). For example, the electronic device (101) may analyze an image using a specific algorithm (e.g., an artificial intelligence algorithm) and automatically set at least one region of interest for the image based on the analysis result.

[0062] In operation (405), the electronic device (101) may display a first portion of a first image and a second portion of a second image based on a first region of interest among at least one region of interest. The at least one region of interest may include a plurality of regions of interest. The electronic device (101) may perform alignment of the plurality of regions of interest. The electronic device (101) may perform alignment of the plurality of regions of interest so that a user can easily explore each region of interest or explore the regions of interest in an order that is meaningful to the user. For example, the electronic device (101) may perform alignment of the plurality of regions of interest according to a specified direction within an image area (e.g., (+) x-axis direction, clockwise direction). For example, the electronic device (101) may perform alignment of the plurality of regions of interest in descending order of the degree of difference between the first image and the second image. For example, the electronic device (101) may perform alignment of the plurality of regions of interest based on personalized information (e.g., preferences preset in a user account, frequency of each object among images stored in the electronic device (101). For example, the electronic device (101) may perform alignment of the plurality of regions of interest based on a shortest path algorithm.

[0063] The electronic device (101) can identify the first region of interest among the at least one region of interest. For example, based on the result of the alignment, the electronic device (101) can identify the region of interest with the highest priority as the first region of interest. After performing the image processing, the electronic device (101) can automatically identify the first region of interest. For example, the electronic device (101) can identify the first region of interest based on a user input. The user input can be an input for selecting the first region of interest among a plurality of regions of interest. The electronic device (101) can identify a first region (e.g., the first region (331) of FIG. 3) and a second region (e.g., the second region (332) of FIG. 3) of an image region (e.g., the image region (330) of FIG. 3) based on the first region of interest. The first region represents a region to be displayed on a screen among the image regions of the first image. The above second area represents an area among the image areas of the second image to be displayed on the screen.

[0064] According to one embodiment, the electronic device (101) can display a first region of interest. The electronic device (101) can use various display methods to make the first region of interest more effectively visible to the user. According to one embodiment, the electronic device (101) can enlarge the first region of interest. As the first region of interest is enlarged, the center of the first region of interest can be located at the center of the display area of ​​the display (230) of the electronic device (101). For example, a dividing line (e.g., dividing line 341 of FIG. 3) within the display area can be located at the center of the display area. One side of the dividing line can include a first image portion of the first image. According to one embodiment, the first image portion corresponds to a part of the first region of interest in the first image. The other side of the dividing line can include a second image portion of the second image. The second image portion corresponds to another part of the first region of interest in the second image. Additionally, according to one embodiment, the electronic device (101) may move a dividing line (e.g., dividing line (341) of FIG. 3) and a visual object (e.g., visual object (343) of FIG. 3) relative to the dividing line to the first region of interest. Additionally, according to one embodiment, the electronic device (101) may move the dividing line and the visual object from the center of the display area of ​​the display (230) of the electronic device (101) to the center of the first region of interest. Additionally, according to one embodiment, the electronic device (101) may perform both enlargement and movement of the first region of interest.

[0065] Although not shown in FIG. 4, the electronic device (101) may perform additional operations before performing the image processing. For example, the electronic device (101) may perform resizing on an image in which the number of pixels is greater than a threshold value. If the pixel values ​​of the image are used as is, speed and memory requirements may increase. According to one embodiment, the electronic device (101) may reduce the size of an image input to the image processing through resizing before performing the image processing. The size of the first image may be smaller than the size of the original image. In addition, for example, the electronic device (101) may perform channel simplification on the original image to reduce the data size. The electronic device (101) may change the color of each pixel's 3-channels (e.g., RGB (red green blue) channels) to 1 channel (e.g., black and white). By reducing the image size and image data, the amount of computation of the electronic device (101) for acquiring a region of interest may be reduced.

[0066] FIG. 5 illustrates examples of object extraction. According to one embodiment, the electronic device (101) can perform object extraction for the image. The electronic device (e.g., the electronic device (101) of FIG. 1) can obtain at least one object included in the image from the image. According to one embodiment, since the object included in the image can be a target of image processing, the electronic device (101) can set an area including the object as an area of ​​interest. The object extraction can be performed based on various algorithms. For example, the object extraction can be performed based on an algorithm using artificial intelligence (AI). Hereinafter, examples of algorithms for the object extraction are described in FIG. 5.

[0067] Referring to FIG. 5, according to one embodiment, the electronic device (101) can perform image recognition (510) on the image. For example, image recognition (510) is a technology for indicating what type of a specific object is by learning various types of images. According to one embodiment, the electronic device (101) can identify that the first object (501) is a puppy, and that the second object (502), the third object (503), and the fourth object (504) are each sheep. Based on the image recognition (510), the electronic device (101) can identify the first object (501), the second object (502), the third object (503), and the fourth object (504) from the image. According to one embodiment, the electronic device (101) may determine an area including each of the first object (501), the second object (502), the third object (503), and / or the fourth object (504) as an area of ​​interest.

[0068] According to one embodiment, the electronic device (101) can perform semantic segmentation (520) on the image. Semantic segmentation (520) is a technique for identifying the type of object in an image by segmenting pixels of the image. The electronic device (101) can identify objects in the image (e.g., a first object (501), a second object (502), a third object (503), and a fourth object (504)) by grouping pixels with similar characteristics (e.g., pixel sets, superpixels). The electronic device (101) can identify a class to which each object belongs. For example, the electronic device (101) can identify that the first object (501) belongs to the class of dogs. The electronic device (101) can identify that the second object (502), the third object (503), and the fourth object (504) belong to the class of sheep. According to one embodiment, the electronic device (101) may determine an area containing each of the first object (501), the second object (502), the third object (503), and / or the fourth object (504) as a region of interest (ROI).

[0069] According to one embodiment, the electronic device (101) may perform object detection (530) on the image. Object detection (530) is a technique for detecting an object corresponding to a class used for training from the image and providing information on the location of the object. For example, object detection (530) may represent each object within the image through a bounding box. According to one embodiment, the electronic device (101) may detect objects within the image (e.g., a first object (501), a second object (502), a third object (503), and a fourth object (504)) based on object detection (530). The electronic device (101) may determine an area including each of the first object (501), the second object (502), the third object (503), and / or the fourth object (504) as an area of ​​interest.

[0070] According to one embodiment, the electronic device (101) can perform instance segmentation (540) on the image. Instance segmentation (540) is a technique for identifying objects within an image by segmenting pixels of the image. Unlike semantic classification (520), it is possible to distinguish between instances within a class as well as the class to which the object belongs. The electronic device (101) can identify the class and the instances belonging to the class. For example, the electronic device (101) can identify a first object (501) in the class of dogs. The electronic device (101) can identify a second object (502), a third object (503), and a fourth object (504) in the class of sheep. The electronic device (101) can determine an area including each of the first object (501), the second object (502), the third object (503), and / or the fourth object (504) as an area of ​​interest.

[0071] In FIG. 5, four algorithms for identifying objects (e.g., a first object (501), a second object (502), a third object (503), and a fourth object (504)) within an image are described. In addition to using the four algorithms individually, it may also be understood as an embodiment of the present disclosure to perform object extraction based on a combination of at least two of the four algorithms. In addition to the four algorithms, an algorithm utilizing machine learning or deep learning may be used to acquire objects within the image. In addition, the present invention is not limited to an embodiment in which objects are automatically classified by artificial intelligence, and after a user arbitrarily sets an area of ​​an object, classification of the object may be performed manually. The electronic device (101) may receive a user input for classifying an object. Based on the user input, the electronic device (101) may identify an object for setting an area of ​​interest within the image. As a non-limiting example, the electronic device (101) can classify objects within the image by performing both automatic classification using artificial intelligence and manual classification based on user input.

[0072] In FIG. 5, an example of object extraction is described to determine a region of interest. In addition to object extraction, the electronic device (101) can acquire a region of interest in other ways. Below, FIGS. 6A to 6E describe examples of acquiring a region of interest from an image in a different way than the object extraction of FIG. 5.

[0073] Figures 6a to 6e show examples of acquisition of a region of interest.

[0074] Referring to FIG. 6A, the electronic device (101) can obtain a region of interest based on image processing (e.g., the image processing of FIG. 3). According to one embodiment, the electronic device (101) can perform image processing. The electronic device (101) can obtain a second image (620) based on image processing of the first image (610). According to one embodiment, the target region (611) of the first image (610) can include a first portion (615). The first portion (615) can have a first resolution. The image processing can convert the first resolution into a second resolution. The resolution represents the number of pixels that can be expressed on one screen. As the resolution of the image increases, the image can provide a clearer screen to the user. For example, the electronic device (101) can perform image processing to improve the resolution of the image by converting it into a second resolution that is higher than the first resolution. According to the image processing, the target area (621) of the second image (620) may have the second resolution. The electronic device (101) may obtain a region of interest from the target area (611) of the first image (610) and the target area (621) of the second image (620). The image processing may include not only resolution correction but also various processing operations. For example, the image processing may include removal of a shadow. An area related to a shadow in the image may have an important meaning in the image processing. The electronic device (101) may identify an area related to the shadow based on a difference image between the first image and the second image according to the image processing. Alternatively, the electronic device (101) may identify an area related to the shadow by detecting an area in which the brightness and contrast of the image are lower than a threshold value. The electronic device (101) may obtain an area of ​​interest, which is an area related to the shadow.For another example, the image processing may include removing reflected light or foreign matter. The electronic device (101) may acquire the area from which the reflected light is removed or the area from which the foreign matter is removed as the region of interest. For another example, the image processing may include adjusting the overall tones for brightness, saturation, or pixel values. The electronic device (101) may acquire the entire area of ​​the image (hereinafter, referred to as the image area) as the region of interest.

[0075] Referring to FIG. 6B, the electronic device (101) can obtain a region of interest based on a difference in image processing. The electronic device (101) can obtain a second image based on image processing of the first image. According to one embodiment, the electronic device (101) can obtain a region of interest based on a difference between values ​​of pixels (e.g., pixel values, frequency component values) of the first image and values ​​of pixels (e.g., pixel values, frequency component values) of the second image. For example, the electronic device (101) can calculate a sum of squared distance (SSD) as a difference between the first image and the second image. The electronic device (101) can calculate the difference between the values ​​of the first image and the values ​​of the second image for each pixel among a plurality of pixels of the image. The electronic device (101) can calculate the square of the difference (hereinafter, referred to as a difference square value) for each pixel. The electronic device (101) can calculate the sum of the squared differences of the pixels in a designated area. Based on the squared difference values ​​of each area, the electronic device (101) can identify an area of ​​interest among a plurality of areas within an image. A larger squared difference value indicates a larger amount of change due to image processing in the area. For example, the electronic device (101) can calculate the minimum square error (MSE) as the difference between the first image and the second image. The electronic device (101) can calculate the squared difference (hereinafter, “squared difference value”) for each pixel. The electronic device (101) can calculate the sum of the squared differences of the pixels in the designated area. The electronic device (101) can calculate the mean squared error (MSE) for the designated area by dividing the calculated sum by the number of pixels. Based on the MSE of each area, the electronic device (101) can identify an area of ​​interest among a plurality of areas within an image.The larger the MSE, the greater the change due to image processing per pixel in the corresponding area. The electronic device (101) can obtain the first candidate region of interest (631) based on the difference between the first image and the second image. However, if only the first candidate region of interest (631) is set as the region of interest, the portion other than the region (631) may not be included in the region of interest, and thus the required image may not be included. For example, if the first candidate region of interest (631) is determined to be only a portion of the face, the portion other than the first candidate region of interest (631) may not be included in the region of interest, and thus the face may be cropped in the image. Since face cropping may cause inconvenience to the user, the electronic device (101) may perform face detection and / or object extraction (e.g., object detection). For example, the electronic device (101) may obtain the second candidate region of interest (632) based on face detection. The electronic device (101) can obtain a region of interest (633) based on the first candidate region of interest (631) and the second candidate region of interest (632). The electronic device (101) can obtain a region of interest (633) that includes both the first candidate region of interest (631) and the second candidate region of interest (632). For example, the shape of the region of interest (633) may be a rectangular box. The electronic device (101) can obtain a region of interest (633) corresponding to a rectangular box that includes both the first candidate region of interest (631) and the second candidate region of interest (632).

[0076] Referring to FIG. 6C, the electronic device (101) can obtain a region of interest (646) based on saliency detection. The saliency detection refers to a technique for separating a background and an object by detecting an area where pixel values ​​change rapidly among the values ​​of pixels in an image. The electronic device (101) can obtain a saliency map (642) by performing the saliency detection on the image (641). The electronic device (101) can detect an object based on the saliency map (642). The image (643) shows the result of separating the background and the object. The background of the image (643) is displayed in black and the object is displayed in white. The electronic device (101) can identify the object separated from the background from the image based on the saliency map (642). The electronic device (101) can determine the area of ​​the object as the region of interest (646). The electronic device (101) can obtain an area of ​​interest (646) based on the above-described saliency detection.

[0077] Referring to FIG. 6D, the electronic device (101) can acquire a region of interest based on reinforcement learning (RL) using user experience. For example, reinforcement learning is learning which action is optimal to perform in a state (673). The agent (675) can perform an action (677). Each time the action (677) is performed, the external environment (680) provides a result to the interpreter (670). The interpreter (670) can provide a reward (671) and a state (673) to the agent (675). The agent (675) is given a reward (671) or the state (673) is changed, and the reinforcement learning can proceed in a direction that maximizes the reward. The electronic device (101) can collect the user's selection history of the region of interest. The electronic device (101) can utilize a learning model that reflects the user's preference through reinforcement learning on the selection history. The electronic device (101) can cluster user preferences based on the learning model. For example, the electronic device (101) can cluster user preferences based on country, region, and race. For example, the learning model can be managed for each user. According to one embodiment, the reinforcement learning can be performed in a background area of ​​the electronic device (101). The electronic device (101) can perform the reinforcement learning at a specific time in the background area. For example, the electronic device (101) can perform the reinforcement learning at the time when image processing is performed. For example, the electronic device (101) can perform the reinforcement learning when the region of interest changes.

[0078] Referring to FIG. 6E, the electronic device (101) can obtain a region of interest (ROI) from a video. According to one embodiment, the electronic device (101) can perform image processing on the video. The electronic device (101) can perform image processing on each frame of the video. The electronic device (101) can identify a time interval in which the before and after results of the image processing are greater than a threshold. According to one embodiment, the electronic device (101) can obtain the ROI based on the ROI of each frame belonging to the time interval. For example, the electronic device (101) can obtain the ROI including all ROIs of the frames belonging to the time interval. As another example, the electronic device (101) can obtain the ROI through an average operation on the ROIs of the frames belonging to the time interval. In addition, the electronic device (101) can detect a frame in which the image difference is the largest among the frames. The electronic device (101) can stop playing the video at the detected frame. The electronic device (101) regards the frame as a single image and can obtain an area of ​​interest for the single image according to the method described through FIGS. 5 to 6d.

[0079] According to one embodiment, a plurality of frames of a video may include an I (intra) frame (690) and a P (predicted) frame (695). Consecutive frames may be encoded as a single frame group. The I frame (690) is a self-contained frame that does not depend on other frames for rendering. The I frame (690) may include all data for an image in the corresponding frame. The P frame (695) is a frame that requires a reference to a previously located frame (e.g., an I frame or a P frame) for rendering, and may include data for a difference between an image of a previous frame and an image of a current frame. Since the image data of the I frame (690) includes independent information for rendering, the electronic device (101) may obtain a region of interest based on the I frame (690) when performing image processing on the video. In addition, an alignment operation for a region of interest described below may also be performed based on the I frames (690) of the video. Since the image data of the P frame (695) represents the difference between the image of the previous frame and the image of the current frame, the image data of the P frame (695) can be used for image tracking. For example, the image tracking may include object tracking. The electronic device (101) can calculate a region of interest in each of the following P frames (695) based on the movement of the object extracted from the I frame (690). The electronic device (101) can obtain the region of interest of the P frame (965) based on the image data of the corresponding P frame (965).

[0080] FIG. 7A illustrates an operational flow of an electronic device (e.g., the electronic device (101) of FIG. 1) for displaying regions of interest. According to one embodiment, the electronic device (101) may display a first image before image processing and a second image after image processing to indicate the results of image processing. The electronic device (101) may display the regions of interest together with the first image and the second image.

[0081] Referring to FIG. 7A, in operation (701), the electronic device (101) (e.g., the processor (220) of FIG. 2) may perform alignment of a plurality of regions of interest. According to one embodiment, the alignment of the plurality of regions of interest may be performed according to the priority of each region of interest. For example, the electronic device (101) may determine the priority of each region of interest to more effectively display the result of the image processing. The priority of a region in which the user is expected to be relatively more interested may be higher than the priority of a region in which the user is expected to be relatively less interested. The priority may be preset based on the user's account information, or may be preset based on the frequency of each object in images stored in the electronic device (101), and / or a preset rule. An example of the alignment of the plurality of regions of interest according to the priority is described in detail through FIG. 8A.

[0082] According to one embodiment, the alignment of the plurality of regions of interest may be performed based on the degree of difference between the first image and the second image in each region of interest. For example, the electronic device (101) may perform the alignment of the plurality of regions of interest in descending order of the degree of difference between the first image and the second image. An example of the alignment of the plurality of regions of interest based on the degree of difference is described in detail through FIG. 8B.

[0083] In one embodiment, the alignment of the plurality of regions of interest may be performed based on the location of each region of interest within the image area. For example, the electronic device (101) may perform the alignment of the plurality of regions of interest along a specified direction (e.g., left to right, counterclockwise, clockwise). An example of the alignment of the plurality of regions of interest along the specified direction is described in detail with reference to FIG. 8C.

[0084] In one embodiment, the alignment of the plurality of regions of interest may be performed based on a shortest path algorithm. For example, the electronic device (101) may align the plurality of regions of interest so that the user feels less inconvenience due to screen movement when sequentially checking the plurality of regions of interest. The plurality of regions of interest may be aligned based on the shortest path with each region of interest as a node. An example of the alignment of the plurality of regions of interest according to the shortest path is described in detail with reference to FIG. 8D.

[0085] In operation (703), the electronic device (101) (e.g., the processor (220) of FIG. 2) may display information about a plurality of regions of interest based on the result of the alignment. The electronic device (101) may sequentially display each of the plurality of regions of interest based on the alignment. According to one embodiment, the electronic device (101) may display information about the plurality of regions of interest in a region other than the region where the result of the image processing is displayed. For example, the region where the result of the image processing is displayed may include a first part (e.g., the first part (310)) of a first image and a second part (e.g., the second part (320)) of a second image. The electronic device (101) may display a dividing line (e.g., the dividing line (341)) and a visual object (e.g., the visual object (343)) between the first part and the second part. The electronic device (101) may display information indicating each of the plurality of regions of interest at the bottom of the first portion of the first image and the second portion of the second image. When a user input indicating a specific region of interest among the plurality of regions of interest is received, the electronic device (101) may enlarge and display the specific region of interest. The electronic device (101) may adjust the first portion of the first image and the second portion of the second image. The adjusted first portion of the first image may include a portion of the specific region of interest. The adjusted second portion of the second image may include another portion of the specific region of interest.

[0086] According to one embodiment, the electronic device (101) may display information about the plurality of regions of interest within an area where the result of the image processing is displayed. For example, the area where the result of the image processing is displayed may include a first portion (e.g., the first portion (310)) of a first image and a second portion (e.g., the second portion (320)) of a second image. The electronic device (101) may display a dividing line (e.g., the dividing line (341)) and a visual object (e.g., the visual object (343)) between the first portion and the second portion. The electronic device (101) may display each of the plurality of regions of interest within an overall size, e.g., an image area (e.g., the image area (330)) for the first image and the second image. As an example, each region of interest may be displayed within the image area via a rectangular box. When a user input indicating a specific region of interest among the plurality of regions of interest is received, the electronic device (101) may enlarge and display the specific region of interest. The electronic device (101) can adjust a first portion of the first image and a second portion of the second image. The adjusted first portion of the first image may include a portion of the specific region of interest. The adjusted second portion of the second image may include another portion of the specific region of interest.

[0087] FIGS. 7b and 7c illustrate examples of a user interface (UI) of an electronic device (e.g., the electronic device (101) of FIG. 1) for displaying areas of interest.

[0088] Referring to FIG. 7B, the electronic device (101) can display a first screen (710a) after image processing is performed. The electronic device (101) can display the screen (710a) after image processing is performed. For example, the electronic device (101) can display a first portion (310) of a first image and a second portion (320) of a second image. The electronic device (101) can include a dividing line (341) and a visual object (343) for controlling the position of the dividing line (341). The electronic device (101) can acquire multiple regions of interest. The electronic device (101) can display multiple regions of interest. For example, the electronic device (101) can acquire a first region of interest (721), a second region of interest (722), and a third region of interest (723). The electronic device (101) can perform alignment for the multiple regions of interest. For example, the first region of interest (721), the second region of interest (722), and the third region of interest (723) may be arranged in that order. The electronic device (101) may display the plurality of regions of interest. For example, the electronic device (101) may display the first region of interest (721), the second region of interest (722), and the third region of interest (723). The electronic device (101) may display a visual object representing each region of interest within the image. The shape of the visual object may be a rectangular box. The electronic device (101) may display information about the regions of interest in the lower region (730). For example, the electronic device (101) may display first information (731) about the first region of interest (721), information (732) about the second region of interest (722), and information (733) about the third region of interest (723). Although an example of displaying a region of interest and information about the region of interest together is described in FIG. 7b, embodiments of the present disclosure are not limited thereto.In one embodiment, the electronic device (101) may display a first region of interest (721), a second region of interest (722), and a third region of interest (723) within an image without a bottom region (730). In another embodiment, the electronic device (101) may display information about each region of interest only in the bottom region (730), without a separate visual object (e.g., a square box) within the image.

[0089] According to one embodiment, the electronic device (101) may receive a user input indicating a first region of interest (721). For example, the electronic device (101) may receive a touch input on the first information (731) for the first region of interest (721) of the lower region (730). The electronic device (101) may control the screen so that the dividing line (341) and the visual object (343) move to the first region of interest (721). The electronic device (101) may display the screen (710b). According to one embodiment, a dividing line (341) for comparing before and after results of image processing may be positioned in the first region of interest (721) indicated by the user input. For example, the dividing line (341) for comparing before and after results of image processing in the first region of interest (721) indicated by the user input may be positioned to cross at least a portion of the first region of interest (721). Figure 7b illustrates an example of a dividing line (341) and a visual object (343) moving to a user-desired region of interest. However, in addition to moving the screen to the region of interest, the electronic device (101) can also enlarge the region of interest and display the enlarged region of interest. An example of enlargement of the region of interest is described below with reference to Figure 7c.

[0090] Referring to FIG. 7c, the electronic device (101) can display a first screen (710a) after image processing is performed. The electronic device (101) can display the screen (710a) after image processing is performed. For example, the electronic device (101) can display a first portion (310) of a first image and a second portion (320) of a second image. The electronic device (101) can include a dividing line (341) and a visual object (343) for controlling the position of the dividing line (341). For a description of the first region of interest (721), the second region of interest (722), and the third region of interest (723), reference may be made to FIG. 7b. For description of the first information (731) for the first area of ​​interest (721), the first information (732) for the second area of ​​interest (722), and the third information (733) for the third area of ​​interest (723), reference may be made to FIG. 7b.

[0091] The electronic device (101) can receive a user input pointing to a first region of interest (721). For example, the electronic device (101) can receive first information (731) about the first region of interest (721) of the lower region (730) or a touch input on the first region of interest (721). The electronic device (101) can enlarge the first region of interest (721). According to one embodiment, the electronic device (101) can divide the first region of interest (721) into a first region and a second region. The electronic device (101) can enlarge the size of the first image portion. According to one embodiment, the first image portion can be a part of the first image corresponding to the first region among the first region of interest (721). The size of the first image portion can be changed based on the size of the display area of ​​the display of the electronic device (101). According to one embodiment, the electronic device (101) can display the first image portion having a changed size through the display. The electronic device (101) can enlarge the size of at least a portion of the second image portion. The second image portion may be a portion of the second image corresponding to the second area among the first region of interest (721). The size of the second image portion may be changed based on the size of the display area of ​​the display of the electronic device (101). The electronic device (101) can display the second image portion having the changed size through the display. The electronic device (101) can display a screen (710c). A dividing line (341) may be located at the substantial center of the screen (710c). A visual object (343) may be located on the dividing line (341). According to one embodiment, even when switching from screen (710a) to screen (710c), the dividing line (341) and the visual object (343) may continue to be displayed in the same position without moving.A dividing line (341) for comparing the before and after results of image processing may be positioned in a first area of ​​interest (721) indicated by user input.

[0092] According to one embodiment, the electronic device (101) can display the first image portion and the second image portion through a user input on the image (e.g., a zoom-in input, a zoom-out input, a drag input), in addition to a user input indicating a region of interest. For example, the electronic device (101) can receive a zoom-in input for enlarging a specific region. For example, the electronic device (101) can receive a zoom-out input for reducing a specific region. For example, the electronic device (101) can receive a drag input for exploring a specific region.

[0093] In FIGS. 7B and 7C, examples are described of displaying the entire image after image processing and then positioning a dividing line (341) in a specific region of interest according to a user input. However, the UIs described through FIGS. 7B and 7C are exemplary, and the embodiments of the present disclosure are not limited thereto. According to one embodiment, the electronic device (101) may immediately display the first region of interest (721) having the highest priority among the regions of interest after image processing. For example, the electronic device (101) may display the screen (710b) without the screen (710a) after the image processing. For example, the electronic device (101) may display the screen (710c) without the screen (710a) after the image processing.

[0094] Figures 8a to 8d show examples of alignment of regions of interest.

[0095] Referring to FIG. 8A, according to one embodiment, the electronic device (101) may sort a plurality of regions of interest based on priorities. The priorities of the regions of interest may reflect the user's preferences. The electronic device (101) may sort a plurality of regions of interest for a plurality of objects. Each region of interest may include an object. The higher the user's preference for an object, the higher the priority may be set. For example, the electronic device (101) may store a plurality of images (810) in a memory (e.g., the memory (130)). The electronic device (101) may perform an analysis on the plurality of images (810). Based on the analysis of the plurality of images (810), the electronic device (101) may obtain a frequency of each object category (e.g., person, object, animal, plant, food, building) of objects included in the plurality of images (810). For example, a histogram (830) represents a frequency of each object category. The histogram (830) represents the frequency of food (831), the frequency of animals (832), the frequency of people (833), and the frequency of plants (834). According to one embodiment, according to the histogram (830), the electronic device (101) 'people plant animal The user's preference in the order of 'food' can be determined. According to one embodiment, the electronic device (101) can sort the regions of interest based on the priority of the object categories. For example, the electronic device (101) can sort the plurality of regions of interest in the order of the person category, plant category, animal category, and food category.

[0096] According to one embodiment, the electronic device (101) may determine priorities for individual people even in the 'People' category. For example, the plurality of images (810) may include people A, people B, people C, people D, and people E belonging to the 'People' category. The electronic device (101) may obtain the frequency of each of people A, people B, people C, people D, and people E within the plurality of images (810). For example, a histogram (850) represents the frequency of each object. The histogram (850) represents the frequency for people A (851), people B (852), people C (853), people D (854), and people E (855). According to the histogram (850), the electronic device (101) may determine priorities for individual people in the 'People' category. Person A Person C E character The electronic device (101) can determine the user's preference in order of 'D'. The electronic device (101) can sort the areas of interest based on the priority of the objects. For example, the electronic device (101) can sort the areas of interest according to the preference of each object, even within the same category.

[0097] In FIG. 8A, an example of determining the priority of an object category and determining the priority of objects within the object category through analysis of a plurality of images (810) stored in an electronic device (101) is described. However, embodiments of the present disclosure are not limited thereto. According to one embodiment, the electronic device (101) may sort a plurality of regions of interest based on the frequency of each object stored in the plurality of images (810), regardless of the object category.

[0098] In FIG. 8A, an example of identifying a user's preferences through analysis of multiple images (810) stored in an electronic device (101) is described. However, embodiments of the present disclosure are not limited thereto. According to one embodiment, a user's preferences may be identified through analysis of not only images stored in the electronic device (101), but also images in a device or cloud storage linked to the user's account of the electronic device (101).

[0099] Referring to FIG. 8B, the electronic device (101) can align a plurality of regions of interest based on the degree of difference between the first image and the second image. The electronic device (101) can obtain the second image based on image processing of the first image. For each region of interest, the first image can include a first image portion corresponding to the region of interest, and the second image can include a second image portion corresponding to the region of interest. The electronic device (101) can indicate the degree of difference of the region of interest based on the difference between the first image portion and the second image portion. The degree of difference can be referred to as a difference score, a difference level, a difference value, difference information, or a term having an equivalent technical meaning therefor for image processing. For example, the electronic device (101) can obtain a first region of interest (861), a second region of interest (862), and a third region of interest (863).

[0100] The electronic device (101) can calculate, for each pixel of the pixels of the first region of interest (861), a difference value between the values ​​of the first image and the values ​​of the second image. The electronic device (101) can calculate a sum of the difference values ​​of the pixels of the first region of interest (861). For example, the sum of the difference values ​​of the pixels of the first region of interest (861) can be approximately 99. The electronic device (101) can calculate, for each pixel of the pixels of the second region of interest (862), a difference value between the values ​​of the first image and the values ​​of the second image. The electronic device (101) can calculate a sum of the difference values ​​of the pixels of the second region of interest (862). For example, the sum of the difference values ​​of the pixels of the second region of interest (862) can be approximately 77. The electronic device (101) can calculate a difference value between the values ​​of the first image and the values ​​of the second image for each pixel of the pixels of the third region of interest (863). The electronic device (101) can calculate a sum of the difference values ​​of the pixels of the third region of interest (863). For example, the sum of the difference values ​​of the pixels of the third region of interest (863) may be approximately 88. Referring to the screen (860), the electronic device (101) can sort the first region of interest (861), the second region of interest (862), and the third region of interest (863) in descending order of the degree of difference. The sorting of the regions of interest can be performed in the order of the first region of interest (861), the third region of interest (863), and the second region of interest (862).

[0101] Referring to FIG. 8C, according to one embodiment, the electronic device (101) may align a plurality of regions of interest along a specified direction. In order to provide an intuitive interface to the user when switching regions of interest, the plurality of regions of interest may be aligned along the specified direction. For example, the electronic device (101) may align the plurality of regions of interest from left to right (or right to left). Referring to the screen (870), the alignment of regions of interest may be performed in the following order: a first region of interest (871), a second region of interest (872), a third region of interest (873), and a fourth region of interest (874).

[0102] In FIG. 8C, alignment from left to right is described as an example, but embodiments of the present disclosure are not limited thereto. According to one embodiment, the electronic device (101) can perform alignment of the plurality of regions of interest in a bottom-up direction (or top-down direction). The electronic device (101) can identify the location of each region of interest among the plurality of regions of interest. If the difference between the locations of the plurality of regions of interest (e.g., (x, y) coordinates when the horizontal is the x-axis and the vertical is the y-axis) in the vertical direction (e.g., the sum of the difference values ​​in the x-axis) is greater than the difference in the horizontal direction (e.g., the sum of the difference values ​​in the x-axis), it may be advantageous for the plurality of regions of interest within the electronic device (101) to be arranged in a vertical direction rather than a horizontal direction. For example, the electronic device (101) can align the plurality of regions of interest in a bottom-up direction (e.g., in a direction in which the y-coordinate increases). In one embodiment, the electronic device (101) can perform alignment of the plurality of regions of interest in a clockwise (or counterclockwise) direction. For example, the electronic device (101) can sequentially align the plurality of regions of interest when a reference point moves along a virtual closed curve (e.g., a circle) within the image, with the regions of interest adjacent to the reference point.

[0103] Referring to FIG. 8D, the electronic device (101) may align multiple regions of interest based on a shortest path algorithm. In order to reduce inconvenience caused by screen movement when switching regions of interest, the multiple regions of interest may be aligned along a designated path. The designated path may be determined based on a shortest path algorithm that uses each region of interest as a node. For example, the shortest path algorithm may include a Dijkstra algorithm, a depth first search algorithm, and / or a breadth first search algorithm. Referring to the screen (880), alignment of regions of interest may be performed in the following order: a first region of interest (881), a second region of interest (882), a third region of interest (883), a fourth region of interest (884), a fifth region of interest (885), and a sixth region of interest (886).

[0104] In addition to the methods described in FIGS. 8A through 8D, various methods can be used to align multiple regions of interest. For example, multiple regions of interest can be aligned based on preferences derived from reinforcement learning. The electronic device (101) can collect a user's selection history of regions of interest. The electronic device (101) can utilize a learning model that reflects the user's preferences through reinforcement learning based on the selection history. The electronic device (101) can align multiple regions of interest based on the learning model.

[0105] FIG. 9 illustrates examples of changes in a display area. A display (e.g., display (220) of FIG. 2) of an electronic device (e.g., electronic device (101) of FIG. 1) can display an image through the display area. According to one embodiment, the size and / or aspect ratio of the display area can be changed depending on the state of the electronic device (101). Hereinafter, in FIG. 9, before the ratio of the display area is changed, the image area and the region of interest may be referred to as a first image area (911) and a first region of interest (951), respectively. After the ratio of the display area is changed, the image area and the region of interest may be referred to as a second image area (912) and a second region of interest (952), respectively.

[0106] Referring to FIG. 9, the display area of ​​the electronic device (101) may be changed. For example, the electronic device (101) may be a rollable device (931). The display of the rollable device (931) (e.g., the display (230) of FIG. 2) may be flexible. For example, the display may include a display area that is exposed outside the housing of the electronic device (101), which provides at least a portion of the outer surface of the electronic device (101). For example, since the display has flexibility, at least a portion of the display may be rollable into the housing or slidable into the housing. For example, the size of the display area may be changed depending on the size of the at least a portion of the display that is rolled into the housing or slid into the housing. For example, the electronic device (101) including the display may be in a plurality of states. The above-described plurality of states may include a first state (932a) that provides the display area having a first size and a second state (932b) that provides the display area having a second size different from the first size. For example, the state of the electronic device (101) may change from the first state (932a) to the second state (932b) that provides the display area having the second size larger than the first size. The size and the width-to-height ratio of the display area may vary.

[0107] For example, the electronic device (101) may be a foldable device (933). The foldable device (933) may include a first housing, a second housing, a hinge structure for connecting the first housing and the second housing, a first display, and a second display. For example, the first display may be disposed on a first surface of the first housing. The second display may be disposed across a second surface opposite the first surface of the first housing and the first surface of the second housing. In a first state (934a), the second surface of the first housing may be disposed to face the first surface of the second housing. The first state (934a) may be referred to as a folded state. In the first state (934a), the foldable device (933) may display an image through a first display area of ​​the first display. In the second state (934b), the second side of the second housing and the first side of the second housing may be arranged to face the same direction. The second state (934b) may be referred to as an unfolded state. In the second state (934b), the foldable device (933) may display an image through the second display area of ​​the second display. For example, the state of the electronic device (101) may change from the first state (934a) to the second state (934b). The first display area of ​​the first display and the second display area of ​​the second display may be different. The size of the first display area and the size of the second display area may be different. The width-to-height ratio of the first display area and the width-to-height ratio of the second display area may be different.

[0108] For example, the electronic device (101) may be a bar-type device (935). The bar-type device (935) may include a display (e.g., the display (230) of FIG. 2). The bar-type device (935) may operate in landscape mode (or horizontal mode) or portrait mode (or vertical mode) depending on the attitude of the bar-type device (935). Depending on the mode of the bar-type device (935), the screen ratio of the display area may vary. For example, the state of the electronic device (101) may change from a first state (936a) to a second state (936b). The horizontal-to-vertical ratio of the display area of ​​the display may vary.

[0109] The electronic device (101) can display an image area corresponding to the entire image (e.g., a first image area (911), a second image area (912)). The electronic device (101) can display an area of ​​interest related to image processing (e.g., a first area of ​​interest (951), a second area of ​​interest (952)). As the width-to-height ratio of the display area changes, the electronic device (101) can convert the first image area (911) into the second image area (912). The second image area (912) can be determined based on the width-to-height ratio of the first image area (911), the size of the display area, and the ratio of the display area. As the width-to-height ratio of the display area changes, the electronic device (101) can convert the first area of ​​interest (951) into the second area of ​​interest (952). The size and aspect ratio of the second area of ​​interest (952) can be determined based on the aspect ratio of the first area of ​​interest (951), the size of the display area, and the aspect ratio of the display area. For example, the electronic device (101) can scale (e.g., enlarge or reduce) the first image area (911) while maintaining the aspect ratio of the first image area (911). The electronic device (101) can enlarge or reduce the first image area (911) so that the horizontal length of the first image area (911) matches the horizontal length of the display area, or so that the vertical length of the first image area (911) matches the vertical length of the display area. The electronic device (101) can obtain the second image area (912) based on the enlargement or reduction of the first image area (911). The electronic device (101) can scale (e.g., enlarge or reduce) the first region of interest (951) while maintaining the horizontal-to-vertical ratio of the first region of interest (951).The electronic device (101) can enlarge or reduce the first region of interest (951) so that the horizontal length of the first region of interest (951) matches the horizontal length of the display area, or so that the vertical length of the first region of interest (951) matches the vertical length of the display area. The electronic device (101) can obtain the second region of interest (952) based on the enlargement or reduction of the first region of interest (951). If the second region of interest (952) is included in the second image area (912), the electronic device (101) can not display the second region of interest (952). For example, if the horizontal length of the second region of interest (952) is smaller than the horizontal length of the second image region (952), and if the vertical length of the second region of interest (952) is smaller than the vertical length of the second image region (952), the electronic device (101) may determine that the second region of interest (952) is included in the second image region (912). Displaying the second region of interest (952) through the display is to provide an improved image processing effect by enlarging the second region of interest (952). However, if the size of the second region of interest (952) is smaller than the size of the second image region (912), the electronic device (101) may have little practical benefit in displaying the second region of interest (952) separately. The electronic device (101) may not display the second region of interest (952). In this way, the operation of determining whether to display a specific region of interest based on a change in the display area (e.g., a change in the ratio of the display area) may be referred to as filtering the region of interest. When the horizontal-to-vertical ratio of the display area changes, the electronic device (101) may filter at least one region of interest among a plurality of regions of interest. Based on the change in the ratio of the display area, the electronic device (101) may perform a transformation on each region of interest among the plurality of regions of interest.The electronic device (101) may exclude display of a display area if the converted image (e.g., the second image area (912)) includes a converted area of ​​interest (e.g., the second area of ​​interest (952)).

[0110] In FIG. 9, an example of changing from the first state to the second state is described, but even when returning from the second state to the first state (e.g., changing from the second state (932b) to the first state (932a)), filtering of the region of interest may be performed. Meanwhile, a region of interest excluded through filtering in the second state (936b) may be extracted again as a region of interest in the first state (936a). Even if the region of interest is excluded from the list of regions of interest to be displayed in the second state (936b), the electronic device (101) may be configured to display it in the first state (931a) based on information about the region of interest stored in another state (e.g., the first state (936a)).

[0111] Although examples of changing the ratio of the display area depending on the state of the electronic device are described in FIG. 9, the embodiments of the present disclosure are not limited thereto. A picture-in-picture (PIP) can be performed under user control. The sizes of the image area and the region of interest can be adjusted based on a window (e.g., a pop-up window, a PIP window) of an application displaying an image. The size of the window can be variable based on user input or a specified control method. The sizes of the image area and the region of interest can be adjusted based on the ratio of the window. Based on the results of the adjustment, the electronic device (101) can perform filtering of the region of interest.

[0112] Figures 10a and 10b show examples of filtering of a region of interest.

[0113] Referring to FIG. 10A, an image (1000) may include an image area (1010) corresponding to the entire image and an area of ​​interest (1015) related to image processing. The electronic device (101) may operate in a portrait mode (e.g., a first state (936a)). The sizes of the image area (1010) and the area of ​​interest (1015) may be changed to match the ratio in the portrait mode. The ratio of the image area (1010) may be maintained, and the size of the image area (1010) may be changed. For example, referring to the screen (1020), the image area (1010) may be scaled such that the horizontal length of the image area (1010) matches the horizontal length of the display area in the portrait mode, or such that the vertical length of the image area (1010) matches the vertical length of the display area in the portrait mode of the electronic device (101). The electronic device (101) may display a screen (1020) to show the entire image to the user. The ratio of the region of interest (1015) may be maintained, and the size of the region of interest (1015) may be changed. For example, referring to the screen (1025), the region of interest (1015) may be scaled so that the vertical length of the region of interest (1015) matches the vertical length of the display area in the portrait mode, or so that the horizontal length of the region of interest (1015) matches the horizontal length of the display area in the portrait mode. The size of the region of interest on the screen (1025) is larger than the size of the image area (1010) on the screen (1020). The electronic device (101) may display the screen (1025) to show the user the region of interest on which image processing has been performed.

[0114] Referring to FIG. 10b, the electronic device (101) can operate in a landscape mode (e.g., the second state (936b)). The sizes of the image area (1010) and the area of ​​interest (1015) can be changed to match the ratio in the landscape mode. The ratio of the image area (1010) can be maintained, and the size of the image area (1010) can be changed. For example, referring to the screen (1030), the image area (1010) can be scaled such that the horizontal length of the image area (1010) matches the horizontal length of the display area in the landscape mode, or such that the vertical length of the image area (1010) matches the vertical length of the display area in the landscape mode. The electronic device (101) can display the screen (1030) to show the entire image to the user. The ratio of the area of ​​interest (1015) can be maintained, and the size of the area of ​​interest (1015) can be changed. For example, referring to screen (1035), the region of interest (1015) can be scaled so that the vertical length of the region of interest (1015) matches the vertical length of the display area of ​​the landscape mode, or so that the horizontal length of the region of interest (1015) matches the horizontal length of the display area of ​​the landscape mode. However, the size of the region of interest on screen (1035) is not larger than the size of the image area (1010) on screen (1030). After scaling, in the landscape mode, the image area (1010) can include the region of interest (1015). There is no practical benefit in displaying the screen (1035) to show the user the region of interest on which image processing has been performed. The electronic device (101) can exclude the region of interest (1015) from the list of regions of interest to be displayed in the landscape mode.

[0115] Although an example of excluding a region of interest from a list of regions of interest to be displayed when an image region includes a region of interest is described in FIG. 10B, embodiments of the present disclosure are not limited thereto. According to one embodiment, even if the image region does not entirely include the region of interest, if an overlapping ratio (e.g., a ratio of an overlapping region to an area of ​​an image region) is greater than or equal to a threshold value, the electronic device (101) may exclude the region of interest from the list. For example, the image region may partially overlap with the region of interest. A portion of the image region may overlap with a portion of the region of interest. The overlapping region may represent a portion of the image region. If the ratio of an area of ​​a portion of the image region to the entire area of ​​the image region is greater than or equal to the threshold value, the electronic device (101) may exclude the region of interest from the list.

[0116] Figures 11a to 11d illustrate examples of combining regions of interest. The combining may include an operation for outputting a single region of interest when the regions of interest overlap.

[0117] Referring to FIG. 11A, a screen (1100a) represents multiple regions of interest. The electronic device (101) can identify objects within an image. For example, the electronic device (101) can identify a person, a face, and a ball. The electronic device (101) can acquire multiple regions of interest based on the objects. The multiple regions of interest can include a first region of interest (1101), a second region of interest (1103), and a third region of interest (1105). For example, if the third region of interest (1105) includes both the second region of interest and the third region of interest (1105), the electronic device (101) has little practical benefit in displaying both the second region of interest (1103) and the third region of interest (1105) to the user. In one embodiment, the electronic device (101) can identify one region of interest based on the second region of interest (1103) and the third region of interest (1105). For example, the electronic device (101) may generate a single region of interest (e.g., a third region of interest (1105)) that includes both regions of interest. For another example, the electronic device (101) may identify a region of interest having a smaller size (e.g., a second region of interest (1103)) among the two regions of interest. For yet another example, the electronic device (101) may identify a region of interest having a larger size (e.g., a third region of interest (1105)) among the two regions of interest.

[0118] Although FIG. 11A illustrates one region of interest encompassing another region of interest, embodiments of the present disclosure are not limited thereto. In one embodiment, if the overlapping area between one region of interest and another region of interest is greater than a threshold value (e.g., a threshold size within the screen), the electronic device (101) may combine the regions of interest. In one embodiment, if the ratio of the overlapping area to the area of ​​the smaller region of interest among the regions of interest is greater than the threshold value, the electronic device (101) may combine the regions of interest. For example, the electronic device (101) may identify a region of interest that is larger in size. The electronic device (101) may identify a region of interest that is smaller in size. The electronic device (101) may generate a new region of interest that includes all of the regions of interest.

[0119] Referring to FIG. 11B, a screen (1100b) represents a display area of ​​interest. The display area of ​​interest includes the area of ​​interest and represents a screen displayed through the display of the electronic device (101). For example, the display area of ​​interest may represent a displayable area when the area of ​​interest is enlarged through a user input for the area of ​​interest in the electronic device (101). The electronic device (101) may operate in a portrait mode (e.g., a first state (936a)). For example, in the portrait mode, the width-to-height ratio of the display area of ​​the electronic device (101) may be 9:16. To correspond to the ratio in the portrait mode, the electronic device (101) may acquire a first display area of ​​interest (1111), a second display area of ​​interest (1113), and a third display area of ​​interest (1115). The first display area of ​​interest (1111) may include the first area of ​​interest (1101). The second display area of ​​interest (1113) may include the second area of ​​interest (1103). The third display area of ​​interest (1115) may include the third area of ​​interest (1105). The second display area of ​​interest (1113) and the third display area of ​​interest (1115) may overlap. Due to the overlap, the electronic device (101) may merge the second display area of ​​interest (1113) and the third display area of ​​interest (1115). For example, the electronic device (101) may display only the third display area of ​​interest (1115), excluding the second display area of ​​interest (1113). The electronic device (101) may calculate a display area of ​​interest for each area of ​​interest and determine whether there is overlap between the display areas of interest, thereby identifying areas of interest to be actually displayed.

[0120] Referring to FIG. 11c, the screen (1100c) indicates regions of interest to be displayed. For example, the electronic device (101) can identify the first region of interest (1101) and the third region of interest (1105) among the first region of interest (1101), the second region of interest (1103), and the third region of interest (1105). The electronic device (101) can display the first region of interest (1101) and the third region of interest (1105).

[0121] Referring to FIG. 11d, the screen (1100d) represents a display region of interest. For example, the display region of interest may represent a displayable region when the region of interest is enlarged through a user input for the region of interest in the electronic device (101). The electronic device (101) may operate in a landscape mode (e.g., the second state (936b)). For example, in the landscape mode, the width-to-height ratio of the display region of the electronic device (101) may be 16:9. To correspond to the ratio in the landscape mode, the electronic device (101) may acquire a fourth display region of interest (1121), a fifth display region of interest (1123), and a sixth display region of interest (1125). The fourth display region of interest (1121) may include the first region of interest (1101). The fifth display region of interest (1123) may include the second region of interest (1103). The sixth display area of ​​interest (1125) may include the third display area of ​​interest (1105). The fifth display area of ​​interest (1123) and the sixth display area of ​​interest (1125) may overlap. Due to the overlapping, the electronic device (101) may merge the fifth display area of ​​interest (1123) and the sixth display area of ​​interest (1125). For example, the electronic device (101) may display only the sixth display area of ​​interest (1125), excluding the fifth display area of ​​interest (1123). The fourth display area of ​​interest (1121) and the fifth display area of ​​interest (1125) may overlap. Due to the overlapping, the electronic device (101) may merge the fourth display area of ​​interest (1121) and the sixth display area of ​​interest (1125). For example, the electronic device (101) can generate a region of interest (1127) including a fourth display region of interest (1123) and a sixth display region of interest (1125). The electronic device (101) can display the generated region of interest (1127) through a display.

[0122] Examples of selection or combination of regions of interest within an image area are described in FIGS. 11A to 11D . In addition to determining the degree of overlap between regions of interest, it may also be understood as an embodiment of the present disclosure to determine the degree of overlap between a region of interest and a display area. For example, the entire image may be displayed through the display area. If the ratio of the displayed region of interest to the display area is greater than a threshold value, the electronic device (101) may control the display of the specific region of interest to be refrained from displaying it.

[0123] In embodiments, an electronic device is provided. The electronic device may include a memory storing instructions, at least one processor, and a display. The instructions, when executed by the at least one processor, may cause the electronic device to obtain a second image based on image processing of a first image. The instructions, when executed by the at least one processor, may cause the electronic device to obtain a plurality of regions of interest (ROIs) based on the first image. The instructions, when executed by the at least one processor, may cause the electronic device to identify a first region of interest among the plurality of ROIs based on alignments of the plurality of ROIs. The instructions, when executed by the at least one processor, may cause the electronic device to display a first portion of the first image and a second portion of the second image through the display based on the first region of interest among the plurality of ROIs.

[0124] In one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to acquire one or more objects included in the first image. The instructions, when executed by the at least one processor, may cause the electronic device to acquire the plurality of regions of interest based on the acquired one or more objects.

[0125] In one embodiment, the plurality of regions of interest may be arranged based on the object-specific frequency of one or more objects within the plurality of images stored in the electronic device. The object-specific frequency may indicate the number of times a corresponding object is included within the plurality of images stored in the electronic device.

[0126] In one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to identify an object category based on a frequency per object and to acquire one or more objects based on the object category. The frequency per object may indicate the number of times a corresponding object is included in a plurality of images stored in the electronic device.

[0127] In one embodiment, the plurality of regions of interest may be ordered based on a degree of difference in the image processing. The degree of difference in the image processing may represent a difference between a first image portion associated with the region of interest within the first image and a second image portion associated with the region of interest within the second image.

[0128] In one embodiment, the plurality of regions of interest may be aligned based on a designated algorithm. The designated algorithm may include a shortest path algorithm that uses each region of interest among the plurality of regions of interest as a node.

[0129] In one embodiment, the image processing may include adjusting at least one of brightness, color, or resolution of the first image. The plurality of regions of interest may be acquired based on the positions of pixels in which at least one of brightness, color, or resolution is adjusted according to the image processing.

[0130] In one embodiment, the plurality of regions of interest may be obtained based on differences between pixels of the first image and pixels of the second image.

[0131] In one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to obtain a first candidate region of interest based on a difference between pixels of the first image and pixels of the second image. The instructions, when executed by the at least one processor, may cause the electronic device to obtain a second candidate region of interest associated with one of one or more objects included in the first image. The instructions, when executed by the at least one processor, may cause the electronic device to obtain a region of interest that includes both the first candidate region of interest and the second candidate region of interest.

[0132] In one embodiment, the plurality of regions of interest may be obtained based on saliency detection to separate one or more objects included in the first image and a background region of the first image.

[0133] In one embodiment, the first image may be associated with a P frame among a plurality of frames. The plurality of regions of interest may be identified based on data included in the P frame.

[0134] In one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to display, through the display, a dividing line for distinguishing the first portion of the first image and the second portion of the second image, and a visual object for controlling the dividing line. The dividing line may be displayed across the first region of interest among the plurality of regions of interest. The visual object may be displayed on the dividing line.

[0135] In one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to display a visual object for each region of interest among the plurality of regions of interest through the display. The instructions, when executed by the at least one processor, may cause the electronic device to display the first region of interest through the display based on a user input on a visual object for the first region of interest.

[0136] In one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to, based on identifying that an aspect ratio of a display area of ​​the display has changed, obtain an image area according to the changed aspect ratio and a second area of ​​interest related to the first area of ​​interest. The image area may represent an entire area for the first image and the second image. The instructions, when executed by the at least one processor, may cause the electronic device to display the image area through the display if the second area of ​​interest is included in the image area. If the second area of ​​interest is not entirely included in the image area, the instructions may cause the electronic device to display the second area of ​​interest through the display.

[0137] In one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to identify a first region and a second region within an image region based on the first region of interest. The first portion of the first image may correspond to the first region within the image region.

[0138] The second portion of the second image may correspond to the second area among the image areas.

[0139] In embodiments, a non-transitory computer-readable medium is provided. The non-transitory computer-readable medium may include a memory storing instructions. The instructions, when executed by at least one processor, may be configured to cause an electronic device to acquire a second image based on image processing of a first image, acquire a plurality of regions of interest based on the first image, identify a first region of interest among the plurality of regions of interest based on alignment of the plurality of regions of interest, and display a first portion of the first image and a second portion of the second image based on the first region of interest among the plurality of regions of interest through a display.

[0140] In one embodiment, the plurality of regions of interest may be arranged based on the object-specific frequency of one or more objects within the plurality of images stored in the electronic device. The object-specific frequency may indicate the number of times a corresponding object is included within the plurality of images stored in the electronic device.

[0141] In one embodiment, the plurality of regions of interest may be ordered based on the order of the degree of difference in the image processing. The degree of difference in the image processing may represent the difference between a first image portion of the first image corresponding to the region of interest and a second image portion of the second image corresponding to the region of interest.

[0142] In one embodiment, the plurality of regions of interest may be obtained based on differences between the one or more objects and pixels of the first image and pixels of the second image.

[0143] In one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to identify an object category based on a frequency per object and to acquire one or more objects based on the object category. The frequency per object may indicate the number of times a corresponding object is included in a plurality of images stored in the electronic device.

[0144] In embodiments, a method performed by an electronic device may include an operation of obtaining a second image based on image processing of a first image. The method may include an operation of obtaining a plurality of regions of interest (ROIs) based on the first image. The method may include an operation of identifying a first region of interest among the plurality of ROIs based on alignment of the plurality of ROIs. The method may include an operation of displaying a first portion of the first image and a second portion of the second image based on the first region of interest among the plurality of ROIs through a display.

[0145] In embodiments, a method performed by an electronic device is provided. The method may include an operation of obtaining a second image based on image processing of a first image. The method may include an operation of obtaining one or more regions of interest (ROIs) based on one or more objects included in the first image. The method may include an operation of displaying a first portion of the first image and a second portion of the second image based on a first region of interest among the one or more ROIs.

[0146] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, electronic devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0147] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0148] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0149] Various embodiments of the present document may be implemented as software (e.g., a program (140) of FIG. 1) including at least one instruction stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101) of FIG. 1). For example, a processor (e.g., a processor (120) of FIG. 1, a processor (220) of FIG. 2) of a machine (e.g., an electronic device (101) of FIG. 1) may call at least one instruction among the at least one instruction stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one instruction called. The at least one instruction may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0150] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0151] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the multiple entities may be separately arranged in other components. According to various embodiments, at least one of the components or operations of the aforementioned components may be omitted, or at least one other component or operation may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In this case, the integrated component may perform at least one function of each of the plurality of components identically or similarly to that performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or at least one other operation may be added.

Claims

1. In electronic devices, Memory that stores instructions; at least one processor; and Including a display, The above instructions, when executed by the at least one processor, cause the electronic device to: A second image is obtained based on image processing of the first image, Based on the first image above, multiple regions of interest (ROIs) are acquired, Based on the alignment of the above multiple regions of interest, a first region of interest is identified among the multiple regions of interest, Causing the display to display a first portion of the first image and a second portion of the second image based on the first region of interest among the plurality of regions of interest. Electronic devices.

2. In claim 1, the instructions, when executed by the at least one processor, cause the electronic device to: Obtaining at least one object included in the first image, Causing to acquire the plurality of regions of interest based on at least one object acquired above, Electronic devices.

3. In claim 2, The above plurality of regions of interest are arranged based on the object-specific frequency of the at least one object within the plurality of images stored in the electronic device, The frequency of the above object indicates the number of times the object is included in a plurality of images stored in the electronic device. Electronic devices.

4. In claim 2, the instructions, when executed by the at least one processor, cause the electronic device to: Identify object categories based on object-specific frequency, Causing to obtain at least one object based on the above object category, The frequency of the above object indicates the number of times the object is included in a plurality of images stored in the electronic device. Electronic devices.

5. In claim 1, The above multiple regions of interest are sorted based on the order of the degree of difference in the image processing, The degree of difference in the image processing refers to the difference between a first image portion associated with the region of interest in the first image and a second image portion associated with the region of interest in the second image. Electronic devices.

6. In claim 1, The image processing comprises adjusting at least one of brightness, color, or resolution of the first image, The above multiple regions of interest are obtained based on the positions of pixels in which at least one of the brightness, the color, or the resolution is adjusted according to the image processing. Electronic devices.

7. In claim 1, The above multiple regions of interest are obtained based on the difference between pixels of the first image and pixels of the second image. Electronic devices.

8. In claim 7, the instructions, when executed by the at least one processor, cause the electronic device to: Obtaining a first candidate region of interest based on the difference between the pixels of the first image and the pixels of the second image, Obtaining a second candidate region of interest associated with at least one object included in the first image, causing an area of ​​interest to be obtained that includes both the first candidate area of ​​interest and the second candidate area of ​​interest; Electronic devices.

9. In claim 1, The above multiple regions of interest are obtained based on saliency detection to separate at least one object included in the first image from the background region of the first image. Electronic devices.

10. In claim 1, The above first image is related to the P (predicted) frame among the plurality of frames, The above multiple regions of interest are identified based on data included in the P frame. Electronic devices.

11. In claim 1, The instructions, when executed by the at least one processor, cause the electronic device to display, through the display, a dividing line for distinguishing the first portion of the first image and the second portion of the second image, and a visual object for controlling the dividing line; The above dividing line is displayed across the first region of interest among the plurality of regions of interest, The above visual object is displayed on the above dividing line, Electronic devices.

12. In claim 1, The above instructions, when executed by the at least one processor, cause the electronic device to: Displaying a visual object for each region of interest among the above multiple regions of interest through the display, Causing to display said first region of interest through said display based on a user input on a visual object for said first region of interest; Electronic devices.

13. In claim 1, the instructions, when executed by the at least one processor, cause the electronic device to: Based on identifying that the width-height ratio of the display area of ​​the above display has changed, an image area according to the changed width-height ratio and a second area of ​​interest related to the first area of ​​interest are obtained, wherein the image area represents the entire area for the first image and the second image, If the second region of interest is included in the image area, the image area is displayed through the display, If the second region of interest is not entirely included in the image area, causing the second region of interest to be displayed through the display, Electronic devices.

14. In a non-transitory computer-readable medium, the non-transitory computer-readable medium includes a memory storing instructions, The above instructions, when executed by at least one processor, cause the electronic device to: A second image is obtained based on image processing of the first image, Acquire multiple regions of interest based on the first image above, Based on the alignment of the above multiple regions of interest, a first region of interest is identified among the multiple regions of interest, A device configured to display a first portion of the first image and a second portion of the second image based on the first region of interest among the plurality of regions of interest, Non-transitory computer-readable medium.

15. In a method performed by an electronic device, An operation of obtaining a second image based on image processing of the first image, and An operation of obtaining multiple regions of interest (ROIs) based on the first image, An operation of identifying a first region of interest among the plurality of regions of interest based on the alignment of the plurality of regions of interest; Including an operation of displaying a first part of the first image and a second part of the second image based on the first region of interest among the plurality of regions of interest. method.

Images