Electronic device including transparent display, and operation method thereof

Abstract

Disclosed are an electronic device including a transparent display and an operation method thereof. An operation method of an electronic device according to an embodiment may comprise an operation of acquiring an image including at least one object by using a camera. The operation method may comprise an operation of acquiring, by using a sensor, information on an angle between a first housing part in which a first display is disposed and a second housing part in which a second display is disposed. The operation method may comprise an operation of generating, on the basis of depth information, a user interface (UI) related to an application being executed in the electronic device. The operation method may comprise an operation of displaying the user interface on the second display. In addition, various other embodiments may be possible.

Description

Electronic device including transparent display and method of operating same

[0001] Embodiments of the present invention relate to an electronic device including a transparent display and a method of operating the same.

[0002] A transparent display is a display that can transmit light incident from the rear while simultaneously displaying graphic images. Transparent displays can be used to implement augmented reality (AR). For example, AR can be implemented by seamlessly overlaying digital content on top of a real-world environment through a transparent display. AR is a technology that provides users with augmented reality images that provide additional information by synthesizing and combining virtual objects or objects based on the real world.

[0003] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0004] A foldable electronic device according to one embodiment may include a foldable housing including a first housing portion and a second housing portion, a first display disposed on a first surface of the first housing portion, a second display disposed on a second surface of the second housing portion, a camera disposed on a third surface of the second housing portion opposite the second surface, a sensor, at least one processor including processing circuitry, and a memory storing instructions.

[0005] The above instructions, when individually or collectively executed by the at least one processor, may cause the foldable electronic device to obtain an image including at least one object using the camera.

[0006] The above instructions, when individually or collectively executed by the at least one processor, may cause the foldable electronic device to obtain information about an angle between the first housing portion and the second housing portion using the sensor.

[0007] The above instructions, when individually or collectively executed by the at least one processor, may cause the foldable electronic device to generate depth information based on the depth information, at least a portion of the image, and information about the angle.

[0008] The above instructions, when individually or collectively executed by the at least one processor, may cause the foldable electronic device to generate a user interface (UI) for an application running on the foldable electronic device based on the depth information.

[0009] The above instructions, when individually or collectively executed by the at least one processor, may cause the foldable electronic device to display the user interface on the second display.

[0010] A method of operating a foldable electronic device according to one embodiment may include an operation of acquiring an image including at least one object using a camera.

[0011] The above operating method may include an operation of obtaining information about an angle between a first housing part in which a first display is arranged and a second housing part in which a second display is arranged, using a sensor.

[0012] The above method of operation may include an operation of generating depth information based on at least a portion of the image and information about the angle.

[0013] The above operating method may include an operation of generating a user interface for an application running on the foldable electronic device based on the depth information.

[0014] The above method of operation may include an operation of displaying the user interface on the second display.

[0015] According to one embodiment, a computer-readable recording medium storing one or more computer programs may include instructions for performing the method in a processor.

[0016] A foldable electronic device according to one embodiment may include a foldable housing including a first housing portion and a second housing portion, a flexible display including a first portion supported by the first housing portion and a second portion supported by a second surface of the second housing portion, an image sensor disposed on a third surface of the second housing portion opposite the second surface, at least one processor including a sensor and processing circuitry, and a memory storing instructions.

[0017] The above instructions, when individually or collectively executed by the at least one processor, may cause the foldable electronic device to acquire an image including at least one object using the image sensor.

[0018] The above instructions, when individually or collectively executed by the at least one processor, may cause the foldable electronic device to obtain information about an angle between the first housing portion and the second housing portion using the sensor.

[0019] The above instructions, when individually or collectively executed by the at least one processor, may cause the foldable electronic device to generate depth information based on at least a portion of the image and information about the angle.

[0020] The instructions, when individually or collectively executed by the at least one processor, may cause the foldable electronic device to generate a user interface for an application running on the foldable electronic device based on the depth information.

[0021] The above instructions, when individually or collectively executed by the at least one processor, may cause the foldable electronic device to display the user interface on the second portion using the flexible display.

[0022] In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0023] FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment.

[0024] FIG. 2 is a diagram for explaining a generative artificial intelligence system according to one embodiment.

[0025] FIGS. 3A to 3E are drawings for explaining an operation method of an electronic device including a transparent display according to one embodiment.

[0026] FIG. 4 illustrates a system for generating a user interface according to one embodiment.

[0027] FIG. 5A and FIG. 5B are drawings for explaining a user interface according to one embodiment.

[0028] FIG. 6A and FIG. 6B are drawings for explaining depth information according to one embodiment.

[0029] FIG. 7 illustrates an example of a user interface according to one embodiment.

[0030] FIG. 8 is a diagram for explaining user viewpoint information according to one embodiment.

[0031] FIGS. 9A and 9B are drawings illustrating examples of electronic devices including a transparent display according to one embodiment.

[0032] FIGS. 10A to 10C are drawings for explaining an operating method of an electronic device according to one embodiment.

[0033] Fig. 11 is a flowchart for explaining an operating method of an electronic device according to one embodiment.

[0034] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are assigned the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted.

[0035] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100), according to one embodiment.

[0036] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0037] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may be implemented as a circuit (e.g., a processing circuit) such as a system on chip (SoC) or an integrated circuit (IC). The processor (120) may include one or more processors. For example, the processor (120) may include a combination of one or more processors, such as a CPU, a GPU, an MPU, an AP, and a CP. Instructions stored in the memory (130) may be executed by one processor to cause the electronic device (101) to perform and / or control the operations of the electronic device (101) to be described with reference to FIGS. 2 to 11. Instructions stored in the memory (130) may be executed by multiple processors to cause the electronic device (101) to perform and / or control the operations of the electronic device (101) to be described with reference to FIGS. 2 to 11.

[0038] According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0039] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0040] The memory (130) can store various data used by at least one component (e.g., the processor (120) or the sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., the program (140)) and input data or output data for commands related thereto. The memory (130) can include a volatile memory (132) or a non-volatile memory (134). According to one embodiment, the memory (130) can include one or more memories. The instructions stored in the memory (130) can be stored in one memory. The instructions stored in the memory (130) can be divided and stored in a plurality of memories. The instructions stored in the memory (130) can be executed by the processor (120) to cause the electronic device (101) to perform and / or control the operations of the electronic device (101) to be described with reference to FIGS. 2 to 11.

[0041] According to one embodiment, the instructions stored in the memory (130) may cause the electronic device (101) to perform one or more operations when individually or collectively executed by at least one processor (e.g., the main processor (121) and / or the auxiliary processor (123)). For example, the instructions stored in the memory (130) may be executed by one processor (e.g., the main processor (121) or an auxiliary processor (123) such as a communication processor) or by a plurality of processors operating cooperatively (e.g., the main processor (121) and the auxiliary processor (123)).

[0042] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0043] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0044] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0045] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0046] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0047] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0048] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0049] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0050] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0051] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0052] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0053] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0054] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0055] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0056] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0057] In one embodiment, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0058] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0059] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0060] An electronic device according to an embodiment disclosed in this document may take various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. The electronic device according to an embodiment of this document is not limited to the aforementioned devices.

[0061] The embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0062] The term "module" used in the embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0063] One embodiment of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0064] According to one embodiment, the method according to one embodiment disclosed in the present document may be provided as a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0065] According to one embodiment, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and arranged in other components. According to one embodiment, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to one embodiment, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0066] FIG. 2 is a diagram for explaining a generative artificial intelligence system according to one embodiment.

[0067] Referring to FIG. 2, according to one embodiment, the generative artificial intelligence system (200) may be a program (e.g., a software module) implemented on an electronic device (e.g., an electronic device (101) of FIG. 1) and / or a server (e.g., a server (108) of FIG. 1).

[0068] According to one embodiment, a User Query / Response Interface (210) can receive user input. The user input can be input in a type (or modality) such as natural language, image, audio, and / or video. Additionally, context information can be transmitted together when the user input is transmitted. The context information can include various side information related to the time when the user input is input to the artificial intelligence system (200). For example, the context information can include information about the application currently being used by the user or information about the user's location. Additionally, the user input can be a mixed type of input of the above-described natural language, image, audio, video, and / or context information. Additionally, the user input can include non-natural language input, such as selecting a menu.

[0069] In one embodiment, the user query / response interface (210) may provide the user with output from the generative artificial intelligence system. The output may include a natural language-based response and / or specific content. The output may also include an action requested by the user.

[0070] In one embodiment, an AI framework (220) may receive user input. Based on the user input (e.g., a user's query), the AI ​​framework (220) may coordinate and / or control one or more components necessary to perform an action corresponding to the user's intent.

[0071] According to one embodiment, user input received from the user query / response interface (210) may be transmitted to a prompt design component (221). The prompt design component (221) may be used to generate a prompt suitable as input to a generative model (e.g., a large language model (LLM) and / or a large multimodal model (LMM)) based on the user input.

[0072] In one embodiment, the prompt design component (221) may be an AI component that utilizes a machine learning algorithm or a neural network. The prompt design component (221) may generate improved prompts over time through learning. The prompt design component (221) may access a knowledge repository (230) to generate prompts based on user input. The knowledge repository (230) may include user preference data, a prompt library, and / or prompt examples. The prompt design component (223) may provide the generated prompts to a generative model (e.g., an LLM and / or an LMM).

[0073] According to one embodiment, the APIs / Plugins management component (223) can communicate with an external information source based on a request for additional information when user input is transmitted to the generative model.

[0074] According to one embodiment, the APIs / Plugins management component (223) can establish a communication channel for communication with the outside of the system (200) via the API. The APIs / Plugins management component (223) can enable access to various data sources via the communication channel. The acquired information can be used to generate prompts by the prompt design component (221) together with user input, or can be used as input for the input of the generative model (250).

[0075] According to one embodiment, the APIs / Plugins management component (223) may request a final action via an API when the final action corresponding to user input, rather than an intermediate action, must be performed by an application or service.

[0076] In one embodiment, the refiner component (225) can fine-tune the output of the generative model (250). For example, the refiner component (225) can determine the relevance (e.g., a score) between the output (e.g., content) of the generative model and the user input. For example, the refiner component (225) can determine whether the output contains biased information (e.g., selective information). For example, the refiner component (225) can determine whether the output contains harmful information (e.g., violent content or profanity).

[0077] In one embodiment, the refinement component (225) may determine the degree of matching (e.g., a score) between the output of the generative model (250) and the user input (e.g., the intent of the user input). If the refinement component (225) determines that the output of the generative model (250) does not correspond to the user input, the refinement component (225) may modify the output to correspond to the user input.

[0078] In one embodiment, the refinement component (225) may provide hints to the user (e.g., hints for prompt generation) to enable the user to obtain information that matches the user's intent from the generative model (250).

[0079] According to one embodiment, a generative model (250) may refer to an artificial intelligence neural network that generates new data (e.g., text, images, audio, or video) based on user input (e.g., user utterance). The generative model (250) may include an image generation model and / or a language generation model.

[0080] In one embodiment, the image generation model may include a generative adversarial network (GAN) and / or a variational autoencoder (VAE). An example of an image generation model is a diffusion-based generative model having the structure of a VAE and a transformer.

[0081] In one embodiment, a language generation model (e.g., ChatGPT) may be a model trained to generate statistically most appropriate output based on input. The language generation model may include an LMM. The LMM can identify various types of input, such as text, images, audio (e.g., speech), and / or video, and generate new data corresponding to the input.

[0082] FIGS. 3A to 3E are drawings for explaining an operation method of an electronic device including a transparent display according to one embodiment.

[0083] FIG. 3A is a drawing for explaining an operation method of an electronic device including a transparent display according to one embodiment.

[0084] Referring to FIG. 3A, according to one embodiment, an electronic device (301) (e.g., the electronic device (101) of FIG. 1) may include at least one transparent display. A transparent display is a display that can transmit light incident from a back side and simultaneously display a graphic image. The transparent display can display a graphic image while allowing the back side of the display panel to be visible by alternately turning on or off light-emitting elements (e.g., light-emitting elements such as light-emitting diodes (LEDs)) included in the display panel. A user (e.g., a user of the electronic device (301)) can view the graphic image displayed through the transparent display together with an object (e.g., an object such as a chest of drawers) located behind the transparent display. The electronic device (301) may include a foldable electronic device. The electronic device (301) may include a foldable housing including a first housing portion and a second housing portion. The first housing portion and the second housing portion may be foldably coupled to each other via at least one hinge device (e.g., a hinge module or a hinge structure). The electronic device (301) may include a first display (310) (e.g., a display module (160) of FIG. 1 ) disposed on a first surface of the first housing portion and a second display (320) (e.g., a display module (160)) disposed on a second surface of the second housing portion. The first display (310) may be a display on which a user primarily performs a task using the electronic device (301). The first display (310) may include a touch panel capable of receiving a user's input (e.g., a touch input).The first display (310) may include a transparent display and / or a general display that is not a transparent display. The second display (320) may include a transparent display. The second display (320) may be a transparent display. The second display (320) may be a transparent display that includes a touch panel on the front side and / or the rear side. The electronic device (301) may include a flexible display. The flexible display of the electronic device (301) may include a first part supported by the first housing part and a second part supported by the second side of the second housing part. The first part and the second part may be divided based on the folding axis so that different screens may be displayed within the flexible display.

[0085] According to one embodiment, the electronic device (301) can obtain an image including at least one object using a camera of the second display (320) (e.g., the camera module (180) of FIG. 1). The camera may include a camera (e.g., a first camera) disposed on a third side of the second housing portion. The third side of the second housing portion may be opposite the second side of the second housing portion. The camera (e.g., the first camera) may include a camera disposed on an opposite side of the second side of the second housing portion where the second display (320) is disposed. For example, the first camera may include a camera disposed on the rear side of the second display (320). The electronic device (301) can obtain an image of an actual space in which the electronic device (301) is placed as viewed from the rear side of the second display (320) using the first camera. For example, the image may include an image of an actual space including an object (330). The electronic device (301) can obtain information about the angle between the first housing portion and the second housing portion using a sensor. The angle between the first housing portion and the second housing portion may correspond to the angle between the first display (310) and the second display (320). The electronic device (301) can generate depth information of an image (e.g., an image obtained using the first camera) based on the information about the angle between the first housing portion and the second housing portion. Depth information is data indicating how far each pixel of the image is from the camera or sensor, and can be used to determine the position of an object included in a two-dimensional image in a three-dimensional space. The electronic device (301) can generate depth information about the actual space in which the electronic device (301) is placed. The electronic device (301) can input an image into a model (e.g., the generative model (250) of FIG. 2) to generate depth information of the image.The electronic device (301) can generate a user interface (UI) (370) for an application (e.g., application (350)) running on the electronic device (301) based on depth information. For example, the user interface (370) may be generated based on functions of the application (350) and data about the application (350). The user interface (370) may include a graphical user interface (GUI) for the application. The electronic device (301) can generate the user interface (370) using a model (e.g., a generative model (250) of FIG. 2). The model (e.g., a generative model (250) of FIG. 2) may be trained to generate a user interface based on depth information and information about the application. The electronic device (301) can display the user interface (370) on the second display (320). The electronic device (301) can display a user interface (370) on a second portion of the flexible display. The electronic device (301) can implement an augmentation effect by displaying the user interface (370) on the second display (320) based on an image (e.g., an image including at least one object) acquired using depth information and a camera (e.g., a first camera). The image can include at least one object. The electronic device (301) can generate the user interface (370) based on at least a portion of the characteristics of at least one object included in the image. The characteristics of the at least one object can include color, brightness, position, or material. The electronic device (301) can obtain information about the actual space in which the electronic device (301) is placed from the image acquired using a camera (e.g., a first camera).For example, the electronic device (301) can obtain characteristics (e.g., information such as color, brightness, location, material, size, shape, location, and arrangement) of an object (e.g., an object such as furniture, a wall, or an object) placed in a real space where the electronic device (301) is placed from an image obtained using the first camera. The electronic device (301) can determine a location on the second display (320) where the user interface (370) is to be displayed based on at least a portion of the characteristics of at least one object. For example, the electronic device (301) can determine a location where the user interface (370) can be combined with the real space to implement an augmentation effect based on information about the characteristics of an object placed in a real space obtained using the first camera. The electronic device (301) can display the user interface (370) on the second display (320). The electronic device (301) can implement an augmentation effect by displaying the user interface (370) on the second display (320) based on information about the actual space acquired using depth information and a camera (e.g., the first camera). The electronic device (301) can provide an augmentation effect to the user interface (370) so that the user interface (370) appears to exist in the actual space. For example, the electronic device (301) can provide an augmentation effect so that the user interface (370) appears to be placed on a desk. The electronic device (301) can generate the user interface (370) in the form of a two-dimensional image or a three-dimensional stereoscopic image. The electronic device (301) can display the user interface (370) with a constant augmentation effect even if the angle between the first display (310) and the second display (320) and / or the user's viewpoint changes. The operation of the electronic device (301) to display the user interface (370) when the angle between the first display (310) and the second display (320) and / or the user's viewpoint changes will be described in detail with reference to FIG. 3b.

[0086] According to one embodiment, the electronic device (301) may include at least one transparent display. In the case of a device that only includes a general display and not a transparent display, the background is blurred to focus the gaze on a specific object, and the specific object is clearly displayed. The electronic device (301) can effectively provide an augmented reality (AR) service to a user by generating and displaying a user interface that takes into account an image of an actual space that is transmitted through the transparent display. The user can also perform tasks that take into account the actual environment shown through the transparent display by using the user interface provided by the electronic device (301).

[0087] According to one embodiment, the electronic device (301) can be used in a video see-through (VST) environment. The VST environment can include a digital environment that extends its functionality by connecting virtual tools and virtual effects to a host program using a user interface. The VST environment can include a virtual reality (VR) environment and / or an augmented reality (AR) environment. The electronic device (301) can obtain information about a real space, create a VST environment, and provide it to a user by using a camera (e.g., a first camera) that is positioned opposite a surface on which a second display (320), which is a transparent display, is positioned or is positioned on the rear of the second display (320). The first camera positioned opposite or on the rear of the second display (320) can include a plurality of cameras of different types. For example, the first camera can include a camera with a general field of view (FoV) and a wide-angle camera. The electronic device (301) can obtain an image including at least one object using a camera (e.g., a first camera). The electronic device (301) can obtain information about a space where a user is actually located using the first camera. The electronic device (301) can render a virtual space and provide it to the user. For example, the electronic device (301) can render a virtual space by reflecting information about a space where a user is actually located and provide it. For example, the electronic device (301) can render a virtual space and provide it to a VST environment regardless of the space where the user is actually located. The electronic device (301) can obtain information about the user's gaze using another camera (e.g., a second camera) located on the same surface as the surface on which the second display (320) is arranged or in front of the second display.The electronic device (301) can obtain an image including the user's eyes using the second camera. The electronic device (301) can obtain the user's viewpoint information, movement, and / or location from the image obtained using the second camera. In response to generating and providing a virtual environment on the second display (320), the electronic device (301) can obtain the user's viewpoint information, movement, and / or location using the second camera. The electronic device (301) can provide a different virtual environment in response to a change in the user's viewpoint information and / or location beyond a predetermined range. For example, if the user moves beyond a predetermined range in the lateral direction of the electronic device (301) while displaying a virtual environment generated based on information about a space in which the user is actually located (e.g., an image obtained using the first camera) on the second display (320), the electronic device (301) can generate an area not included in the image capturing the actual space using a model (e.g., the generative model (250) of FIG. 2) and provide the virtual environment. If the electronic device (301) determines that the user has moved more than a predetermined distance and that the virtual environment cannot be provided using only the information about the actual space acquired using the first camera, the electronic device (301) may provide an area generated using a model (e.g., the generative model (250) of FIG. 2) as a virtual environment. The electronic device (301) may determine whether the user's viewpoint information and / or location has changed beyond a predetermined range based on whether the change value of the user's viewpoint information acquired using the second camera has exceeded a predetermined threshold. In response to the user's viewpoint information and / or location having changed beyond a predetermined range, the electronic device (301) may determine a different camera to use among a plurality of cameras included in the first camera.For example, if the electronic device (301) determines that the user has moved the electronic device (301) in a lateral direction of a predetermined range and the change in the user's viewpoint information has exceeded a predetermined threshold, the electronic device (301) may change the first camera from a camera with a normal field of view to a wide-angle camera to obtain a wider range of images of the space where the user is actually located. The electronic device (301) may generate a virtual environment by reflecting the acquired wider range of images and display it on the second display (320) to provide a VST environment to the user. The electronic device (301) may generate a virtual space based on information about the actual space and information about the application the user was working on. For example, if the user is working on document editing in a space with high brightness and wants to continue working in a VST environment, the electronic device (301) may generate and provide a virtual space with low brightness to facilitate document editing. In response to receiving a user's input for using the running application in the VST environment, the electronic device (301) may generate a user interface suitable for the VST environment. The electronic device (301) can generate a user interface suitable for a VST environment using a generative model (e.g., the generative model (250) of FIG. 2). For example, the electronic device (301) can input information about the VST environment and information about a running application into the generative model (250) to obtain a user interface suitable for the VST environment. The electronic device (301) can obtain a template for generating a user interface from a server (e.g., the server (108) of FIG. 1). For example, the electronic device (301) can obtain a user interface template created by another user and uploaded to the server (108) to generate a user interface.

[0088] FIG. 3b is a drawing for explaining a user interface when the angle between displays is changed according to one embodiment.

[0089] Referring to FIG. 3B, according to one embodiment, the angle between the first display (310) and the second display (320) may be changed. For example, as the angle between the first housing portion where the first display (310) is disposed and the second housing portion where the second display (320) is disposed is changed, the angle between the first display (310) and the second display (320) may be changed. As each housing portion rotates about the axis of at least one hinge device that connects the first housing portion and the second housing portion, the angle between the first housing portion and the second housing portion may be changed. The electronic device (301) may obtain information about the angle between the first housing portion and the second housing portion using a sensor (e.g., the sensor module (176) of FIG. 1). For example, the electronic device (301) can obtain information about the folding angle between the first housing portion and the second housing portion by using a sensor included in the foldable housing (e.g., a sensor such as an acceleration sensor or a gyroscope). The electronic device (301) can further include another camera (e.g., a second camera) disposed on a second surface of the second housing portion. The second camera can include a camera disposed on the front of the second display (320). The electronic device (301) can obtain another image including the user's eye through the other camera (e.g., the second camera). The electronic device (301) can identify the position of the user's eye within the other image. The electronic device (301) can obtain the user's point of view based on the position of the user's eye. The electronic device (301) can modify the user interface (370) in real time based on the position of the user's eye. The electronic device (301) can reflect the user's viewpoint information in real time on the user interface (370).The electronic device (301) can display the modified user interface on the second display (320). The electronic device (301) can display the user interface (370) so that it can be consistently visible from the user's viewpoint even when the angle between the first display (310) and the second display (320) and / or the position (e.g., viewpoint) of the user's eyes changes. For example, the electronic device (301) can display the user interface (370) generated in a three-dimensional form by rotating it in a corresponding direction (e.g., upward direction, downward direction) as the angle between the first display (310) and the second display (320) and / or the user's viewpoint changes, thereby displaying the user interface (370) so that it can be consistently visible to the user. For example, the electronic device (301) can update the user interface (370) generated in a two-dimensional form as the angle between the first display (310) and the second display (320) and / or the user's viewpoint changes, so that the user interface (370) can be displayed so that it appears consistently to the user. When the electronic device (301) generates the user interface (370) in a two-dimensional form, the electronic device (301) can generate user interface images that are viewed from various angles in advance. When the electronic device (301) generates the user interface (370) in a two-dimensional form, the electronic device (301) can detect a change in the angle between the first display (310) and the second display (320) and / or the user's viewpoint, and generate and display the user interface (370) in real time.

[0090] FIG. 3c is a drawing for explaining a user interface according to one embodiment.

[0091] Referring to FIG. 3C, according to one embodiment, the electronic device (301) can display a user interface (370) on the second display (320). The electronic device (301) can display the user interface (370) on the second portion of the flexible display. The electronic device (301) can generate the user interface (370) based on information about an application running on the electronic device (301) (e.g., application (350) of FIG. 3A). For example, when Paint, a program for creating and / or editing images, is running on the electronic device (301), the electronic device (301) can generate a palette user interface associated with the function of Paint. The electronic device (301) may not display the user interface displayed on the first display (310) in response to determining that the user interface displayed on the first display (310) and the user interface (e.g., the user interface (370)) displayed on the second display (320) overlap. For example, the electronic device (301) may display the user interface only on the second display (320) in response to determining that the user interface for one function overlaps on the first display (310) and the second display (320). By displaying the user interface to be displayed on the first display (310) on the second display (320), the electronic device (301) may enable the user to use a wide area of ​​the first display (310) to perform a task. The electronic device (301) may acquire an image using a camera (e.g., the first camera) of the second display (320). The first camera may include a camera positioned on the back of the second display (320). The electronic device (301) may input an image acquired using the first camera into a model to acquire depth information.The electronic device (301) can generate further depth information using an image acquired from a camera (e.g., a first camera), and can modify further depth information based on information about an angle (e.g., information about an angle between the first housing portion and the second housing portion) to generate depth information. The depth information may be obtained by inputting an image acquired using a camera (e.g., a first camera) into a model and correcting the acquired depth information based on the angle between the first housing portion and the second housing portion. The electronic device (301) can display a user interface (370) on the second display (320). The electronic device (301) can display the user interface (370) on the second display (320) based on the depth information. A user can use a function of an application (e.g., a drawing program) through the user interface (370) displayed on the second display (320). The electronic device (301) can execute a function corresponding to the user interface (370) in response to a user input (370) for the user interface (370). For example, in response to a user input (360) for the palette user interface, the electronic device (301) can execute a pen function of a color selected by the user. The user can perform a task using the function selected by the user on the first display (310), which is the main work area. For example, the user can draw a picture on the first display (310) using the pen function of the selected color.

[0092] FIG. 3D is a drawing for explaining the operation of an electronic device in an unfolded state according to one embodiment.

[0093] Referring to FIG. 3D , according to one embodiment, when a change in the angle between the first display (310) and the second display (320) is detected, the electronic device (301) can correct the depth information based on the changed angle. The electronic device (301) can input an image acquired using the first camera of the second display (320) into a model to acquire depth information, and when a change in the angle between the first display (310) and the second display (320) is detected, the electronic device (301) can correct the depth information. When the angle between the first display (310) and the second display (320) changes within a defined error range, the electronic device (301) may not perform correction on the depth information. For example, when the angle between the first display (310) and the second display (320) changes within an error range that does not affect the depth information, the electronic device (301) may not correct the depth information. The electronic device (301) can display a user interface (370) on the second display (320) without considering depth information when the second display (320) is closer to the floor than a predetermined distance. The electronic device (301) can display the user interface (370) by changing it into a two-dimensional form when the second display (320) is closer to the floor than a predetermined distance. For example, when the electronic device (301) is almost completely unfolded and the second display (320) is closer to the floor than a predetermined distance, the user interface (370) can be changed into a two-dimensional form and displayed as a user interface (380).

[0094] FIG. 3e is a drawing for explaining the operation of an electronic device according to one embodiment in a fully folded state.

[0095] Referring to FIG. 3E, according to one embodiment, the electronic device (301) may refrain from generating a user interface (e.g., user interface 370) and displaying the user interface 370 on the second display (320) based on at least a portion of determining that the electronic device (301) is in a fully folded state. The electronic device (301) may determine the state of the electronic device (301) based on an angle between the first housing portion and the second housing portion obtained using a sensor. The state of the electronic device (301) may include a fully unfolded state (e.g., an unfolded state or an unfolded state), a fully folded state (e.g., a folded state or a folded state), and an intermediate state. In the fully folded state of the electronic device (301), the first display (310) may be visible through the second display (320), which is a transparent display. The electronic device (301) can run an application (e.g., a drawing program) even when fully folded. The electronic device (301) can display the running application on the first display (310) or the second display (320). The electronic device (301) can use the first display (310) to display another user interface stored in a memory (e.g., the memory (130) of FIG. 1). For example, the electronic device (301) can display a user interface related to the running application on the first display (310). The other user interface may be related to the application running on the electronic device (301). The user can use the application running on the electronic device (301).For example, a user can use an application running on the electronic device (301) through a touch input to the first display (310) and / or the second display (320), or an input using an external device (e.g., a mouse, a keyboard, a tablet, a microphone). The electronic device (301) can detect a motion of the user lifting the second display (320). The electronic device (301) can detect a change in the angle between the first display (310) and the second display (320) and recognize a motion of the user lifting the second display (320) to unfold the electronic device (301). In response to the user unfolding the electronic device (301) by a predetermined angle or more, the electronic device (301) can display an extended user interface on the first display (310) and the second display (320). For example, in response to the angle between the first display (310) and the second display (320) being greater than or equal to a predetermined angle, the electronic device (301) may expand and display a user interface that was previously displayed only on the first display (310) or the second display (320) on both the first display (310) and the second display (320). The electronic device (301) may provide an interface that asks the user whether to expand and display the user interface. For example, in response to the angle between the first display (310) and the second display (320) being greater than or equal to a predetermined angle, the electronic device (301) may display an interface that asks the user whether to expand and display a user interface for a running application on the displays (e.g., the first display (310), the second display (320)). The electronic device (301) may display the user interface based on the user's response.

[0096] FIG. 4 illustrates a system for generating a user interface according to one embodiment.

[0097] Referring to FIG. 4, according to one embodiment, a user interface generation system (400) may generate a user interface to be displayed on an electronic device (e.g., the electronic device (301) of FIG. 3A, the electronic device (101) of FIG. 1). The user interface generation system (400) may receive a first camera input (403), a second camera input (405), and application information (401). The user interface generation system (400) may output a user interface (480).

[0098] According to one embodiment, the user interface generation system (400) may include one or more components (410-475). The components (410-475) may be software modules implemented on an electronic device (e.g., the electronic device (301) of FIG. 3A, the electronic device (101) of FIG. 1) or a server (e.g., the server (108) of FIG. 1). The components (410-475) are illustrated as an example for describing the user interface generation system (400). Accordingly, the user interface generation system (400) may include various variations of the components (410-475) as long as the operations of the user interface generation system (400) described in the present disclosure can be implemented. For example, two or more components may be combined, or one or more components may be added or omitted. Alternatively, the user interface generation system (400) may further include one or more components (e.g., components (210-250) of FIG. 2) of a generative artificial intelligence system (e.g., generative artificial intelligence system (200) of FIG. 2).

[0099] According to one embodiment, the user interface generation system (400) may include a depth estimation module (410), an angle measurement module (420), a depth correction module (430), a point of view (pov) information generation module (440), a generative model (470), and a post-processing module (475).

[0100] According to one embodiment, the depth estimation module (410) may receive a first camera input (403). The first camera input (403) may include an image acquired using a first camera (e.g., the camera module (180) of FIG. 1) of a second display (e.g., the second display (320)). The first camera may include a camera disposed on the rear of the second display (320). The first camera may include a sensor (e.g., a camera) capable of measuring depth for the captured image. For example, the first camera may include a sensor capable of measuring depth, such as a time of light (TOF) camera, a 3D TOF camera, a red-green-blue-depth (RGBD) camera, a depth sensor, an infrared sensor, and a laser. It should be noted that the types of sensors that the first camera may be configured to measure depth are not limited to the examples described above. The depth estimation module (410) can estimate depth information for an image acquired using the first camera of the second display (320). The electronic device (301) can estimate depth information of an image in response to receiving a user input. The user input may refer to a user input for moving an application (e.g., software including an application program or a system program) displayed on the first display (310) to the second display (320). For example, a gesture input may be an input including a user's intent to move an application displayed on the first display (310) to the second display (320), regardless of the format such as a touch input, a mouse input, a gaze input, a keyboard input, or a voice input. The electronic device (301) can estimate depth information of an image using the depth estimation module (410). The depth estimation module (410) can estimate depth information of an image using a model (e.g., a depth estimation model).The depth estimation model may include an artificial intelligence (AI) model, such as a machine learning model or a deep learning model. The depth estimation model may be a model trained using training data pairs consisting of a single image and depth ground truth data for the single image. For example, the depth estimation model may be a model trained using a single red-green-blue (RGB) image and a single-channel depth ground truth data pair corresponding to the single RGB image. The depth estimation model may be a model pre-trained using the training data pairs. For example, the depth estimation model may be a model pre-trained using a single RGB image and a single-channel depth ground truth data pair corresponding to the single RGB image. The depth estimation model may include multiple convolutional layers. The depth estimation model may estimate depth information in real time for an image (e.g., the first camera input (403)). The depth estimation module (410) can obtain high-accuracy depth information in real time using a pre-learned depth estimation model. The image may include a single RGB (red-green-blue) image. The depth estimation model can estimate and output depth information in real time from the single RGB image. For example, the depth estimation model can estimate and output depth information including a depth image (e.g., a depth image of 1 channel) from the single RGB image in real time. The depth information can be estimated based on the distance between the space located in front of the electronic device (301) and the electronic device (301). When the distance between the electronic device (301) and the space located in front of the electronic device (301) is greater than a predetermined value, the depth estimation module (410) can consider and output depth information only up to a predetermined range.For example, if the electronic device (301) is used outdoors and the distance between the space located in front of the electronic device (301) and the electronic device (301) is very far, the depth estimation module (410) may consider and output depth information only up to a set range. The depth information output by the depth estimation module (410) may include depth information in the form of a depth map. The depth estimation module (410) may transmit the estimated depth information for the image, which is the first camera input (403), to the depth correction module (430).

[0101] According to one embodiment, the angle measurement module (420) can measure an angle between a first display (e.g., the first display (310) of FIG. 3A) and a second display (e.g., the second display (320) of FIG. 3A). The angle measurement module (420) can obtain information about a folding angle between the first housing portion and the second housing portion by using a sensor (e.g., a sensor such as an acceleration sensor or a gyroscope) included in the foldable housing of the electronic device (301). The electronic device (301) can measure the angle between the first display (310) and the second display (320) in response to receiving a user input from a user. The electronic device (301) can measure the angle between the first display (310) and the second display (320) in response to receiving a user input to move an application displayed on the first display (310) to the second display (320). The angle between the first display (310) and the second display (320) may include two angles: an inner angle and an outer angle (e.g., a supplementary angle of the inner angle, angle 650 of FIGS. 6A and 6B ). The angle between the first display (310) and the second display (320) will be described in detail later with reference to FIGS. 6A and 6B . The angle measurement module (420) may obtain information about the angle between the first housing portion and the second housing portion using a sensor included in the foldable housing. The sensor may include an acceleration sensor, a gyroscope, and a sensor associated with a hinge device (e.g., a hinge sensor). The angle measurement module (420) can obtain the angle between the first display (310) and the second display (320) from a sensor (e.g., a hinge sensor) associated with a hinge that connects the first display (310) and the second display (320) and / or a sensor included in the foldable housing.A hinge sensor is a sensor for measuring the relative angle between two objects, and can be attached to a hinge to measure the angle, convert the measured angle into an electrical signal, and process it. The range and accuracy of the angle that the sensor can recognize may vary depending on the characteristics of the electronic device (301). The electronic device (301) can set the range of angles that the angle measurement module (420) can output (e.g., minimum folding angle between displays, maximum folding angle between displays). The electronic device (301) can preset the range of angles that the sensor can recognize (e.g., folding angle). The electronic device (301) can preset the range of angles that the sensor can recognize so that the angle measurement module (420) can output an angle having a value within a certain range. The angle measurement module (420) can transmit the angle between the first display (310) and the second display (320) to the depth correction module (430).

[0102] According to one embodiment, the depth correction module (430) may receive depth information of an image (e.g., first camera input (403)) from the depth estimation module (410). The depth correction module (430) may receive an angle between the first display (310) and the second display (320) from the angle measurement module (420). The depth correction module (430) may modify (e.g., correct) the depth information based on the angle between the first display (310) and the second display (320). The depth correction module (430) may correct the depth information based on the position of the first camera. For example, since the relative position change of the first camera may vary depending on the form in which the electronic device (301) is used (e.g., a form in which it is used horizontally or vertically), the depth correction module (430) may correct the depth information by taking into account the position of the first camera. Since depth information according to the angle between the first display (310) and the second display (320) may vary depending on the position of the first camera on the back of the second display (320) (e.g., vertical position such as top, middle, bottom), the depth correction module (430) may take into account the position of the first camera when correcting the depth information.

[0103] According to one embodiment, the depth correction module (430) may, in response to a detection of a change in the angle between the first display (310) and the second display (320), correct the depth information based on the changed angle. The depth correction module (430) may not perform correction on the depth information if the angle between the first display (310) and the second display (320) changes within a defined error range. For example, the depth correction module (430) may not correct the depth information if the angle between the first display (310) and the second display (320) changes within an error range that does not affect the depth information. If the angle between the first display (310) and the second display (320) changes beyond the defined error range, the depth information of the image, which is the first camera input (403), also changes, and thus the depth information may need to be corrected. For example, if the changed angle increases compared to the initial angle (e.g., if the user further unfolds the second display (320), the electronic device (301) can obtain an image of a space close to the floor surface where the electronic device (301) is located using the first camera. For another example, if the changed angle decreases compared to the initial angle (e.g., if the user further covers the second display (320), the electronic device (301) can obtain an image of a space farther away using the first camera. The initial angle may refer to an angle at a point in time when the electronic device (301) obtains depth information using the depth estimation model (410). If the changed angle increases compared to the initial angle and the second display (320) becomes closer to the floor surface by a predetermined distance or more, the electronic device (301) can display an application on the second display (320) without considering the depth information.The electronic device (301) may provide a notification to the user that the angle has exceeded a defined maximum value (e.g., hinge limits) when the changed angle increases from the initial angle and the second display (320) comes closer to the floor by a defined distance or more. For example, the electronic device (301) may provide a notification to the user that the angle between the first display (310) and the second display (320) has exceeded a defined maximum value and therefore requires angle correction. The operation of the electronic device (301) displaying a user interface when the second display (320) comes closer to the floor by a defined distance or more is substantially the same as that described with reference to FIG. 3D, and any redundant description thereof will be omitted. The hinge limits may be set by the user and may be determined based on the structural characteristics of the electronic device (301). For example, the maximum and / or minimum values ​​of the folding angle may be determined based on the structural characteristics of the electronic device (301). If the changed angle is smaller than the initial angle and depth information for the image cannot be estimated, the electronic device (301) may display the application on the second display (e.g., the second display (320) of FIG. 3A) without considering the depth information. For example, if the distance between the electronic device (301) and the space viewed through the first camera is larger than a predetermined value, the electronic device (301) may display the application on the second display (320) without considering the depth information. If the changed angle is smaller than the initial angle and depth information for the image cannot be estimated, the electronic device (301) may provide a notification to the user that the angle between the first display (310) and the second display (320) is smaller than a predetermined minimum value and therefore angle correction is necessary.

[0104] According to one embodiment, the point-of-view generation module (440) may receive a second camera input (405). The second camera input (405) may include data captured by a second camera of the second display (e.g., the camera module (180)). For example, the second camera input (405) may include another image including the user's eyes. The second camera may include a camera positioned in front of the second display (320). The point-of-view generation module (440) may input the second camera input (405) into a point-of-view estimation model to obtain point-of-view information of the user. The point-of-view information of the user may include information regarding the point of view from which the user is looking. For example, the point-of-view information may include information such as the location of a spot from which the user is looking and the angle from which the user is looking at a specific spot. The point-of-view estimation model may include an artificial intelligence model such as a machine learning model or a deep learning model. The viewpoint estimation model may be a model trained through training data pairs consisting of an image containing a human face (e.g., a face including eyes) and viewpoint correct data. The viewpoint estimation model may receive an image containing a human face and information about a display (e.g., hardware information) and output user viewpoint information. The operation of the viewpoint estimation model to generate user viewpoint information will be described in detail later with reference to FIG. 8. The viewpoint information generation module (440) may output viewpoint information generated using the viewpoint estimation model to the postprocessing module (475).

[0105] According to one embodiment, the generative model (470) (e.g., the generative model (250) of FIG. 2) may receive corrected depth information from the depth correction module (430). The generative model (470) may generate a user interface (UI) for an application running on the electronic device (301) based on the depth information. The generative model (470) may be trained to generate a user interface based on the depth information and the information about the application. The generative model (470) may be a model trained through a training data pair consisting of correct answer data about application information and a function corresponding to the application. The generative model (470) may receive a first camera input (403). The generative model (470) may obtain information about a space projected behind the second display (320) based on the first camera input (403). The electronic device (301) may generate a user interface in response to receiving a user input from a user. The user input may refer to a user input that moves an application (e.g., software including an application program or a system program) displayed on the first display (310) to the second display (320). For example, the user input may be a user input containing an intention to move an application displayed on the first display (310) to the second display (320), such as swiping, moving, or expanding the application displayed on the first display (310) to the second display (320). The user input may be an input containing the user's intention to move an application displayed on the first display (310) to the second display (320), regardless of the input format such as a touch input, a mouse input, a keyboard input, or a voice input. The electronic device (301) may generate a user interface to be displayed on the second display (320) in response to receiving the user input.The electronic device (301) can generate a user interface using the generative model (470). The electronic device (301) can generate a user interface suitable for the second display (320), which is a transparent display. For example, the electronic device (301) can generate a user interface that reflects information about a space behind the second display (320), which is a transparent display, using the generative model (470). The electronic device (301) can generate a prompt using at least one of the angle between the first display (310) and the second display (320), information about an application running on the electronic device (301) (e.g., information such as an icon, function, and basic user interface of the application), an image of the space in front of the electronic device (301) acquired using the first camera of the second display (320), and depth information. The application information (401) can include information about an application running on the electronic device (301) (e.g., software including an application program and a system program). For example, the application information (401) may include information such as the application's icon, function, and basic user interface. The electronic device (301) may generate a prompt for generating a user interface to be displayed together with the real space visible through the second display (320), which is a transparent display. The electronic device (301) may input the prompt into the generative model (470) to generate the user interface. The generative model (470) may generate the basic user interface of the application running on the electronic device (301) by modifying it to be suitable for the transparent display. The generative model (470) may generate a function to be provided together with the user interface based on the function of the application.For example, the generative model (470) can analyze functions that are determined to be supported by the application in addition to the basic functions provided by the application, and provide them together with a user interface. The generative model (470) can create a user interface that can be executed immediately for functions that require complex option selection to be executed. The generative model (470) can increase the user's convenience in using the application by creating new functions to be provided together with the user interface. The generative model (470) can output the user interface to the post-processing module (475).

[0106] According to one embodiment, the post-processing module (475) can receive a user interface from the generative model (470). The post-processing module (475) can receive viewpoint information from the viewpoint information generation module (440). Based on the viewpoint information, the post-processing module (475) can correct (e.g., warp) the user interface and output the user interface (480). The post-processing module (475) can render the user interface differently in real time based on the user's viewpoint information according to the second camera input (405). For example, the post-processing module (475) can pre-render the user interface viewed from various positions and store it in the electronic device (301). Based on the user's viewpoint information, if there is a lost portion of the user interface depending on the viewing position (e.g., if a part of the user interface is not displayed depending on the viewing position and / or angle), the post-processing module (475) can correct the lost portion using the generative model (470). The generative model (470) can compensate for the lost portion by performing out-painting. The electronic device (301) can display the user interface differently in real time on the second display (320) according to viewpoint information through the post-processing module (475). Since the user sees the user interface (480) along with the space behind the second display (320), which is a transparent display, the electronic device (301) can output the user interface (480) that matches the user's intention by displaying the user interface (480) differently in real time on the second display (320) based on the user's viewpoint information.

[0107]

[0108] FIG. 5A and FIG. 5B are drawings for explaining a user interface according to one embodiment.

[0109] According to one embodiment, FIGS. 5A and 5B illustrate an example in which an electronic device (501) (e.g., the electronic device (101) of FIG. 1, the electronic device (301) of FIG. 3A) is used while being placed in a specific space (e.g., a specific space such as on a desk (505) or on a user's lap). Although FIGS. 5A and 5B illustrate the electronic device (501) being used while being placed in a specific space, it should be noted that this is merely an example, and the environment in which the electronic device (501) is used is not limited. For example, the electronic device (501) may also be used while the user is carrying the electronic device (501) and moving around. The electronic device (501) may include a first display (510) disposed on a first surface of a first housing portion (e.g., the first display (310) of FIG. 3A) and a second display (520) disposed on a second surface of a second housing portion (e.g., the second display (320) of FIG. 3B). The first housing portion and the second housing portion may be foldably coupled to each other by at least one hinge device. The first display (310) may include a transparent display and / or a general display that is not a transparent display. The second display (520) may be a transparent display. The electronic device (501) may be used in a form in which the first housing portion is positioned on the bottom. For example, the electronic device (501) may be used in a form in which the first display (510) is positioned on the bottom in a partially folded state. The first display (510) may be a display that a user primarily uses to perform a task. For example, the first display (510) can display an application running on the electronic device (501) to allow the user to perform tasks through the application (e.g., tasks such as image editing, video viewing, web surfing, document writing, and programming).The electronic device (501) can display a running application (e.g., an application running on the electronic device (501) among the applications (550)) on the first display (510).

[0110] According to one embodiment, the second housing portion may include a camera (e.g., a first camera) disposed on the third side and another camera (e.g., a second camera) disposed on the second side. The first camera may include a camera disposed on the rear side of the second display (520). The second camera may include a camera disposed on the front side of the second display (520). The cameras (e.g., the first camera and the second camera) of the second display (520) may include cameras for tasks requiring photography, such as image capturing, video conferencing, and video streaming. The cameras of the second display (520) may include cameras built into the electronic device (501) and / or external cameras separately connected to the electronic device (501). The electronic device (501) may acquire an image using the first camera of the second display (520). The image may be a space located in front of the electronic device (501) captured by the first camera. The electronic device (501) can generate depth information of an image. The depth information may be depth information obtained by inputting an image into a model (e.g., a depth estimation model) and corrected based on the angle between the first display (510) and the second display (520). If there is a change in the angle between the first display (510) and the second display (520), the depth information obtained by inputting an image into a model (e.g., a depth estimation model) may no longer be accurate. Therefore, the electronic device (501) can correct the depth information based on the angle between the first display (510) and the second display (520). The angle between the first display (510) and the second display (520) may be a value obtained and output by an angle measurement module (e.g., an angle measurement module (420) of FIG. 4) from a sensor associated with a hinge.The operation of the electronic device (501) to generate depth information for an image acquired using the first camera is substantially the same as the operation of the depth estimation module (410) and the depth correction module (430) described with reference to FIG. 4, and any overlapping description thereof will be omitted.

[0111] According to one embodiment, the electronic device (501) may generate a user interface (e.g., user interface (570), user interface (480) of FIG. 4) in response to receiving user input (e.g., gesture input (530)) from a user. The user input may refer to a user input that moves an application (e.g., software including an application program or a system program) displayed on the first display (510) to the second display (520). For example, the gesture input (530) may be a user input that moves an application displayed on the first display (510) to the second display (520), such as swiping, moving, or expanding the application displayed on the first display (510) to the second display (520). The user input may be an input that includes the user's intention to move the application displayed on the first display (510) to the second display (520), regardless of the input format such as touch input, mouse input, keyboard input, or voice input. The electronic device (501) may generate a user interface (e.g., user interface (570), user interface (480) of FIG. 4) regarding the application running on the electronic device (501). The electronic device (501) can input information about a running application (e.g., application information (401)), depth information corrected based on an angle between the first display (510) and the second display (520) (e.g., output of a depth correction module (430)), information about an application displayed on the first display (310) (e.g., information such as an icon, function, and menu information of the application), and an image acquired using a first camera of the second display (520) (e.g., first camera input (403)) into a generative model (e.g., generative model (250) of FIG. 2, generative model (470) of FIG. 4) to generate a user interface (570).The electronic device (501) may generate a user interface (570) based on information about the running application. For example, when a user uses Paint, a program for creating and / or editing images, the electronic device (501) may display a user interface (e.g., user interface (570)) in the form of a drawing tool in response to receiving a user input (e.g., gesture input (530)) from the user to move the Paint program displayed on the first display (510) to the second display (520). For example, when a program for editing a document (e.g., a program such as Word) is running on the electronic device (501), the electronic device (501) may display a user interface in the form of a blank sheet of paper, thereby allowing the user to conveniently create a document. The user interface (570) may include a user interface regarding a function provided by the application running on the electronic device (501) and / or a user interface regarding a function that is analyzed to be supported but not provided by default. The electronic device (501) can generate functions and / or user interfaces that are not provided by default in the application but are analyzed to be supported by using a generative model (e.g., the generative model (250) of FIG. 2, the generative model (470) of FIG. 4). For example, the electronic device (501) can generate user interfaces such as a palette user interface (Palette UI) for selecting colors and a painting tool selecting UI based on the functions of a drawing program (e.g., an image creation function, an image editing function).For example, even if the electronic device (501) does not provide the basic functions of the drawing program, it can also provide additional functions by generating a user interface in a form that matches the drawing program (e.g., an easel-type user interface, a water container-type user interface). The user interface (570) can include a user interface that can directly execute a function that requires complex option selection to be executed. An application executable on the electronic device (501) (e.g., the application (550)) can include components for a user interface to be displayed on a transparent display. Since the components for the user interface included in the application (550) itself may be limited due to issues such as the capacity of the application, the electronic device (501) can generate a user interface (570) to be displayed on the second display (520) by using the generative model (470) without using the user interface components included in the application itself.

[0112] According to one embodiment, the electronic device (501) may display a user interface (570) differently on the second display (520) based on the user's viewpoint information. The electronic device (501) may obtain the user's viewpoint information using a second camera of the second display (520). The second camera may include a camera positioned in front of the second display (520). The electronic device (501) may input an image of the user (e.g., the user's face) captured using the second camera into an artificial intelligence model (e.g., a viewpoint estimation model). The viewpoint estimation model may include an artificial intelligence model such as a machine learning model or a deep learning model. The electronic device (501) may use the viewpoint estimation model to find an eye from an image of the user's face and obtain the user's gaze information. For example, the electronic device (501) can detect major facial parts such as eyes, nose, mouth, and ears from an image of the user's face using a viewpoint estimation model, and track the user's eyes (e.g., pupils) to obtain gaze information. The electronic device (501) can perform correction (e.g., warping) on ​​the user interface (570) based on the viewpoint information, and if there is a lost part in the user interface depending on the viewpoint (e.g., if a part of the user interface is not displayed depending on the viewing position and / or angle), the electronic device can perform correction using an artificial intelligence model (e.g., the generative model (250) of FIG. 2, the generative model (470) of FIG. 4). The generative model (470) can perform out-painting on the lost part to correct it. The electronic device (501) can display a user interface (570) that reflects depth perception based on depth information. For example, the electronic device (501) can display the size of the user interface (570) differently based on depth information to reflect perspective.The electronic device (501) can display a user interface (570) by reflecting perspective, thereby allowing the user to perform a task by reflecting an actual space (e.g., a space in which the user is performing a task using the electronic device (501). The electronic device (501) can display the user interface (570) differently in real time on the second display (520) based on the user's viewpoint information. The electronic device (501) can generate images of the user interface (570) viewed from various angles in advance and display the images corresponding to the user's viewpoint. The electronic device (501) can generate and display the user interface (570) in real time according to the position from which the user is looking, based on the user's viewpoint information. The electronic device (501) can display the user interface (570) so that it can be consistently viewed from the user's viewpoint even when the user's viewpoint changes, based on the user's viewpoint information.

[0113] According to one embodiment, when the electronic device (501) displays the user interface (570) on the second display (520), if a user input is received, the electronic device (501) may change the user interface (570). For example, when the electronic device (501) receives an input from the user to move the user interface (570) displayed on the second display (520), the electronic device (501) may display the user interface (570) at a different location according to the user's intention. In response to receiving an additional gesture input from the user to move the application displayed on the first display (510) to the second display (520), the electronic device (501) may generate a new user interface for the application corresponding to the additional gesture input. The operation of the electronic device (501) generating the user interface for the application corresponding to the additional gesture input is identical to the operation of the user interface system (400) described with reference to FIGS. 1 to 4, and thus, a duplicate description thereof will be omitted. The electronic device (501) may place a user interface generated for an application corresponding to an additional gesture input together with a user interface (570) that was previously displayed on the second display (520). For example, if the electronic device (501) receives an additional gesture input from a user, since user interfaces for two or more applications must be displayed on the second display (520), the electronic device (501) may place and display each of the user interfaces on the second display (520). The electronic device (501) may enable the user to manually place the user interfaces. The electronic device (501) may automatically place the user interfaces based on information about each application. For example, the electronic device (501) may place a plurality of user interfaces based on information about each application (e.g., information such as the application's icon and function) and / or information such as the user's work pattern.The electronic device (501) can arrange multiple user interfaces using a model (e.g., an artificial intelligence model). The electronic device (501) can arrange multiple user interfaces by inputting information about each application and / or information such as the user's work pattern into the artificial intelligence model. If the electronic device (501) additionally receives a gesture input from the user to move an application displayed on the first display (510) to the second display (520), the electronic device (501) can change the user interface (570) corresponding to the application displayed on the second display (520). For example, in response to receiving an additional gesture input from the user, the electronic device (501) can change and display the application corresponding to the user interface (570) as a user interface in a predetermined form (e.g., a predetermined form such as an icon or a graphic image with depth information removed). The electronic device (501) can display the changed user interface at the bottom of the second display (520). The electronic device (501) can mainly display the running application added to the second display (520) according to the user's intention by changing the user interface (570) for the application that was previously displayed on the second display (520) (e.g., changing it to a user interface in the form of a small icon) and displaying it. The user can re-run the application that was previously being worked on through the changed user interface displayed on the second display (520) (e.g., a user interface in the form of a small icon displayed at the bottom of the second display (520). The electronic device (501) can display the existing application in full screen on the second display (520) in response to the user's input for the changed user interface. The electronic device (501) can provide a preview (thumbnail) so that the user can check the functions that can be supported by the running application.

[0114] FIG. 6A and FIG. 6B are drawings for explaining depth information according to one embodiment.

[0115] Referring to FIGS. 6A and 6B , according to one embodiment, an electronic device (601) (e.g., the electronic device (101) of FIG. 1 , the electronic device (301) of FIG. 3A , and the electronic device (501) of FIG. 5A ) may include a first display (610) disposed on a first surface of a first housing portion (e.g., the first display (310) of FIG. 3A , the first display (510) of FIG. 5A ) and a second display (620) disposed on a second surface of a second housing portion (e.g., the second display (320) of FIG. 3A , the second display (520) of FIG. 5A ). The first housing portion and the second housing portion may be foldably coupled to each other using at least one hinge device. The second display (620) may be a transparent display. The electronic device (601) may acquire an image including at least one object using a camera (e.g., the first camera (640)). The camera (e.g., the first camera (640)) may include a camera positioned on a third side of the second housing portion opposite the second side of the second housing portion. For example, the first camera (640) may include a camera positioned on the opposite side of the second display (620) or on the rear side. The electronic device (601) may obtain an image of a space positioned in front of the electronic device (601) using the first camera (640) positioned on the opposite side of the second display (620) or on the rear side. The electronic device (601) can obtain another image including the user's eyes through another camera (e.g., the second camera (830)) disposed on the second surface of the second housing portion. For example, the another camera can include a camera disposed on the same surface as the second display (620) or disposed in front. The electronic device (601) can identify the position of the user's eyes within the another image. The electronic device (601) can obtain the user's viewpoint information based on the position of the user's eyes.The electronic device (601) can modify the user interface in real time based on the position of the user's eyes. The electronic device (601) can display the user interface differently in real time on the second display (620) based on the user's viewpoint information acquired using the second camera (630). The electronic device (601) can generate depth information of the image based on the angle (e.g., angle (650), angle (655)) between the image and the first housing part and the second housing part. Information about the angle (e.g., angle (650), angle (655)) may be obtained from a sensor (e.g., sensor module (176) of FIG. 1). The electronic device (601) can preset a range of angles (e.g., folding angles) that the sensor can recognize. The angle (650) may be an angle formed between a floor surface on which the first housing part is placed and the second housing part. The angle (655) may be an internal angle formed between the first housing part and the second housing part. The electronic device (601) can obtain the angle (650) and / or the angle (655) using the sensor. The electronic device (601) can generate depth information of the image based on at least one of the angles (650) and (655) between the first display (610) and the second display (610). The angle between the displays (620) can be obtained. For example, based on the relationship (e.g., the complementary angle relationship) between the angles (650) and (655), the electronic device (601) can obtain the angle between the first display (610) and the second display (620). The electronic device (601) can input an image obtained using the first camera (640) into a model (e.g., a depth estimation model) and correct the obtained depth information based on the angle (650) to generate depth information. The depth information (depth map) is data indicating how far each pixel of the image is from the camera or sensor, and can be used to determine the position of an object included in a two-dimensional image in a three-dimensional space.The electronic device (601) can correct depth information in response to a detected change in the angle (650). The electronic device (601) can input an image acquired using the first camera (640) into a model (e.g., a depth estimation model) to correct the acquired depth information (e.g., an output of the depth estimation module (410) of FIG. 4) in response to a detected change in the angle (e.g., angle (650), angle (655)) between the first display (610) and the second display (620). The electronic device (601) can correct the depth information using a depth correction module (e.g., a depth correction module (430)). The electronic device (601) may not perform correction on the depth information if the angle (e.g., angle (650), angle (655)) between the first display (610) and the second display (620) changes within a predetermined error range. For example, the electronic device (601) may not correct the depth information if the angle (650) is changed within an error range that does not affect the depth information. The operation of the electronic device (601) to generate the depth information is substantially the same as the operation performed by the depth estimation module (410) and the depth correction module (430) described with reference to FIG. 4, and any overlapping descriptions thereof will be omitted.

[0116] FIG. 7 illustrates an example of a user interface according to one embodiment.

[0117] Referring to FIG. 7, according to one embodiment, an electronic device (701) (e.g., the electronic device (101) of FIG. 1, the electronic device (301) of FIG. 3A, the electronic device (501) of FIG. 5A, and the electronic device (601) of FIG. 6) may generate a different user interface depending on the actual space in which the electronic device (701) is located. For example, the electronic device (701) may generate a different user interface based on information about the actual space in which it is located (e.g., information such as the type of place, atmosphere, brightness, structure, and color temperature). When the electronic device (701) is located in a featureless, general space (715), the electronic device (701) may generate a user interface (710) with a basic design for a running application. When the electronic device (701) is located in a space (735) with many plants, the electronic device (701) may generate a user interface (730) that reflects the atmosphere of the space (735) for a running application.

[0118] FIG. 8 is a diagram for explaining user viewpoint information according to one embodiment.

[0119] Referring to FIG. 8, according to one embodiment, point-of-view (pov) information may include information about a point of view from which a user (810) is looking. The point-of-view information may be obtained through eye-tracking. An electronic device (e.g., the electronic device (101) of FIG. 1 , the electronic device (301) of FIG. 3A , the electronic device (501) of FIG. 5A , and the electronic device (601) of FIG. 6 ) may perform eye-tracking on a user (810) using a second camera (e.g., the second camera (830)) of a display (e.g., the second display (620) of FIG. 6 ). The second camera (830) may be disposed on the same surface as the second display or may include a camera disposed in front. The electronic device (801) may obtain an image including the user's eyes through the second camera (830). The electronic device (801) can input an image of the user (810) captured using the second camera (830) into an artificial intelligence model (e.g., a viewpoint estimation model). For example, the electronic device (801) can input an image of the user's (810) face captured using the second camera (830) into the viewpoint estimation model. The viewpoint estimation model can include an artificial intelligence model such as a machine learning model or a deep learning model. The electronic device (601) can use the viewpoint estimation model to find eyes from an image of the user's (810) face and obtain gaze information of the user. For example, the electronic device (601) can use the viewpoint estimation model to detect major facial parts such as eyes, nose, mouth, and ears from an image of the user's (810) face and track the eyes (e.g., pupils) of the user (810) to obtain gaze information. The electronic device (601) can generate the user's (810) viewpoint information using the viewpoint information generation module (440) described with reference to FIG. 4.The viewpoint information generation module (440) can input an image of the user (810) acquired using the camera (830) into the viewpoint estimation model to acquire viewpoint information of the user (810). The viewpoint estimation model can receive an image containing a human face and information about the display (e.g., hardware information) and output the user's viewpoint information. The viewpoint estimation model can receive the face image of the user (810) acquired using the camera (830) and search for major parts of the face (e.g., major parts of the face such as the eyes, nose, mouth, and ears). The viewpoint estimation model can detect pupils in the face of the user (810) for eye-tracking. By detecting the pupils of the user (810), the viewpoint estimation model can acquire information (e.g., focus) about the gaze that the user (810) is looking at. The viewpoint estimation model can match information about the gaze of the user (810) to the size of the panel of the display (e.g., the second display (620) of FIG. 6A). Since the viewpoint information may vary depending on the size of the display, the viewpoint estimation model can secure and use information about the display (e.g., hardware information) in advance. The hardware information may include device information internally stored in the electronic device (601). By using information about the display (e.g., hardware information) as input to the viewpoint estimation model, viewpoint information about the display of the user (810) can be accurately generated. The viewpoint information generation module (440) can regenerate the viewpoint information when the movement of the user (810) is detected. The electronic device (601) can display the user interface differently on the display in real time based on the viewpoint information of the user (810). The electronic device (601) can use a postprocessing module (e.g., the postprocessing module (475) of FIG. 4) to render the user interface differently in real time according to the viewpoint information and display it on the display.The electronic device (601) can display a user interface differently in real time on a display (e.g., the second display (620) of FIG. 6A) based on the viewpoint information of the user (810). The electronic device (601) can generate images of the user interface viewed from various angles in advance and display the images corresponding to the viewpoint of the user (810). The electronic device (601) can generate and display the user interface in real time according to the position from which the user is looking based on the viewpoint information of the user (810). The electronic device (601) can display the user interface so that it can be consistently viewed from the user's viewpoint even when the viewpoint of the user (810) changes based on the viewpoint information of the user (810).

[0120] FIGS. 9A and 9B are drawings illustrating examples of electronic devices including a transparent display according to one embodiment.

[0121] According to one embodiment, an electronic device (901) (e.g., the electronic device (101) of FIG. 1) may include at least one transparent display (e.g., the display (920)). When the display (920) is not displaying an image, the display (920) may transmit light incident from the rear, so that an object (930) present behind the display (920) may be visible through the display (920). When a user (e.g., a user of the electronic device (901)) executes an application on the electronic device (901), the electronic device (901) may display an application window (940) on the display (920). When the application window (940) is displayed on the display (920), the electronic device (901) may control the display panel so that an object (930) present behind the display (920) is not visible. When a user reduces a portion of an application window (940) while running an application, an object (930) located behind the display (920) may be visible through a portion of the display (920) excluding the application window (940). The electronic device (901) may generate and display a user interface (970) on a portion of the display (920) excluding the application window (940). The electronic device (901) may generate and provide a user interface in addition to a user interface for a function basically provided in the application. The electronic device (901) may generate a user interface using an artificial intelligence model (e.g., a generative model (250) of FIG. 2, a generative model (470) of FIG. 4). For example, the electronic device (901) may generate and provide a user interface to the user by combining the functions of an application or based on a function that can be provided through an artificial intelligence model (e.g., an AI assistant).The user interface (970) may be substantially the same as the user interface described with reference to FIGS. 1 to 8 (e.g., the user interface (370) of FIG. 3a, the user interface (480) of FIG. 4, the user interface (570) of FIG. 5b, and the user interfaces (710, 730) of FIG. 7).

[0122] FIGS. 10A to 10C are drawings for explaining an operating method of an electronic device according to one embodiment.

[0123] Referring to FIGS. 10A to 10C , according to one embodiment, an electronic device (1001) (e.g., the electronic device (101) of FIG. 1 ) may include at least one transparent display (e.g., the display (1010)). The electronic device (1001) may include a display (1020) coupled with the transparent display (e.g., the display (1010)). The display (1020) may include a display disposed in a first housing portion in which the display (1010) is disposed and a second housing portion that are foldably coupled to each other using at least one hinge device. The display (1020) may include a transparent display or a general display that is not a transparent display. The electronic device (1001) may include a foldable device. The state of the electronic device (1001) may include a fully unfolded state (e.g., a state of the electronic device (1001)), a fully folded state (e.g., a state 1002, 1003 of the electronic device (1001) being fully folded), and a partially folded state. FIGS. 10A to 10C illustrate examples of content (e.g., images and / or videos) being displayed on the electronic device (1001). The videos may include videos stored on the electronic device (1001) and / or videos provided through a streaming service. The display (1010) may include a blackout screen controlled by a motor. For example, the display (1010) may include a thin blackout screen on the back that is controlled by a motor that includes a rollable device. The electronic device (1001) can control a blackout screen to be unfolded on the back of the display (1010) while content is displayed on the display (1010).The electronic device (1001) can control the unfolding of a blackout screen to prevent a background (e.g., an actual space) from being reflected behind the content, thereby enhancing the user's viewing immersion. While viewing content using the electronic device (1001), the user can completely fold the electronic device (1001). The content can be displayed on the display (1010) located at the front of the electronic device (1001) in a completely folded state (1002). The electronic device (1001) can control the unfolding of the blackout screen in the completely folded state (1002) to prevent a background from being reflected behind the content. When displaying content on the display (1010) in a completely folded state (1003) of the electronic device (1001), whether or not the blackout screen is used may vary depending on an option set by the user. For example, the electronic device (1001) may not use the blackout screen if the user has set the electronic device (1001) not to use the blackout screen when the electronic device (1001) is fully folded (1002) (e.g., set the back to be transparent). The electronic device (1001) may obtain the folding speed of the electronic device (1001) from a sensor associated with a hinge that connects the display (1010) and the display (1020). The electronic device (1001) may transmit the folding speed (e.g., a signal) of the electronic device (1001) to a module (e.g., a motor control module) that controls a motor that controls the blackout screen. Since the folding speed of the electronic device (1001) may not be constant, the signal obtained from the sensor associated with the hinge may need to be transmitted to the motor control module. The motor control module may control the motor so that the length of the blackout screen changes as the electronic device (1001) is folded. For example, when the electronic device (1001) is in a completely folded state (1002), the length of the blackout screen needs to be extended by the folded portion, so the electronic device (1001) can control the motor to extend the blackout screen.The length of the blackout screen can be actively changed depending on the degree to which the electronic device (1001) is folded.

[0124] According to one embodiment, the electronic device (1001) may separate an object (e.g., object (1050)) included in content and display it on the display (1010) in a fully folded state (1002, 1003). The electronic device (1001) may input the content into a model (e.g., segmentation AI model) to separate the background and the foreground. The electronic device (1001) may use the segmentation AI model to detect the foreground from the content. The foreground may include an object (e.g., object (1050)) included in the content. If all objects included in the content are separated into the foreground, the electronic device (1001) may lower content immersion and slow down the processing speed, and therefore, the criteria for objects to be separated into the foreground may be determined in advance. For example, the electronic device (1001) may predetermine criteria such as the type of the main object to be segmented into the foreground (e.g., the type of the main object such as a person, an animal other than a person, or a vehicle) and the saliency of the object. The segmentation artificial intelligence model may recognize and classify objects included in the content. The segmentation artificial intelligence model may be a model trained using a training data pair including an image including an object and correct data for the object included in the image. The segmentation artificial intelligence model may receive at least one frame of the content and recognize an object included in the frame. The segmentation artificial intelligence model may recognize an object included in the frame using a semantic segmentation technique that does not separately classify objects of the same type. Since the electronic device (1001) seeks to recognize the main object to be segmented into the foreground, the segmentation artificial intelligence model may be sufficient to recognize only the main object. The segmentation artificial intelligence model may classify and output the type of the recognized object.The electronic device (1001) can determine which objects to separate from the content as foreground based on the type of objects output by the segmentation artificial intelligence model. The electronic device (1001) can analyze the type of objects and determine which objects to separate as foreground based on priorities. For example, the electronic device (1001) can determine which objects to separate as foreground among objects classified according to a predetermined priority (e.g., priority in the order of people, animals, and objects). The priority can be determined based on the user's content viewing history. For example, if the user primarily views game content or animation, game characters or animation characters can be set as high priorities. If an object included in the content is not included in the priority, the electronic device (1001) can perform saliency detection to determine which objects to separate as foreground. The electronic device (1001) can input content into a saliency detection model and perform saliency detection on objects included in the content. When the electronic device (1001) completes the detection of priority, it can determine an object to be separated as the foreground based on the priority and / or the result of the detection of priority. When the electronic device (1001) cannot determine an object to be separated as the foreground based on the result of the detection of priority and / or the result of the detection of priority, it can not separate the foreground. The electronic device (1001) can determine an area excluding an object separated as the foreground (e.g., an object (1050)) as the background. The electronic device (1001) can display the object (1050) separated as the foreground on the display (1010) located at the front, and display the background on the display (1020) located at the rear. The background can include an area corresponding to the foreground in the content by replacing it with black. Since the display (1010) located at the front is a transparent display, the electronic device (1001) can create a physical sense of space according to the thickness of the display (1010) when it is completely folded (1003).A sense of space may be created between the foreground displayed on the display (1010) and the background displayed on the display (1020).

[0125] According to one embodiment, the electronic device (1001) may display additional shadow information for an object (1050) included in the foreground displayed on the display (1010) in addition to the physical sense of space according to the thickness of the display (1010) in a fully folded state (1003). The electronic device (1001) may generate and display shadows (1055, 1057) for the object (1050) separated from the foreground. The electronic device (1001) may generate shadows (1055, 1057) corresponding to the object (1050) using a model (e.g., the generative model (250) of FIG. 2). The electronic device (1001) may input the original frame of the content and the image of the object (1050) into the model. The model can generate and output a shadow (1055, 1057) that matches the object (1050) based on information about the movement of the object (1050) from the original frame and information about a light source object (e.g., lighting (1030)) included in the original frame. The model can generate the shadow (1055, 1057) to change according to the movement of the object (1050). The model can generate the shadow (1055, 1057) differently according to information about the lighting (10030) (e.g., information such as light intensity and position). The shadow (1055, 1057) can be displayed on the display (1010) located at the front in a fully folded state (1003) of the electronic device (1001). The electronic device (1001) can display shadows (1055, 1057) on the display (1010) by considering the physical sense of space generated according to the thickness of the display (1010). For example, the electronic device (1001) can adjust the degree of curvature of the shadows (1055, 1057) and display them on the display (1010) by considering the physical sense of space generated according to the thickness of the display (1010). The electronic device (1001) can provide a content viewing function with enhanced realism by reflecting shadow information about the object (1050) and displaying it on the display (1010).

[0126] Fig. 11 is a flowchart for explaining an operating method of an electronic device according to one embodiment.

[0127] Referring to FIG. 11, according to one embodiment, operations 1110 to 1170 may be substantially identical to operations performed by the electronic device (e.g., the electronic device (101) of FIG. 1) described with reference to FIGS. 1 to 10. Therefore, redundant descriptions will be omitted.

[0128] In operation 1110, the electronic device (101) may receive a user input from a user to move an application displayed on a first display (e.g., the first display (310) of FIG. 3A) to a second display (e.g., the second display (320) of FIG. 3A). The user input may refer to a user input to move an application (e.g., software including an application program or a system program) displayed on the first display (310) to the second display (320). For example, the gesture input may be an input including an intent of the user to move the application displayed on the first display (310) to the second display (320), regardless of the format such as a touch input, a mouse input, a gaze input, a keyboard input, or a voice input. The application displayed on the first display (310) may include an application running on the electronic device (101).

[0129] In operation 1120, the electronic device (101) may acquire an image using a first camera of a second display (e.g., the second display (320) of FIG. 3A). The first camera may be positioned opposite the second display (320) of the electronic device (101) or may be positioned at the rear. The first camera may include a sensor (e.g., a camera) for acquiring depth information. The electronic device (101) may input the image acquired using the first camera into a model (e.g., a depth estimation model) to acquire depth information.

[0130] In operation 1125, the electronic device (101) may obtain an angle between a first display (e.g., the first display (310) of FIG. 3A) and a second display (e.g., the second display (320) of FIG. 3A) from a sensor associated with a hinge. The second display (320) may be a transparent display coupled to the first display (310) by a hinge. The electronic device (101) may preset a range of angles that a sensor associated with the hinge (e.g., a hinge sensor) may recognize. For example, if the angle recognized by the hinge sensor is outside a predetermined range of angles, the electronic device may determine the angle to be a value within the predetermined range of angles.

[0131] In operation 1130, the electronic device (101) may input an image into a model (e.g., a depth estimation model) to obtain depth information, and may correct the acquired depth information based on the angle between the first display (e.g., the first display (310) of FIG. 3A) and the second display (e.g., the second display (320) of FIG. 3A). When a change in the angle between the first display (310) and the second display (320) is detected, the electronic device (101) may correct the depth information based on the changed angle. After obtaining an image using the first camera of the second display (320), if the change in the angle between the first display (310) and the second display (320) is within a predetermined range (e.g., an error range), the electronic device (101) may perform operation 1140 without performing operation 1130. For example, if it is determined that the depth information obtained by inputting an image into a model (e.g., a depth estimation model) does not need to be corrected according to the angle between the first display (310) and the second display (320), the electronic device (101) can generate a user interface based on the depth information that is the output of the model (e.g., a depth estimation model).

[0132] In operation 1140, the electronic device (101) may generate a user interface for an application (e.g., the user interface (370) of FIG. 3A). The electronic device (101) may generate a user interface for an application running on the electronic device (101). The user interface may be generated by inputting depth information and information about the application into a model (e.g., the generative model (250) of FIG. 2, the generative model (470) of FIG. 4). The electronic device (101) may use the model (e.g., the generative model (250) of FIG. 2, the generative model (470) of FIG. 4) to generate a user interface (370) for the application running on the electronic device (101) and display the user interface (370) on the second display (320).

[0133] In operation 1150, the electronic device (101) may acquire the user's viewpoint information using a second camera of a second display (e.g., the second display (320) of FIG. 3A). The second camera may include a camera positioned in front of the second display (320). The electronic device (101) may capture a user (e.g., the user's face) using the second camera. For example, the electronic device (101) may capture the user's face in real time using the second camera. The electronic device (101) may input the user's face image acquired using the second camera into an artificial intelligence model (e.g., a viewpoint estimation model) to acquire the user's viewpoint information.

[0134] In operation 1160, the electronic device (101) may determine whether the user's movement has been detected. The electronic device (101) may determine whether the user's movement has been detected based on the result of capturing the user using the second camera of the second display (e.g., the second display (320) of FIG. 3A). When the user's movement is detected, the user's viewpoint information may also change, so the electronic device (101) may determine whether the user's movement has been detected and re-acquire the viewpoint information. For example, when the user's movement is detected, the electronic device (101) may re-acquire and update the viewpoint information. In response to detecting the user's movement, the electronic device (101) may perform operation 1150 again. In response to not detecting the user's movement, the electronic device (101) may perform operation 1170.

[0135] In operation 1170, the electronic device (101) may display a user interface (e.g., the user interface (370) of FIG. 3A) differently in real time on a second display (e.g., the second display (320) of FIG. 3A) based on viewpoint information. The electronic device (101) may display the user interface (370) so that it appears to the user in a consistent manner based on the user's viewpoint information.

[0136] In one embodiment, operations 1110 through 1170 may be performed sequentially, but are not limited thereto. For example, two or more operations may be performed in parallel.

[0137] A foldable electronic device (101; 301; 501; 601; 701; 901; 1001) according to one embodiment comprises a foldable housing including a first housing portion and a second housing portion, a first display (160; 310; 510; 610; 1020) arranged on a first surface of the first housing portion, a second display (160; 320; 520; 620; 920; 1010) arranged on a second surface of the second housing portion combined with the first display (160; 310; 510; 610; 1020), a camera (180; 640) arranged on a third surface of the second housing portion opposite the second surface, a sensor, a first display (160; 310; 510; 610; 1020) 610; 1020), a second display (160; 320; 520; 620; 920; 1010) combined with the first display (160; 310; 510; 610; 1020), at least one processor (120) including a processing circuit, and a memory (130) storing instructions.

[0138] The above instructions, when individually or collectively executed by the at least one processor (120), may cause the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) to obtain an image including at least one object using the camera (180; 640) of the second display (160; 320; 520; 620; 920; 1010).

[0139] The above instructions, when individually or collectively executed by the at least one processor (120), may cause the electronic device (101; 301; 501; 601; 701; 901; 1001) to obtain information about an angle between the first housing portion and the second housing portion using the sensor.

[0140] The above instructions, when individually or collectively executed by the at least one processor (120), may cause the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) to generate depth information based on at least a portion of the image and information about the angle.

[0141] The above instructions, when individually or collectively executed by the at least one processor (120), may cause the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) to generate a user interface (UI) (370; 480; 570; 710; 730; 970) for an application running on the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) based on the depth information.

[0142] The above instructions, when individually or collectively executed by the at least one processor (120), may cause the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) to display the user interface (370; 480; 570; 710; 730; 970) on the second display (160; 320; 520; 620; 920; 1010).

[0143] The above second display (160; 320; 520; 620; 920; 1010) may be a transparent display.

[0144] The above instructions, when individually or collectively executed by the at least one processor (120), may cause the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) to generate further depth information using the image acquired from the camera (180; 640), at least as a part of generating the depth information.

[0145] The above instructions, when individually or collectively executed by the at least one processor (120), may cause the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) to modify the other depth information based on the information about the angle to generate the depth information.

[0146] The above foldable electronic device (101; 301; 501; 601; 701; 901; 1001) may further include another camera (180; 630; 830) arranged on the second surface of the second housing portion.

[0147] The above instructions, when individually or collectively executed by the at least one processor (120), may cause the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) to acquire another image including the user's eyes through the another camera (180; 630; 830).

[0148] The above instructions, when individually or collectively executed by the at least one processor (120), may cause the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) to identify the position of the user's eyes within the another image.

[0149] The above instructions, when individually or collectively executed by the at least one processor (120), may cause the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) to modify the user interface in real time based on the position of the user's eyes.

[0150] The above instructions, when individually or collectively executed by the at least one processor (120), may cause the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) to display a modified user interface on the second display (160; 320; 520; 620; 920; 1010).

[0151] The above instructions, when individually or collectively executed by the at least one processor (120), may cause the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) to generate the user interface (370; 480; 570; 710; 730; 970) using the model (250; 470).

[0152] The above model (250; 470) may be trained to generate a user interface based on the depth information and information about the application.

[0153] The above instructions, when individually or collectively executed by the at least one processor (120), may cause the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) to generate the user interface (370; 480; 570; 710; 730; 970) based on at least a portion of the characteristics of the at least one object.

[0154] The above characteristics may include color, brightness, position, or material.

[0155] The above instructions, when individually or collectively executed by the at least one processor (120), may cause the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) to generate the user interface (370; 480; 570; 710; 730; 970) in response to receiving a user input from a user.

[0156] The instructions, when individually or collectively executed by the at least one processor (120), cause the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) to generate the user interface (370; 480; 570; 710; 730; 970) and display the user interface (370; 480; 570; 710; 730; 970) on the second display (160; 320; 520; 620; 920; 1010) based at least in part on determining that the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) is in a fully folded state. 970) can be avoided.

[0157] The above instructions, when individually or collectively executed by the at least one processor (120), may cause the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) to display another user interface associated with the application stored in the memory (130) using the first display (160; 310; 510; 610; 1020).

[0158] A method of operating a foldable electronic device (101; 301; 501; 601; 701; 901; 1001) according to one embodiment may include an operation of acquiring an image including at least one object using a camera (180; 640).

[0159] The above operating method may include an operation of obtaining information about an angle between a first housing part in which a first display (160; 310; 510; 610; 1020) is arranged and a second housing part in which a second display (160; 320; 520; 620; 920; 1010) is arranged using a sensor.

[0160] The above method of operation may include an operation of generating depth information based on at least a portion of the image and information about the angle.

[0161] The above operating method may include an operation of generating a user interface (370; 480; 570; 710; 730; 970) for an application running on the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) based on the depth information.

[0162] The above method of operation may include an operation of displaying the user interface (370; 480; 570; 710; 730; 970) on the second display (160; 320; 520; 620; 920; 1010).

[0163] The above second display (160; 320; 520; 620; 920; 1010) may be a transparent display.

[0164] The operation of generating the above depth information may include an operation of generating another depth information using the image acquired from the camera (180; 640).

[0165] The operation of generating the depth information may include an operation of generating the depth information by modifying the other depth information based on information about the angle.

[0166] The displaying action may include obtaining another image including the user's eyes using another camera (180; 630; 830).

[0167] The displaying action may include identifying the position of the user's eyes within the other image.

[0168] The displaying action may include modifying the user interface in real time based on the position of the user's eyes.

[0169] The displaying action may include displaying the modified user interface on the second display (160; 320; 520; 620; 920; 1010).

[0170] The above another camera (180; 630; 830) may be disposed on the opposite side of the second housing portion where the camera (180; 640) is disposed.

[0171] The operation of generating the user interface (370; 480; 570; 710; 730; 970) may include an operation of generating the user interface (370; 480; 570; 710; 730; 970) using the model (250; 470).

[0172] The above model (250; 470) may be trained to generate a user interface based on the depth information and information about the application.

[0173] The operation of generating the user interface (370; 480; 570; 710; 730; 970) may further include an operation of generating the user interface (370; 480; 570; 710; 730; 970) based on at least a portion of the characteristics of the at least one object.

[0174] The above characteristics may include color, brightness, position, or material.

[0175] The above method of operation may further include an operation of generating the user interface (370; 480; 570; 710; 730; 970) in response to receiving user input from a user.

[0176] The method of operation may further include generating the user interface (370; 480; 570; 710; 730; 970) and refraining from displaying the user interface (370; 480; 570; 710; 730; 970) on the second display (160; 320; 520; 620; 920; 1010) based at least in part on determining that the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) is in a fully folded state.

[0177] The above method of operation may further include an operation of displaying another user interface associated with the application stored in the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) using the first display (160; 310; 510; 610; 1020).

[0178] According to one embodiment, a computer-readable recording medium storing one or more computer programs may include instructions for performing the method in a processor.

[0179] A foldable electronic device (101; 301; 501; 601; 701; 901; 1001) according to one embodiment may include a foldable housing including a first housing portion and a second housing portion, a flexible display including a first portion supported by the first housing portion and a second portion supported by a second surface of the second housing portion, an image sensor disposed on a third surface of the second housing portion opposite the second surface, at least one processor (120) including a sensor and a processing circuit, and a memory (130) storing instructions.

[0180] The above instructions, when individually or collectively executed by the at least one processor (120), may cause the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) to acquire an image including at least one object using the image sensor.

[0181] The above instructions, when individually or collectively executed by the at least one processor (120), may cause the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) to obtain information about an angle between the first housing portion and the second housing portion using the sensor.

[0182] The above instructions, when individually or collectively executed by the at least one processor (120), may cause the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) to generate depth information based on at least a portion of the image and information about the angle.

[0183] The above instructions, when individually or collectively executed by the at least one processor (120), may cause the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) to generate a user interface for an application running on the foldable electronic device based on the depth information.

[0184] The above instructions, when individually or collectively executed by the at least one processor (120), may cause the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) to display the user interface on the second portion using the flexible display.

[0185] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0186] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish the corresponding component from other corresponding components and do not limit the corresponding components in any other respect (e.g., importance or order). When a component (e.g., a first) is referred to as "coupled" or "connected" to another (e.g., a second) component, with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0187] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0188] Various embodiments of the present document may be implemented as software (e.g., a program (1740)) including one or more instructions stored in a storage medium (e.g., an internal memory (1736) or an external memory (1738)) readable by a machine (e.g., an electronic device (1701)). For example, a processor (e.g., a processor (1720)) of the machine (e.g., an electronic device (1701)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0189] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0190] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In a foldable electronic device (101; 301; 501; 601; 701; 901; 1001), A foldable housing comprising a first housing portion and a second housing portion; A first display (160; 310; 510; 610; 1020) arranged on the first surface of the first housing portion; A second display (160; 320; 520; 620; 920; 1010) arranged on the second surface of the second housing portion; A camera (180; 640) arranged on the third side of the second housing portion opposite the second side; sensor; At least one processor (120) comprising processing circuitry; and Memory for storing instructions (130) Including, The above instructions, when individually or collectively executed by the at least one processor (120), cause the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) to: Obtain an image containing at least one object using the above camera (180; 640), Using the above sensor, information about the angle between the first housing part and the second housing part is obtained, Generating depth information based on at least a portion of the image and information about the angle, Based on the depth information, a user interface (UI) (370; 480; 570; 710; 730; 970) for an application running on the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) is generated, To display the above user interface (370; 480; 570; 710; 730; 970) on the second display (160; 320; 520; 620; 920; 1010), Foldable electronic devices (101; 301; 501; 601; 701; 901; 1001).

2. In paragraph 1, The above second display (160; 320; 520; 620; 920; 1010) is A transparent display, Foldable electronic devices (101; 301; 501; 601; 701; 901; 1001).

3. In any one of paragraphs 1 and 2, The above instructions, when individually or collectively executed by the at least one processor (120), cause the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) to: As at least part of generating the depth information, generating another depth information using the image acquired from the camera (180; 640), Based on the information about the angle, the other depth information is modified to generate the depth information. Foldable electronic devices (101; 301; 501; 601; 701; 901; 1001).

4. In any one of paragraphs 1 to 3, Another camera (180; 630; 830) arranged on the second surface of the second housing portion Including more, The above instructions, when individually or collectively executed by the at least one processor (120), cause the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) to: Another image including the user's eyes is acquired through another camera (180; 630; 830), Identifying the position of the user's eyes within another image, Modify the user interface in real time based on the position of the user's eyes, To display the modified user interface on the second display (160; 320; 520; 620; 920; 1010), Foldable electronic devices (101; 301; 501; 601; 701; 901; 1001).

5. In any one of paragraphs 1 to 4, The above instructions, when individually or collectively executed by the at least one processor (120), cause the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) to: Using the model (250; 470), the user interface (370; 480; 570; 710; 730; 970) is created, The above model (250; 470) is Based on the depth information and information about the application, it is trained to generate a user interface. Foldable electronic devices (101; 301; 501; 601; 701; 901; 1001).

6. In any one of paragraphs 1 to 5, The above instructions, when individually or collectively executed by the at least one processor (120), cause the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) to: Generate the user interface (370; 480; 570; 710; 730; 970) based on at least a part of the characteristics of the at least one object, The above characteristics are, Including color, brightness, position, or material, Foldable electronic devices (101; 301; 501; 601; 701; 901; 1001).

7. In any one of paragraphs 1 to 6, The above instructions, when individually or collectively executed by the at least one processor (120), cause the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) to: In response to receiving user input from a user, generating the user interface (370; 480; 570; 710; 730; 970). Foldable electronic devices (101; 301; 501; 601; 701; 901; 1001).

8. In any one of paragraphs 1 to 7, The above instructions, when individually or collectively executed by the at least one processor (120), cause the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) to: Based at least in part on determining that the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) is in a fully folded state, generating the user interface (370; 480; 570; 710; 730; 970) and displaying the user interface (370; 480; 570; 710; 730; 970) on the second display (160; 320; 520; 620; 920; 1010) are avoided. Using the first display (160; 310; 510; 610; 1020), display another user interface associated with the application stored in the memory (130). Foldable electronic devices (101; 301; 501; 601; 701; 901; 1001).

9. In the operating method of a foldable electronic device (101; 301; 501; 601; 701; 901; 1001), An operation of acquiring an image containing at least one object using a camera (180; 640); An operation of obtaining information about an angle between a first housing part in which a first display (160; 310; 510; 610; 1020) is arranged and a second housing part in which a second display (160; 320; 520; 620; 920; 1010) is arranged using a sensor; An operation of generating depth information based on at least a portion of the image and information about the angle; An operation of generating a user interface (370; 480; 570; 710; 730; 970) for an application running on the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) based on the depth information; and An operation of displaying the above user interface (370; 480; 570; 710; 730; 970) on the second display (160; 320; 520; 620; 920; 1010) including, How it works.

10. In paragraph 9, The above second display (160; 320; 520; 620; 920; 1010) is A transparent display, How it works.

11. In any one of paragraphs 9 to 10, The operation of generating the above depth information is: An operation of generating further depth information using the image acquired from the camera (180; 640); and An operation of generating the depth information by modifying the other depth information based on the information about the angle. including, How it works.

12. In any one of paragraphs 9 to 11, The above displaying action is, An action of acquiring another image including the user's eyes using another camera (180; 630; 830); An action of identifying the position of the user's eyes within said another image; An operation of modifying the user interface in real time based on the position of the user's eyes; and An action of displaying a modified user interface on the second display (160; 320; 520; 620; 920; 1010) including, How it works.

13. In any one of paragraphs 9 to 12, Another camera (180; 630; 830) above, The camera (180; 640) is disposed on the opposite side of the second housing portion. How it works.

14. In any one of paragraphs 9 to 13, The action of generating the above user interface (370; 480; 570; 710; 730; 970) is: An operation of generating the user interface (370; 480; 570; 710; 730; 970) using the model (250; 470) Including, The above model (250; 470) is Based on the depth information and information about the application, it is trained to generate a user interface. How it works.

15. In a foldable electronic device (101; 301; 501; 601; 701; 901; 1001), A foldable housing comprising a first housing portion and a second housing portion; A flexible display comprising a first part supported by the first housing part and a second part supported by the second surface of the second housing part; An image sensor disposed on a third surface of the second housing portion opposite the second surface; sensor; At least one processor (120) comprising processing circuitry; and Memory for storing instructions (130) Including, The above instructions, when individually or collectively executed by the at least one processor (120), cause the foldable electronic device (101; 301; 501; 601; 701; 901; 1001) to: Obtaining an image including at least one object using the image sensor, Using the above sensor, information about the angle between the first housing part and the second housing part is obtained, Generating depth information based on at least a portion of the image and information about the angle, Based on the depth information, generate a user interface for an application running on the foldable electronic device, Using the flexible display, the user interface is displayed on the second part. Foldable electronic devices (101; 301; 501; 601; 701; 901; 1001).

Images