Wearable electronic device for displaying extended reality images, method of operating the same, and recording medium

Abstract

Disclosed is a wearable electronic device that: displays, through a display, a first screen including a plurality of objects arranged at a first location based on an image captured by using a camera; selects at least one object to be copied from the plurality of objects based on a first gesture of a user; determines a second location on the first screen at which the at least one object is to be displayed based on a second gesture of the user; provides a property of the second location and information about the at least one object to a generative AI model stored in a memory in order to perform at least one of in-painting of a portion of the at least one object hidden by another object or out-painting of the at least one object based on the property of the second location; performs the at least one of in-painting and out-painting in order to acquire an object image corresponding to the at least one object; and displays, through the display, a second screen including the object image arranged at the second location based on a third gesture of the user.

Description

Technical Field

[0001] Embodiments of this disclosure relate to a wearable electronic device for displaying extended reality images, its operation method, and a recording medium. Background Technology

[0002] Wearable electronic devices, such as augmented reality glasses (AR glasses), video see-through (VST) devices, virtual reality (VR) devices, and other extended reality (XR) devices, are offering an increasing number of services and additional features. To meet the diverse needs of users and improve the efficiency of electronic devices, communication service operators and equipment manufacturers are competing to develop electronic devices with differentiated and diversified functions. Therefore, the various functions offered by wearable electronic devices are increasingly evolving.

[0003] Extended reality (XR) devices, such as augmented reality glasses (AR glasses), video see-through (VST) devices, and virtual reality (VR) devices, provide users with realistic experiences by displaying virtual images when worn on the user's body. XR devices (such as AR glasses, VST devices, and VR devices) can replace the availability of smartphones in various fields such as gaming, education, and social networking services. Through XR devices (e.g., AR / glasses, VST devices, or VR devices), users can be provided with realistic content and interact with it to feel as if they are in a virtual world. Summary of the Invention

[0004] Solution to the problem

[0005] According to embodiments, a wearable electronic device worn on a user's head may include a camera, a display, at least one processor, and a memory storing instructions. When executed by at least one processor, the instructions according to embodiments enable the wearable electronic device to display a first screen, including a plurality of objects arranged at a first position, on the display based on images captured using the camera. When executed by at least one processor, the instructions according to embodiments enable the wearable electronic device to select at least one object to be copied from the plurality of objects based on a first gesture of the user. When executed by at least one processor, the instructions according to embodiments enable the wearable electronic device to determine a second position on the first screen for displaying at least one object based on a second gesture of the user. When executed by at least one processor, the instructions according to embodiments enable the wearable electronic device to perform at least one of in-painting or out-painting of the at least one object on the first screen based on the attributes of the second position by providing attributes of the second position and information about the at least one object to a generative AI model stored in memory, and to obtain an object image corresponding to the at least one object by performing at least one of in-painting or out-painting. When executed by at least one processor, the instructions according to embodiments enable the wearable electronic device to display a second screen, including an image of an object located at the second position, on the display based on a third gesture of the user.

[0006] According to an embodiment, a method for operating a wearable electronic device worn on a user's head may include: displaying a first screen comprising a plurality of objects arranged at a first position via a display included in the wearable electronic device, based on an image captured using a camera included in the wearable electronic device. According to an embodiment, the method for operating the wearable electronic device may include: selecting at least one object to be copied from the plurality of objects based on a first gesture of the user. According to an embodiment, the method for operating the wearable electronic device may include: determining a second position on the first screen for displaying at least one object based on a second gesture of the user. According to an embodiment, the method for operating the wearable electronic device may include: providing attributes of the second position and information about the at least one object stored in a generative AI model in a memory included in the wearable electronic device; performing at least one of in-painting or out-painting of the at least one object based on the attributes of the second position; and obtaining an object image corresponding to the at least one object by performing at least one of in-painting or out-painting. According to an embodiment, the method for operating the wearable electronic device may include: displaying a second screen comprising an image of an object located at the second position via a display based on a third gesture of the user.

[0007] According to an embodiment, a non-transitory recording medium may store instructions that, when executed by at least one processor, cause at least one processor to: display a first screen comprising a plurality of objects arranged at a first position via a display included in a wearable electronic device worn on a user's head, based on an image captured using a camera included in the wearable electronic device; select at least one object to be copied from the plurality of objects based on a first gesture of the user; and determine a second position on the first screen for displaying the at least one object based on a second gesture of the user. By providing attributes of the second position and information about the at least one object to a generative AI model stored in a memory included in the wearable electronic device, at least one of in-line drawing of the portion of the at least one object occluded by at least one other object or out-line drawing of the at least one object is performed based on the attributes of the second position, and an object image corresponding to the at least one object is obtained by performing at least one of in-line drawing or out-line drawing, and a second screen comprising the object image located at the second position is displayed via a display based on a third gesture of the user. Attached Figure Description

[0008] Figure 1 This is a diagram illustrating an electronic device in a network environment according to various embodiments;

[0009] Figure 2 This is a perspective view showing the internal configuration of a wearable electronic device according to an embodiment of the present disclosure;

[0010] Figure 3a and Figure 3b This is a view showing the front and rear surfaces of a wearable electronic device according to an embodiment;

[0011] Figure 4 This is a schematic block diagram illustrating a wearable electronic device according to an embodiment;

[0012] Figure 5a This is a flowchart illustrating an operation method of a wearable electronic device according to an embodiment;

[0013] Figure 5b This is a flowchart illustrating an operation method of a wearable electronic device according to an embodiment;

[0014] Figure 6a This is a view illustrating the functionality of copying and pasting objects displayed on a screen by a wearable electronic device according to an embodiment;

[0015] Figure 6b This is a view illustrating the function of a wearable electronic device cutting out an object displayed on a screen according to an embodiment;

[0016] Figure 7This is a flowchart illustrating the in-line and out-line drawing performed by a wearable electronic device using a generative AI model on an image representing at least one object, according to an embodiment;

[0017] Figure 8 The illustration shows in-line and out-line drawing performed by a wearable electronic device using a generative AI model on an image representing at least one object, according to an embodiment.

[0018] Figure 9 This is a flowchart illustrating a method for selecting at least one object by a wearable electronic device according to an embodiment;

[0019] Figure 10a and Figure 10b This is a view illustrating a method for selecting at least one object by a wearable electronic device according to an embodiment;

[0020] Figure 11 This is a flowchart illustrating a method for determining the position of at least one selected object by a wearable electronic device according to an embodiment;

[0021] Figure 12a and Figure 12b A method for determining the location of at least one selected object by a wearable electronic device according to an embodiment is illustrated.

[0022] Figure 13 This is a flowchart illustrating a method for editing an object image by a wearable electronic device according to an embodiment;

[0023] Figure 14 A method for editing an object image by a wearable electronic device according to an embodiment is shown;

[0024] Figure 15 This is a flowchart illustrating a method for displaying an image of an object via a wearable electronic device according to an embodiment; and

[0025] Figure 16 This is a view illustrating a method for displaying an image of an object via a wearable electronic device according to an embodiment. Detailed Implementation

[0026] Figure 1 This is a block diagram illustrating an electronic device 101 in a network environment 100 according to various embodiments. Reference Figure 1In network environment 100, electronic device 101 can communicate with at least one of electronic devices 102 via a first network 198 (e.g., a short-range wireless communication network), or with electronic device 104 or server 108 via a second network 199 (e.g., a long-range wireless communication network). According to an embodiment, electronic device 101 can communicate with electronic device 104 via server 108. According to an embodiment, electronic device 101 may include a processor 120, memory 130, input module 150, sound output module 155, display module 160, audio module 170, sensor module 176, interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module (SIM) 196, or antenna module 197. In an embodiment, at least one of the components (e.g., connection terminal 178) may be omitted from electronic device 101, or one or more other components may be added to electronic device 101. According to an embodiment, some of the components (e.g., sensor module 176, camera module 180, or antenna module 197) may be integrated into a single component (e.g., display module 160).

[0027] Processor 120 can execute, for example, software (e.g., program 140) to control at least one other component (e.g., hardware or software component) of electronic device 101 coupled to processor 120, and can perform various data processing or calculations. According to embodiments, as at least part of data processing or calculation, processor 120 can store commands or data received from another component (e.g., sensor module 176 or communication module 190) in volatile memory 132, process the commands or data stored in volatile memory 132, and store the resulting data in non-volatile memory 134. According to embodiments, processor 120 may include a main processor 121 (e.g., central processing unit (CPU) or application processor (AP)) or an auxiliary processor 123 (e.g., graphics processing unit (GPU), neural processing unit (NPU), image signal processor (ISP), sensor central processor, or communication processor (CP)) that is operationally independent of or combined with the main processor 121. For example, when electronic device 101 includes a main processor 121 and an auxiliary processor 123, the auxiliary processor 123 may be configured to use less power than the main processor 121 or be configured to be used for a specific function. The auxiliary processor 123 may be implemented separately from the main processor 121 or as part of the main processor 121.

[0028] When the main processor 121 is inactive (e.g., in sleep) state, the auxiliary processor 123 (rather than the main processor 121) can control at least some of the functions or states associated with at least one component of the electronic device 101 (e.g., display module 160, sensor module 176, or communication module 190), or when the main processor 121 is active (e.g., running an application), the auxiliary processor 123 can work with the main processor 121 to control at least some of the functions or states associated with at least one component of the electronic device 101 (e.g., display module 160, sensor module 176, or communication module 190). According to embodiments, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., camera module 180 or communication module 190) functionally associated with the auxiliary processor 123. According to embodiments, the auxiliary processor 123 (e.g., a neural processing unit) may include hardware structures specified for processing artificial intelligence models. Artificial intelligence models can be generated via machine learning. This learning can be performed, for example, by an electronic device 101 performing artificial intelligence or via a separate server (e.g., server 108). The learning algorithm can include, but is not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. The artificial intelligence model can include multiple layers of artificial neural networks. The artificial neural network can be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of these, but is not limited thereto. Additionally or alternatively, the artificial intelligence model can include software structures in addition to hardware structures.

[0029] Memory 130 may store various data used by at least one component of electronic device 101 (e.g., processor 120 or sensor module 176). The various data may include, for example, software (e.g., program 140) and input or output data for commands associated therewith. Memory 130 may include volatile memory 132 or non-volatile memory 134.

[0030] Program 140 may be stored as software in memory 130 and may include, for example, an operating system (OS) 142, middleware 144, or application 146.

[0031] Input module 150 can receive commands or data from outside electronic device 101 (e.g., a user) that will be used by other components of electronic device 101 (e.g., processor 120). Input module 150 may include, for example, a microphone, mouse, keyboard, keys (e.g., buttons), or digital pen (e.g., stylus).

[0032] The audio output module 155 can output audio signals to the outside of the electronic device 101. The audio output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as playing multimedia or playing records. The receiver can be used to receive incoming calls. According to an embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0033] Display module 160 can visually provide information to the outside of electronic device 101 (e.g., to a user). Display 160 may include, for example, a display, a holographic device, or a projector, and control circuitry for controlling a respective one of the display, holographic device, and projector. According to an embodiment, display 160 may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by a touch.

[0034] The audio module 170 can convert sound into electrical signals and vice versa. According to an embodiment, the audio module 170 can obtain sound via the input module 150, or output sound via the sound output module 155 or headphones of an external electronic device (e.g., electronic device 102) that is directly (e.g., wired) connected to the electronic device 101 or wirelessly connected.

[0035] Sensor module 176 can detect the operating state of electronic device 101 (e.g., power or temperature) or the environmental state outside electronic device 101 (e.g., user state), and then generate an electrical signal or data value corresponding to the detected state. According to embodiments, sensor module 176 may include, for example, a gesture sensor, gyroscope sensor, atmospheric pressure sensor, magnetic sensor, accelerometer, grip sensor, proximity sensor, color sensor, infrared (IR) sensor, biometric sensor, temperature sensor, humidity sensor, or illuminance sensor.

[0036] Interface 177 may support one or more specific protocols used to enable electronic device 101 to connect directly (e.g., wired) or wirelessly to external electronic device (e.g., electronic device 102). According to embodiments, interface 177 may include, for example, a High Definition Multimedia Interface (HDMI), a Universal Serial Bus (USB) interface, a Secure Digital Card (SD) interface, or an audio interface.

[0037] Connection terminal 178 may include a connector, through which electronic device 101 can be physically connected to an external electronic device (e.g., electronic device 102). According to embodiments, connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0038] The haptic module 179 can convert electrical signals into mechanical stimuli (e.g., vibration or motion) or electrical stimuli that can be recognized by a user through his touch or kinesthesia. According to embodiments, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0039] Camera module 180 can capture still images or moving images. According to an embodiment, camera module 180 may include one or more lenses, an image sensor, an image signal processor, or a flash.

[0040] The power management module 188 manages the power supply to the electronic device 101. According to an embodiment, the power management module 188 may be implemented as at least part of, for example, a power management integrated circuit (PMIC).

[0041] Battery 189 can power at least one component of electronic device 101. According to an embodiment, battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable rechargeable battery, or a fuel cell.

[0042] Communication module 190 can support establishing a direct (e.g., wired) or wireless communication channel between electronic device 101 and external electronic devices (e.g., electronic device 102, electronic device 104, or server 108), and perform communication via the established communication channel. Communication module 190 may include one or more communication processors that can operate independently of processor 120 (e.g., application processor (AP)) and support direct (e.g., wired) or wireless communication. According to embodiments, communication module 190 may include wireless communication module 192 (e.g., cellular communication module, short-range wireless communication module, or Global Navigation Satellite System (GNSS) communication module) or wired communication module 194 (e.g., local area network (LAN) communication module or power line communication (PLC) module). A corresponding one of these communication modules can communicate via a first network 198 (e.g., a short-range communication network, such as Bluetooth). TM The wireless communication module 192 can communicate with external electronic devices 104 via a Wi-Fi Direct or Infrared Data Association (IrDA) network or a second network 199 (e.g., a long-range communication network such as a traditional cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a local area network (LAN) or a wide area network (WAN))). These various types of communication modules can be implemented as a single component (e.g., a single chip) or as multiple components separate from each other (e.g., multiple chips). The wireless communication module 192 can use subscriber information (e.g., an International Mobile Subscriber Identity (IMSI)) stored in the subscriber identification module 196 to identify or verify electronic devices 101 in a communication network (e.g., a first network 198 or a second network 199).

[0043] Wireless communication module 192 can support 5G networks and next-generation communication technologies, such as New Radio (NR) access technologies, following 4G networks. NR access technologies can support enhanced mobile broadband (eMBB), massive machine-type communications (mMTC), or ultra-reliable and low-latency communications (URLLC). Wireless communication module 192 can support high-frequency bands (e.g., millimeter-wave bands) to achieve, for example, high data transmission rates. Wireless communication module 192 can support various technologies used to ensure performance in high-frequency bands, such as beamforming, massive MIMO, full-dimensional MIMO (FD-MIMO), array antennas, analog beamforming, or massive antennas. Wireless communication module 192 can support various requirements specified in electronic device 101, external electronic device (e.g., electronic device 104), or network system (e.g., second network 199). According to an embodiment, the wireless communication module 192 may support peak data rates (e.g., 20 Gbps or higher) for implementing eMBB, loss coverage (e.g., 164 dB or lower) for implementing mMTC, or U-plane delay (e.g., 0.5 ms or less for each of the downlink (DL) and uplink (UL), or 1 ms or less round trip) for implementing URLLC.

[0044] Antenna module 197 can transmit or receive signals or power to or from an external source (e.g., an external electronic device). According to an embodiment, antenna module 197 may include an antenna comprising a radiator formed of a conductor or conductive pattern formed on a substrate (e.g., a printed circuit board (PCB)). According to an embodiment, antenna module 197 may include multiple antennas (e.g., an antenna array). In this case, at least one antenna suitable for a communication scheme used in a communication network (such as a first network 198 or a second network 199) can be selected from the multiple antennas by, for example, communication module 190. Signals or power can then be transmitted or received between communication module 190 and external electronic device via the selected at least one antenna. According to an embodiment, other components besides the radiator (e.g., a radio frequency integrated circuit (RFIC)) may be further formed as part of antenna module 197.

[0045] According to an embodiment, antenna module 197 can form a millimeter-wave antenna module. According to an embodiment, the millimeter-wave antenna module may include: a printed circuit board; an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high-frequency band (e.g., millimeter-wave band); and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top or side surface) of the printed circuit board and capable of transmitting or receiving signals in the specified high-frequency band.

[0046] At least some of the aforementioned components may be coupled to each other and transmit signals (e.g., commands or data) between them via a peripheral communication scheme (e.g., bus, general purpose input and output (GPIO), serial peripheral interface (SPI) or mobile industrial processor interface (MIPI)).

[0047] According to an embodiment, commands or data can be sent or received between electronic device 101 and external electronic device 104 via server 108 connected to a second network 199. External electronic devices 102 or 104 can each be the same as or a different type of device as electronic device 101. According to an embodiment, all or some operations to be performed at electronic device 101 can be performed at one or more of external electronic devices 102, 104, or 108. For example, if electronic device 101 is required to automatically perform a function or service or to perform a function or service in response to a request from a user or another device, electronic device 101 may request one or more external electronic devices to perform at least a portion of the function or service, instead of running the function or service, or electronic device 101 may request one or more external electronic devices to perform at least a portion of the function or service in addition to running the function or service. Upon receiving a request, one or more external electronic devices may perform at least a portion of the requested function or service, or perform additional functions or services related to the request, and transmit the result of the performance to electronic device 101. Electronic device 101 may provide the result as at least a partial response to the request, with or without further processing of the result. For this purpose, technologies such as cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing can be used. Electronic device 101 can use, for example, distributed computing or mobile edge computing to provide ultra-low latency services. In another embodiment, external electronic device 104 may include Internet of Things (IoT) devices. Server 108 may be an intelligent server using machine learning and / or neural networks. According to embodiments, external electronic device 104 or server 108 may be included in a second network 199. Electronic device 101 can be applied to intelligent services (e.g., smart homes, smart cities, smart cars, or healthcare) based on 5G communication technology or IoT-related technologies.

[0048] Figure 2 This is a perspective view illustrating the internal configuration of a wearable electronic device according to an embodiment of the present disclosure;

[0049] refer to Figure 2 According to embodiments of the present disclosure, the wearable electronic device 200 may include at least one of an optical output module 211, a display component 201, and a camera module 250.

[0050] According to embodiments of the present disclosure, the light output module 211 may include a light source capable of outputting an image and a lens for guiding the image to the display member 201. According to embodiments of the present disclosure, the light output module 211 may include at least one of a liquid crystal display (LCD), a digital mirror device (DMD), liquid crystal on silicon (LCoS), an organic light-emitting diode (OLED), or a micro-light-emitting diode (micro-LED).

[0051] According to embodiments of this disclosure, the display component 201 may include an optical waveguide (e.g., a waveguide). According to embodiments of this disclosure, an image output from the optical output module 211, incident on one end of the optical waveguide, can propagate within the optical waveguide and be provided to the user. According to embodiments of this disclosure, the optical waveguide may include at least one of a diffractive element (e.g., a diffractive optical element (DOE) or a holographic optical element (HOE)) or a reflective element (e.g., a mirror). For example, the optical waveguide may use at least one diffractive element or reflective element to guide the image output from the optical output module 211 to the user's eye.

[0052] According to embodiments of this disclosure, camera module 250 can capture still images and / or moving images. According to embodiments, camera module 250 can be disposed within a lens frame and can be disposed around display member 201.

[0053] According to embodiments of this disclosure, the first camera module 251 can capture and / or identify the trajectory of a user's eyes (e.g., pupil or iris) or gaze. According to embodiments of this disclosure, the first camera module 251 can periodically or non-periodically send information related to the trajectory of the user's eyes or gaze (e.g., trajectory information) to a processor (e.g., ...). Figure 1 According to embodiments of the present disclosure, the first camera module 251 can acquire information related to the movement and positioning of a user's eyes (or multiple eyes), wherein such information can be used (e.g. by the first camera module 251 or the processor 120) to determine and track the user's gaze direction, such as determining the location the user is looking at on an XR screen or in an XR environment.

[0054] According to embodiments of this disclosure, the second camera module 253 can capture external images. For example, the second camera module 253 can capture images (or multiple images) of the surrounding environment of the wearable electronic device.

[0055] According to embodiments of this disclosure, the third camera module 255 can be used for hand detection and tracking, as well as for recognizing user gestures (e.g., hand movements). According to embodiments of this disclosure, the third camera module 255 can be additionally or alternatively used for 3DoF or 6DoF head tracking, position (space, environment) recognition, and / or motion recognition. The second camera module 253 can also be used for hand detection, tracking, and recognizing user gestures. According to embodiments of this disclosure, at least one of the first camera module 251 to the third camera module 255 can be replaced by a sensor module (e.g., a LiDAR sensor). For example, the sensor module may include at least one of a vertical-cavity surface-emitting laser (VCSEL), an infrared sensor, and / or a photodiode.

[0056] Figure 3a and Figure 3b This is a view showing the front and rear surfaces of a wearable electronic device according to an embodiment;

[0057] refer to Figure 3a and Figure 3b In an embodiment, camera modules 311, 312, 313, 314, 315 and 316 and / or depth sensor 317 for obtaining information related to the surrounding environment of the wearable electronic device 300 may be disposed on the first surface 310 of the housing.

[0058] In an embodiment, camera modules 311 and 312 can acquire images relating to the surrounding environment of the wearable electronic device.

[0059] In embodiments, camera modules 313, 314, 315, and 316 can acquire images when a user wears a wearable electronic device. Camera modules 313, 314, 315, and 316 can be used for hand detection, tracking, and recognition of user gestures (e.g., hand movements). Camera modules 313, 314, 315, and 316 can be used for 3DoF or 6DoF head tracking, position (spatial or environmental) recognition, and / or motion recognition. In embodiments, camera modules 311 and 312 can be used for hand detection and for tracking and recognizing user gestures.

[0060] In an embodiment, depth sensor 317 may be configured to send and receive signals reflected from an object and to identify the distance to the object, such as time-of-flight (TOF). As an alternative to or addition to depth sensor 317, camera modules 313, 314, 315, and 316 may identify the distance to the object.

[0061] According to an embodiment, camera modules 325 and 326 and / or display 321 (and / or lens) for facial recognition (e.g., eye tracking) may be disposed on the second surface 320 of the housing.

[0062] In an embodiment, facial recognition camera modules 325 and 326 adjacent to the display can be used to identify the user's face (e.g., track the movement of the user's face) and / or identify and / or track one or both of the user's eyes.

[0063] In an embodiment, the display 321 (and / or lens) may be disposed on the second surface 320 of the wearable electronic device 300. In an embodiment, the wearable electronic device 300 may not include camera modules 315 and 316 among the plurality of camera modules 313, 314, 315, and 316. Although not explicitly stated... Figure 3a and Figure 3b As shown, however, the wearable electronic device 300 may also include Figure 2 At least one of the components shown.

[0064] As described above, according to the embodiments, the wearable electronic device 300 may have a shape factor intended to be worn on a user's head. The wearable electronic device 300 may also include straps and / or wearable components to be secured to a part of the user's body. The wearable electronic device 300 may provide a user experience based on extended reality (e.g., augmented reality, virtual reality, and / or mixed reality) when worn on a user's head.

[0065] Figure 4 This is a block diagram schematically illustrating a wearable electronic device according to an embodiment.

[0066] refer to Figure 4 According to an embodiment, wearable electronic device 401 (e.g., Figure 1 Electronic devices 101 Figure 2 Wearable electronic devices 200 or Figure 3a and Figure 3b Wearable electronic devices 300 may include a camera 410 (e.g., Figure 1 The camera module 180), and at least one processor 420 (e.g., Figure 1 The processor 120), memory 430 (e.g., Figure 1 The memory 130), the display 460 (e.g., Figure 1 The display 160) and communication circuit 490 (e.g., Figure 1 The communication module 190). According to an embodiment, the wearable electronic device 401 may also include at least one sensor (not shown) (e.g., communication module 190). Figure 1 (Sensor module 176).

[0067] According to an embodiment, the wearable electronic device 401 can be implemented as... Figure 1 Electronic devices 101 Figure 2 Wearable electronic devices 200 and Figure 3a and Figure 3b The wearable electronic device 300 is the same as or similar to the wearable electronic device 300. According to embodiments, the wearable electronic device 401 can be implemented as an augmented reality (AR) glasses device, an extended reality (XR) device, a mixed reality (MR) device, a virtual reality (VR) device, or a video see-through (VST) device. However, this is merely an example, and the embodiments of this disclosure can be implemented as various devices.

[0068] According to an embodiment, memory 430 may store at least one instruction enabling at least one operation of wearable electronic device 401. For example, memory 430 may store instructions that, when executed individually or jointly by at least one processor 420, cause wearable electronic device 401 to perform at least one operation described below.

[0069] Traditional wearable electronic devices can provide the ability to copy at least one of multiple objects displayed on a screen on the wearable electronic device's display and paste it to another location. However, traditional wearable electronic devices only offer the ability to copy and paste images obtained using a camera. Therefore, objects pasted onto the screen displayed through the wearable electronic device's display may not display naturally. Furthermore, when the object to be copied has parts obscured by other objects, the object may be pasted onto the screen without restoring the obscured parts.

[0070] The wearable electronic device 401 according to an embodiment can use a generative AI model to provide the function of copying (or selecting) at least one of a plurality of objects displayed on a screen (e.g., an extended reality screen) and pasting it to another location. For example, the generative AI model can be stored in memory 430 or an external electronic device (e.g., a server). For example, when the generative AI model is stored in an external electronic device (e.g., a server), the electronic device 201 can send information about at least one copied object to the external electronic device (e.g., the server) and receive information about modified object images of the corresponding objects from the external electronic device (e.g., the server). For example, the wearable electronic device 401 can use the generative AI model to perform modifications based on the location where at least one copied (selected) object is pasted. For example, when the object to be copied has a portion that is occluded by other objects, the wearable electronic device 401 can use the generative AI model to restore the occluded portion and paste it onto the screen. For example, when the object to be copied is moved to another location in virtual reality space (e.g., when a cut function is performed), the wearable electronic device 401 can use the generative AI model to restore the portion where the object was originally located.

[0071] Therefore, the wearable electronic device 401 can display objects pasted onto the screen to naturally match the entire environment. Furthermore, when displaying augmented reality or virtual reality images in three-dimensional (3D) space, the wearable electronic device 401 can realistically provide the user with the corresponding 3D effect of the augmented reality (AR) or virtual reality (VR) image.

[0072] According to an embodiment, the processor 420 can control the overall operation of the wearable electronic device 401.

[0073] According to an embodiment, based on images (or multiple images) captured using camera 410, processor 420 can display a first screen including at least one object via display 460. For example, processor 420 can display a first screen including multiple objects positioned at a first location via display 460 based on images captured using camera 410. For example, the first screen can be implemented as a screen including augmented reality (AR) images, a screen including virtual reality (VR) images, extended reality (XR) images, or mixed reality (MR) images. Such a screen can be referred to as an extended reality (XR) screen. The object (or multiple objects) can include real objects captured by camera 410 and / or virtual objects generated or rendered by processor 420. The first location can indicate the position of the object or multiple objects on the first screen.

[0074] According to an embodiment, processor 420 can recognize a user's gestures and execute functions specified for the recognized gestures. For example, processor 420 can recognize gestures using the user's body (e.g., the user's hand) based on images obtained using camera 410. For example, a user's gestures may include the shape of the user's hand, the movement of the user's hand, changes in the user's hand, the shape of the user's fingers, or changes in the user's fingers. For example, various gestures may correspond to different functions.

[0075] According to another embodiment, processor 420 can recognize gestures via a controller of wearable electronic device 401. For example, processor 420 can recognize gestures of the controller recognized by camera 410. Alternatively, processor 420 can receive information about the movement of the controller via communication circuitry 490 and recognize the controller's gestures based on the received information. The controller may be part of wearable electronic device 401 or may be a peripheral device. The controller may be configured to be recognized by camera 410 or to communicate with processor 420 via communication circuitry 490 to facilitate user input, for example, via gestures. The controller may also include other user input devices, such as buttons, touchpads, joysticks, microphones, and any other suitable input devices.

[0076] According to an embodiment, based on a user's first gesture, processor 420 can select at least one object displayed at a first location on a first screen for copying. For example, processor 420 can select at least one of a plurality of objects based on recognizing the user's first gesture using camera 410. For example, at least one object can represent an object to be copied to and pasted to another location on the first screen. Alternatively, at least one object can represent an object to be copied to and pasted to a screen different from the first screen displayed via display 460 of wearable electronic device 401. For example, the first gesture can include a gesture of the user opening their hand. For example, processor 420 can recognize or obtain information corresponding to the selected at least one object (e.g., information about an image, the texture of the object, and / or the color of the object). Information about the selected at least one object can be stored in memory 430 (e.g., clipboard).

[0077] According to an embodiment, processor 420 may select at least one object touched by a first gesture (e.g., a user's palm) or indicated by the first gesture (e.g., indicated by the surface of a user's palm) as the object to be copied. For example, processor 420 may select multiple objects (or a group of multiple objects). Alternatively, processor 420 may select a single object.

[0078] According to an embodiment, processor 420 may determine a second location for displaying at least one selected object based on recognizing a second gesture of the user using camera 410 (or based on recognizing a change from a first gesture to a second gesture). For example, the second gesture may be different from the first gesture. For example, the second location may indicate the location on a first screen for displaying at least one selected or copied object. Alternatively, the second location may indicate the location on a screen different from the first screen for displaying at least one selected or copied object. Processor 420 may recognize or obtain attributes of the second location (e.g., coordinate information about the second location, the environment of the second location (e.g., an AR environment or a VR environment), the texture of the second location, the presence of another object at the second location, and / or the slope of the second location).

[0079] According to an embodiment, processor 420 can obtain an object image corresponding to at least one object based on a second gesture using a generative artificial intelligence (AI) model stored in memory 430. For example, processor 420 can use the generative AI model to perform drawing operations (e.g., in-painting and / or out-painting operations) to generate a new image based on at least one of the attributes of the second location and information about the at least one object. Processor 420 can perform in-painting (e.g., drawing operations to generate the parts of the at least one object occluded by another object (e.g., parts of the object not recognized by camera 410) by inputting information about the at least one object and, in some examples, the attributes of the second location into the generative AI model. The in-painting operation can be used to generate parts of the object that are not visible at the first location, such that these parts are visible when the object is displayed at the second location. In some examples, the attributes of the second location can be used to determine which parts of the object will be visible in the second location, such that only those parts are generated by the in-painting operation. Furthermore, processor 420 can perform out-painting (e.g., generating drawing operations for the portion of lighting, shadows, and / or decoration) on at least a portion (e.g., generating drawing operations for the portion of lighting, shadows, and / or decoration) based on the attributes of the second location by inputting the attributes of the second location and information about at least one object into a generative AI model. Out-painting operations can be used to modify the object so that the object visually matches the attributes of the second location. For example, out-painting operations may include generating shadows (or lighting) to be cast by the object in the second location, generating shadows (or lighting) to be cast on the object based on objects in or around the second location, and / or changing the visual style or decoration of the object to match the visual style or decoration of the second location. Processor 420 can obtain an object image corresponding to at least one object by performing at least one of in-painting or out-painting on at least one object using a generative AI model. For example, the object image may include a 3D image. The 3D image may be a 3D representation of the selected at least one object, wherein aspects or portions of the object that are not visible on the first screen when the object is displayed at the first location are generated by processor 420 using a generative AI model.

[0080] According to an embodiment, processor 420 can perform operations to modify at least one selected object based on attributes of a second location. For example, processor 420 can extract attributes of the second location (e.g., attribute values ​​of the environment at the second location) and can modify (e.g., shape, form, orientation, angle, and / or brightness) at least one selected object based on the extracted attributes. For example, processor 420 can perform operations to modify at least one selected object based on the extracted attributes of the second location, as well as the aforementioned in-paint and / or out-paint operations. In some examples, in addition to performing in-paint and / or out-paint operations using a generative AI model, processor 420 can also perform operations to extract attributes of the second location and modify at least one selected object based on the extracted attributes without using a generative AI model. Alternatively, processor 420 can perform operations to modify at least one selected object based on the extracted attributes of the second location (in some examples without using a generative AI model) without performing in-paint and / or out-paint operations.

[0081] According to an embodiment, based on the user's third gesture recognized by camera 410 (or based on the recognition of a change in the third gesture relative to the second gesture), processor 420 can display an object image located at a second position via display 460. For example, processor 420 can display a second screen including the object image set at the second position via display 460. In some examples, the second screen can be defined as the screen displaying the object image at the second position, i.e., the screen where the copy / cut and paste process is completed. For example, the third gesture may be different from the second gesture. The third gesture may be the same as or different from the first gesture. For example, processor 420 can display the object image at a second position on an existing first screen. Alternatively, processor 420 can display the object image at a second position on a screen different from the existing first screen.

[0082] According to an embodiment, based on displaying (or pasting) an object image at a second location, the processor 420 may maintain display of at least one object at a first location (e.g., copy and paste function). According to another embodiment, the processor 420 may display an object image at a second location without displaying at least one object at the first location (e.g., cut function). For example, a third gesture for the copy and paste function may be specified differently from a third gesture for the cut function.

[0083] According to embodiments, generative artificial intelligence (AI) models can use existing content such as text, audio, and / or images to generate new content similar to the corresponding content. For example, a generative AI model can learn patterns in the content to generate new content as a result of inference. For example, a generative AI model can generate (or obtain or output) a new image by performing at least one of an in-painting operation or an out-painting operation on an image. It should be understood that this disclosure is not limited to any particular generative AI model, and any suitable image-based generative AI model can be used to perform in-painting and out-painting operations.

[0084] According to embodiments, an in-painting operation can instruct an action (or function) to generate (or colorize) portions (or unrecognized portions) removed from an image using a generative AI model to match their surrounding environment. Alternatively, an in-painting operation can instruct an action to generate (or colorize) portions of an image occluded by some objects using a generative AI model to match their surrounding environment. For example, an in-painting operation can be used to generate portions of an object that are not visible at a first location, such that these portions are visible when the object is displayed at a second location. In some examples, attributes of the second location can be used to determine which portions of the object will be visible at the second location, such that only those portions are generated by the in-painting operation.

[0085] According to embodiments, outlining operations can indicate or include operations (or functions) that extend the outer region of an image while using a generative AI model to generate (or colorize) an extended portion to match an existing image. For example, outlining operations can be used to modify an object so that the object visually matches the attributes of a second location. For example, outlining operations can include generating shadows (or lighting) to be cast by an object in the second location, generating shadows (or lighting) to be cast on an object based on objects in or around the second location, and / or changing the visual style or decoration of an object to match the visual style or decoration of the second location.

[0086] According to an embodiment, based on displaying (or pasting) an object image at a second location, the processor 420 may maintain display of at least one object at a first location (e.g., copy and paste function). According to another embodiment, the processor 420 may display an object image at a second location without displaying at least one object at the first location (e.g., cut function). For example, a third gesture for the copy and paste function may be specified differently from a third gesture for the cut function.

[0087] According to an embodiment, processor 420 can change the augmented reality environment of the first screen to a virtual reality environment based on recognizing a user's fourth gesture using camera 410 (or based on recognizing a change from a second gesture to a fourth gesture). For example, if the first screen represents an AR environment, the user can use the fourth gesture to change to a screen representing a VR environment. According to an embodiment, when it is recognized that another object (e.g., a real object or a virtual object) exists at a second location in the changed virtual reality environment, processor 420 can display the object image so that it does not overlap with the other object. For example, processor 420 can adjust the position of other objects or object images relative to the second location (or the environment corresponding to the second location). For example, the fourth gesture can be a gesture of a hand different from the user's hand corresponding to the first gesture. For example, the fourth gesture can include a gesture of flipping a hand or a gesture of waving or swiping a hand.

[0088] According to another embodiment, processor 420 can change the virtual reality environment of the first screen to an augmented reality environment based on recognizing a user's fourth gesture using camera 410 (or based on recognizing a change from a second gesture to a fourth gesture). For example, if the first screen represents a VR environment, the user can use the fourth gesture to change to a screen representing an AR environment. According to an embodiment, when another object is detected at a second location in the changed augmented reality environment, processor 420 can display both the other object and an object image at the second location. For example, processor 420 can display an object image to overlap with the other object.

[0089] According to an embodiment, the processor 420 can identify the gaze (e.g., gaze direction) of a user wearing the wearable electronic device 401 using at least one sensor (not shown). For example, the at least one sensor can be implemented as a sensor capable of identifying the gaze by recognizing the eyes (e.g., pupil or iris) of the user wearing the wearable electronic device 401. Alternatively, according to an embodiment, the processor 420 can use a gaze tracking camera (e.g., Figure 2 The first camera module 251 (or the face recognition camera modules 325 and 326 of FIG3) is used to identify the gaze of a user wearing the wearable electronic device 401. According to an embodiment, the processor 420 can select at least one of a plurality of objects included in the first screen to be copied by further considering the user's gaze. For example, based on identifying the user's gaze, the processor 420 can determine which objects included in the first screen the user is looking at and can select the objects accordingly.

[0090] According to an embodiment, the processor 420 can perform operations such as selecting an object to be copied, selecting a location to paste the copied object, rendering an image corresponding to the copied object, and displaying the rendered image at the corresponding location. The specific operations used here are described in detail below.

[0091] At least some of the operations of the wearable electronic device 401 described below can be performed by the processor 420 or a generative AI model. However, for ease of description, the following operations are described as being performed by the wearable electronic device 401.

[0092] Figure 5a This is a flowchart illustrating an operation method of a wearable electronic device according to an embodiment.

[0093] refer to Figure 5a According to an embodiment, in operation 501, a wearable electronic device (e.g., Figure 4 Wearable electronic devices 401 can be accessed via a display (e.g., Figure 4 The display 460) shows a first screen including multiple objects. For example, the wearable electronic device 401 can be based on using a camera (e.g., Figure 4 The wearable electronic device 401 uses images captured by camera 410 to display a first screen comprising multiple objects. For example, the first screen may be an augmented reality screen or an extended reality screen. For example, based on the images obtained using camera 410, the multiple objects may correspond to real objects and / or virtual objects. Alternatively, wearable electronic device 401 may display a virtual first screen (e.g., a virtual reality screen) without using camera 410.

[0094] According to an embodiment, in operation 503, the wearable electronic device 401 can select a specific object displayed at a first position on the first screen based on first user input. For example, the first user input may include a user-specified gesture, input to a button on the wearable electronic device 401 (or its controller), or input to a control menu (e.g., a control menu displayed on the display 460). For example, the wearable electronic device 401 may modify an image corresponding to the selected first object into a 3D object. Alternatively, the wearable electronic device 401 may recognize multiple objects included on the first screen (e.g., all objects included on the screen) as 3D objects, and may recognize a 3D object corresponding to the selected first object 615.

[0095] According to an embodiment, in operation 505, the wearable electronic device 401 can determine a second position for moving a specific object based on second user input. For example, the second position may indicate a position different from a first position on a first screen. Alternatively, the second position may indicate a position where the specific object will be displayed on a screen different from the first screen (e.g., augmented reality screen -> virtual reality screen). For example, the second user input may be different from the first user input. For example, the second user input may include a user-specified gesture, input to a button on the wearable electronic device 401 (or a controller of the wearable electronic device 401), or input to a control menu (e.g., a control menu displayed on the display 460).

[0096] According to an embodiment, in operation 507, the wearable electronic device 401 can use a generative AI model to obtain a modified object for a specific object based on features (or attributes) of a second location. For example, the wearable electronic device 401 can modify a specific object to match the second location. For example, the wearable electronic device 401 can modify a specific two-dimensional (2D) object into a 3D object based on an image obtained by camera 410. The wearable electronic device 401 can modify at least one of the size, brightness, angle, orientation, or shape of the 3D object to match the second location. Furthermore, the wearable electronic device 401 can use a generative AI model to perform in-painting operations (e.g., generating / coloring occluded portions of a specific object) and / or out-painting operations (e.g., adding shadows to a specific object) on a specific object. In addition, the wearable electronic device 401 can perform operations to modify at least one selected object based on attributes of the second location. For example, the processor 420 can extract attributes of the second location (e.g., attribute values ​​of the environment at the second location) and can modify (e.g., shape, form, orientation, angle, and / or brightness) at least one selected object based on the extracted attributes. In some examples, it is possible to perform a modified object based on features (or attributes) of a second location in operation 507 without using a generative AI model. For example, processor 420 can extract attributes of the second location (e.g., attribute values ​​of the environment at the second location) and can modify at least one selected object based on the extracted attributes (e.g., shape, form, orientation, angle, and / or brightness) without using a generative AI model.

[0097] According to an embodiment, in operation 509, the wearable electronic device 401 may display a modified object (e.g., a 3D object) at a second location on the screen. For example, the wearable electronic device 401 may display the modified object at a second location on a first screen. Alternatively, the wearable electronic device 401 may display the modified object at a second location on a screen different from the first screen (e.g., augmented reality screen -> virtual reality screen). In this case, the wearable electronic device 401 may display the modified object at the second location in response to a third user input. For example, the third user input may be different from the second user input. For example, the third user input may include a user-specified gesture, input to a button on the wearable electronic device 401 (or the controller of the wearable electronic device 401), or input to a control menu (e.g., a control menu displayed on the display 460).

[0098] According to an embodiment, the wearable electronic device 401 can modify the corresponding object by considering the theme or dominant atmosphere of the second location (or environment of the second location) where the selected object is to be pasted. For example, the wearable electronic device 401 can extract at least one keyword (e.g., a keyword indicating the theme or dominant atmosphere of the second location) at the second location (or environment of the second location) where the selected object is to be pasted, and can use the extracted keyword to regenerate (e.g., modify) the selected object to suit the environment where the selected object is to be attached. For example, the operation of regenerating (or modifying) the selected object can be performed in operation 507. Thereafter, the wearable electronic device 401 can paste and display the regenerated (or modified) object at the second location.

[0099] Using the methods described above, the wearable electronic device 401 can modify the shape and / or appearance of the object to be copied among multiple objects displayed on the screen to adapt to the characteristics (or attributes) of another location (or space), and can display the modified object (e.g., a 3D object) at the corresponding location (or space). Furthermore, the wearable electronic device 401 can provide the ability to easily and efficiently copy and paste not only objects corresponding to real objects displayed on the screen, but also objects corresponding to virtual objects.

[0100] For ease of description, user input is described as user gestures in the preceding and following text. However, according to embodiments, user input can be implemented as various inputs other than gestures (e.g., voice, input to controller buttons, and input to control menus).

[0101] Figure 5b This is a flowchart illustrating an operation method of a wearable electronic device according to an embodiment.

[0102] refer to Figure 5b According to an embodiment, in operation 551, a wearable electronic device (e.g., Figure 4 Wearable electronic devices 401) can be based on the use of a camera (e.g., Figure 4 The image captured by the camera 410 is used to display a first screen including multiple objects set at a first position. For example, the first position may indicate that multiple objects are set on a display (e.g., a monitor). Figure 4 The position displayed on the first screen (460) of the monitor.

[0103] According to an embodiment, in operation 553, the wearable electronic device 401 can select at least one object to be copied from a plurality of objects included in a first screen based on a user's first gesture. For example, the wearable electronic device 401 can copy at least one selected object via the first gesture. Furthermore, the wearable electronic device 401 can recognize or obtain information about the copied at least one object (e.g., an image indicating the copied at least one object).

[0104] According to an embodiment, in operation 555, the wearable electronic device 401 can determine a second position on the first screen for displaying (or pasting) at least one object based on a user's second gesture. For example, the second position may be different from the first position.

[0105] According to an embodiment, in operation 557, the wearable electronic device 401 can obtain an object image corresponding to at least one object by performing at least one of in-painting or out-painting on an image indicating at least one object to be copied, using a generative AI model. For example, the object image may include an image of a portion of the at least one copied object occluded by another object, generated by performing in-painting. Furthermore, the object image may include an image generated by considering information about a second position (e.g., lighting) in the at least one copied object via out-painting (e.g., a shadow image of at least one selected object). For example, the object image may be rendered as a 3D image.

[0106] According to an embodiment, in operation 559, the wearable electronic device 401 can display a second screen including an image of an object positioned at a second location based on a user's third gesture. For example, when the object to be copied is a virtual object, the wearable electronic device 401 can maintain the display of at least one object at an existing first location on the second screen based on displaying the object image at the second location, or it can choose not to display at least one object at the first location. For example, the wearable electronic device 401 can maintain the display of at least one object at an existing first location on the second screen based on displaying the object image at a second location in a virtual reality space or extended reality space, or it can choose not to display at least one object at the first location. According to an embodiment, based on displaying the object image at a second location in an augmented reality space, the wearable electronic device 401 can maintain the display of at least one object at an existing first location on the second screen, or it can choose not to display at least one object at the first location. For example, in an augmented reality space, the wearable electronic device 401 can control the display not to display a portion of a corresponding object at a first location in an image obtained by a camera.

[0107] Using the above method, wearable electronic device 401 can use a generative AI model to display objects pasted on the screen to naturally match the entire environment.

[0108] Figure 6a This is a view illustrating the functionality of copying and pasting objects displayed on a screen using a wearable electronic device, according to an embodiment.

[0109] refer to Figure 6a According to embodiments, wearable electronic devices (e.g., Figure 4 Wearable electronic devices 401 can be accessed via a display (e.g., Figure 4 The display 460 shows a first screen 610 that includes multiple objects. For example, the wearable electronic device 401 may display the first screen 610 based on images captured by the camera 410. For example, the first screen 610 may include augmented reality images.

[0110] According to an embodiment, the wearable electronic device 401 can use the camera 410 to recognize a user's first gesture 601. For example, the first gesture 601 may include a gesture of opening all of the user's fingers.

[0111] According to an embodiment, the wearable electronic device 401 can recognize a region 611 indicated by a first gesture (e.g., a region indicating a copy range). The wearable electronic device 401 can recognize multiple objects included in the region 611. For example, multiple objects may be displayed at a first location. The wearable electronic device 401 can select a first object 615 from the multiple objects as the object to be copied. For example, the wearable electronic device 401 can determine or select an indicator output from the user's hand (e.g., ...). Figure 10b The object indicated by the indicator 1050 is designated as the first object 615. The wearable electronic device 401 can copy or save the image corresponding to the selected first object 615. For example, the first object 615 can be identified as an object touched by the hand of a user indicating a first gesture or an object indicated by the user's palm. For example, the wearable electronic device 401 can modify the image corresponding to the selected first object into a 3D object. Alternatively, the wearable electronic device 401 can recognize multiple objects included in the first screen (e.g., all objects included in the screen) as 3D objects, and can recognize or obtain the 3D object corresponding to the selected first object 615.

[0112] According to an embodiment, the wearable electronic device 401 can recognize a user's second gesture 602. For example, the second gesture may include a gesture of closing all of the user's fingers. The wearable electronic device 401 can determine a second position for displaying a first object 615 in response to the second gesture. For example, the second position may be determined as a position touched by the hand of a user indicating the second gesture or a position indicated by the user's hand. According to an embodiment, the wearable electronic device 401 can modify at least one selected (or copied) object based on attributes of the second position (e.g., the second position of the corresponding object displayed on the screen via display 460). The wearable electronic device 401 can use a generative AI model to generate a 3D object obtained by changing the attributes of the first object according to the attributes of the second position. For example, the wearable electronic device 401 can generate an object image 625 corresponding to the first object 615 (e.g., an image corresponding to the 3D object obtained by changing the attributes of the first object according to the attributes of the second position). For example, the second gesture may be a gesture that changes after the user's hand moves to the second position. For example, the second gesture may represent the user's hand changing to the second gesture after moving to the second position while maintaining a first gesture. Alternatively, the second gesture can indicate that the user's hand changes from the first position to the second gesture and then moves to the second position.

[0113] According to an embodiment, the wearable electronic device 401 can recognize a user's third gesture 603. For example, the third gesture may include a gesture of opening all of the user's fingers. In response to the third gesture 603, the wearable electronic device 401 can display a second screen 620 via a display 460, including an object image 625 corresponding to the first object 615 positioned at a second location. In this case, the wearable electronic device 401 can display the first object 615 at a first location on the second screen. For example, the third gesture 603 may be a gesture designated for the function of pasting a specific object.

[0114] By reference Figure 6a The described method allows the wearable electronic device 401 to copy a first object 615 and display an object image 625 corresponding to the copied first object 615 at a second location.

[0115] Figure 6b This is a view illustrating the functionality of an object displayed on a screen by a wearable electronic device according to an embodiment.

[0116] refer to Figure 6b ,and Figure 6a In contrast, wearable electronic device 401 can be configured with different specified functions based on the type of third gesture.

[0117] According to an embodiment, the wearable electronic device 401 can recognize a user's third gesture 604. For example, the third gesture may include a gesture of opening two of the user's fingers. In response to the third gesture 603, the wearable electronic device 401 can display a second screen 630 via a display 460, including an object image 625 corresponding to the first object 615 positioned at a second location. In this case, the wearable electronic device 401 may not display the first object 615 at the first location on the second screen. For example, the third gesture 604 may be a gesture designated for the function of cutting a specific object.

[0118] Using the method described above, the wearable electronic device 401 can easily copy objects displayed on the display 460 and paste or cut the copied objects. Therefore, the usability of the wearable electronic device 401 can be increased.

[0119] Figure 7 This is a flowchart illustrating the in-line and out-line drawing performed by a wearable electronic device using a generative AI model on an image representing at least one object, according to an embodiment.

[0120] refer to Figure 7 According to an embodiment, in operation 701, a wearable electronic device (e.g., Figure 4The wearable electronic device 401 can perform in-painting based on information about at least one selected (or copied) object. For example, the wearable electronic device can input an image representing at least one object into a generative AI model to perform in-painting on the image. For example, the wearable electronic device 401 can perform in-painting to generate an image corresponding to a portion of the at least one object that is occluded by another object. Furthermore, when at least one object is cut, the wearable electronic device 401 can perform in-painting on the cut portion and generate an image corresponding to that portion.

[0121] According to an embodiment, wearable electronic device 401 can modify at least one selected (or copied) object based on attributes of a second location (e.g., a second location on a screen where the corresponding object is displayed via display 460), and can display the modified at least one object at the second location. According to an embodiment, in operation 703, wearable electronic device 401 can perform outlining based on attributes of the second location (e.g., a second location on the screen displayed via display 460) of the at least one selected (or copied) object to be displayed (or pasted). For example, the wearable electronic device can perform outlining on an image representing at least one object by inputting the attributes of the second location into a generative AI model. For example, wearable electronic device 401 can perform outlining based on an image corresponding to at least one object to generate an image associated with the second location (e.g., an image corresponding to shadows and / or lighting associated with the second location). For example, at least one of the size, direction, or brightness of the shadow of at least one object at the second location may differ from at least one of the size, direction, or brightness of the shadow of at least one object at the first location.

[0122] According to an embodiment, in operation 705, the wearable electronic device 401 can render an object image corresponding to at least one object by performing in-painting and / or out-painting. For example, the object image can be implemented as a 3D image.

[0123] According to an embodiment, in operation 707, the wearable electronic device 401 can display an image of an object at a second location via a display 460 in response to a third gesture.

[0124] Figure 8 The illustration shows in-line and out-line drawing performed by a wearable electronic device using a generative AI model on an image representing at least one object, according to an embodiment.

[0125] refer to Figure 8 According to embodiments, wearable electronic devices (e.g., Figure 4The wearable electronic device 401 can select or determine a first object (e.g., an object corresponding to a real or virtual object) displayed on a first screen as the object to be copied in response to user input (e.g., gesture or input to a button). For example, when the first object is an object corresponding to a real object, the wearable electronic device 401 can select or determine the object to be copied based on a camera (e.g., ...). Figure 4 The wearable electronic device 401 can use images obtained by camera 410 to identify a 2D image corresponding to a first object. The wearable electronic device 401 can use a generative AI model to transform the 2D image corresponding to the first object into a 3D object or to identify the 3D object. For example, the wearable electronic device can transform the 2D image corresponding to the first object into a 3D object and identify it based on user input identified for manipulating the first object (e.g., user input requesting a copy of the first object).

[0126] According to an embodiment, the first image 810 corresponding to the selected first object may include an unidentified portion 815 (or an inaccurately identified portion) that is occluded by another object.

[0127] According to an embodiment, a generative AI model can generate (e.g., colorize) an image corresponding to an unidentified portion 815 of the first image 810 by performing an in-painting operation on the first image 810. In other words, the generative AI model can generate an image (e.g., pixel values ​​of the corresponding portion) of the deleted or unidentified (or occluded) portion 815 within the first image 810. The generative AI model can generate and output a second image 820 including the image generated by the in-painting operation.

[0128] According to an embodiment, a generative AI model can perform an outlining operation on a first image 810 based on attributes of a second location of a first object to be displayed, to generate (e.g., colorize) an image corresponding to the portion of the first image 810 to be expanded. In other words, the generative AI model can generate an image (e.g., pixel values ​​of the corresponding portion) of the portion to be expanded beyond the first image 810. The generative AI model can generate and output an object image including a third image 830 generated through the outlining operation.

[0129] According to an embodiment, a generative AI model can generate and output object images (e.g., images including second image 820 and third image 830) by performing both in-line drawing and out-line drawing operations.

[0130] According to an embodiment, in response to a third gesture, the wearable electronic device 401 can display an object image generated (or rendered) using a generative AI model at a second location.

[0131] Using the above method, wearable electronic device 401 can configure the screen to naturally match the entire environment by displaying object images generated using a generative AI model on the screen.

[0132] Figure 9 This is a flowchart illustrating a method for selecting at least one object by a wearable electronic device according to an embodiment.

[0133] refer to Figure 9 According to an embodiment, in operation 901, a wearable electronic device (e.g., Figure 4 Wearable electronic devices 401 can recognize a first gesture through a display (e.g., Figure 4 The area to be copied is displayed on a first screen (display 460). For example, the first gesture may include a gesture of opening all of the user's fingers. In this case, the wearable electronic device 401 can determine the area indicated by the user's palm as the area to be copied.

[0134] According to an embodiment, in operation 903, the wearable electronic device 401 can identify the distance between a user's hand and an object located in the area to be copied by analyzing an image captured using camera 410. For example, in operation 903, the wearable electronic device 401 can identify whether the distance between the user's hand and the object located in the area to be copied is shorter than a specified distance. For example, the specified distance can be specified as a distance indicating that the user's hand is touching or substantially touching a real object.

[0135] According to an embodiment, when it is identified that the distance between the user's hand and an object located in the area to be copied is less than a specified distance ("Yes" in operation 903), in operation 905, the wearable electronic device 401 can select or determine at least one object touched by the user's hand as the object to be copied. For example, when the corresponding object is a real object, the wearable electronic device 401 can use camera 410 to identify whether the user's hand is touching the real object. Alternatively, when the corresponding object is a virtual object, the wearable electronic device 401 can identify whether the image representing the user's hand is touching the virtual object or within a specified distance.

[0136] According to an embodiment, when it is determined that the distance between the user's hand and an object located in the area to be copied is not less than a specified distance (No in operation 903), in operation 907, the wearable electronic device 401 can select or determine at least one object indicated by the user's hand as the object to be copied. For example, a pointer displayed relative to the user's hand or palm can be used to select or determine the object to be copied.

[0137] According to an embodiment, in operation 909, the wearable electronic device 401 can display information (e.g., a preview image) about the selected at least one object via a display 460. For example, the wearable electronic device 401 can display at least one preview image corresponding to the at least one object on a first screen. Alternatively, the wearable electronic device 401 can display an indicator (e.g., an indicator using color, solid lines, dashed lines, and / or shading) on ​​the first screen to distinguish the selected at least one object from other objects. For example, the indicator can be displayed on at least one selected object.

[0138] According to another embodiment, the wearable electronic device 401 can display at least one preview image corresponding to at least one object included in the area to be copied on a first screen. Thereafter, the wearable electronic device 401 can recognize user input on the at least one preview image to select or determine the object to be copied.

[0139] According to an embodiment, in operation 911, the wearable electronic device 401 can determine the object to be copied and can identify, store, or copy information about the object (e.g., an image indicating the object to be copied). According to an embodiment, the wearable electronic device 401 can ultimately select the object to be copied based on user input to at least one preview image (e.g., additional user input or the user's gaze). For example, the image representing the object to be copied may include a portion occluded by another object. According to an embodiment, at least one preview image associated with the object to be copied may be an image of the portion occluded by another object generated through an in-paint operation.

[0140] Figure 10a and Figure 10b This is a view illustrating a method for selecting at least one object by a wearable electronic device according to an embodiment.

[0141] refer to Figure 10a According to embodiments, wearable electronic devices (e.g., Figure 4 Wearable electronic devices 401 can be accessed via a display (e.g., Figure 4 The display 460 shows a first screen 1010 that includes multiple objects. For example, the wearable electronic device 401 may display the first screen 1010 based on images captured by the camera 410. For example, the first screen 1010 may include augmented reality images.

[0142] According to an embodiment, the wearable electronic device 401 can identify an area 1011 to be copied in a first screen in response to a first gesture 1001 of the user (e.g., a gesture of opening all of the user's fingers) recognized by the camera 410. The wearable electronic device 401 can identify objects included in the area 1011 to be copied.

[0143] According to an embodiment, the wearable electronic device 401 can display preview images 1016, 1017, and 1018 corresponding to objects included in the region 1011 to be copied. The wearable electronic device 401 can recognize user input (e.g., input using at least one of a user's gesture or gaze) for selecting a first preview image 1016 from the preview images 1016, 1017, and 1018. The wearable electronic device 401 can determine a first object corresponding to the first preview image as the object to be copied based on the user input. According to an embodiment, the wearable electronic device 401 can select multiple preview images based on user input. For example, when multiple preview images are selected, the wearable electronic device 401 can determine multiple objects as multiple target objects.

[0144] According to another embodiment, the preview image can display multiple objects forming a group. When a user selects a corresponding preview image based on user input, the wearable electronic device 401 can identify the multiple objects as those to be copied.

[0145] According to another embodiment, when a user's gaze is directed at the first preview image 1016, the wearable electronic device 401 can recognize a first gesture 1001. In response to recognizing the first gesture 1001, the wearable electronic device 401 can determine a first object corresponding to the first preview image 1016 to which the user's gaze is directed as the object to be copied.

[0146] According to an embodiment, the wearable electronic device 401 may display a notification on a first screen 1030 for identifying a selected first object. For example, an object 1021 that can distinguish the first object from other objects may be displayed on the first object displayed on the first screen 1030.

[0147] According to an embodiment, the wearable electronic device 401 can determine the location for displaying the copied first object in response to recognizing a second gesture 1002 (e.g., a gesture of closing all of the user's fingers or making a fist). For example, while maintaining the second gesture 1002, the wearable electronic device 401 can determine the location for displaying the copied first object as the area where the user's hand is moving or the area where the user's hand is located. Subsequently, the wearable electronic device 401 can display the copied first object at the corresponding location in response to recognizing a third gesture (e.g., a gesture of opening all of the user's fingers).

[0148] refer to Figure 10b According to an embodiment, the wearable electronic device 401 may display a pointer 1050 on a first screen 1010 and / or 1020, which is relative to the user's hand (or palm) 1030. The wearable electronic device 401 may use the pointer 1050 to select or determine the object to be copied.

[0149] Using the above method, the wearable electronic device 401 can easily and effectively select or determine the object to be copied from the first screen.

[0150] Figure 11 This is a flowchart illustrating a method for determining the position of at least one selected object by a wearable electronic device according to an embodiment.

[0151] refer to Figure 11 According to an embodiment, in operation 1101, after selecting at least one object to be copied, the wearable electronic device 401 can use a camera (e.g., Figure 4 The camera 410 detects or recognizes the user's second gesture.

[0152] According to an embodiment, in operation 1103, the wearable electronic device 401 can identify the environment in which at least one object is to be pasted or displayed. For example, the wearable electronic device 401 can identify whether the environment in which at least one object is to be pasted or displayed is an augmented reality environment or a virtual reality environment.

[0153] According to an embodiment, in operation 1105, the wearable electronic device 401 can confirm a second location to be pasted or displayed for at least one object, and can confirm coordinate information about the second location.

[0154] According to an embodiment, in operation 1107, the wearable electronic device 401 can identify attributes of the second location (e.g., lighting, texture, and / or the presence of another object). For example, the wearable electronic device 401 can identify the attributes of the second location by considering the environment in which at least one object is to be pasted or displayed (e.g., an augmented reality environment or a virtual reality environment). For example, in an augmented reality environment, the wearable electronic device 401 can obtain information about the actual lighting identified using camera 410 as information about the attributes of the second location. Alternatively, in a virtual reality environment, the wearable electronic device 401 can obtain information about the rendered virtual lighting as information about the attributes of the second location. For example, when the environment in which at least one object is to be pasted or displayed is a virtual reality environment (or virtual space), the wearable electronic device 401 can identify the overall theme of the corresponding virtual reality environment (e.g., oil painting style, watercolor style, web style, or Gogh style) and can use a generative AI model to modify at least one object based on the identified theme of the virtual reality environment. For example, the wearable electronic device 401 can modify at least one of the shape, color, orientation, slope, or size of at least one object such that at least one object fits or matches the corresponding theme.

[0155] According to an embodiment, the wearable electronic device 401 can perform outlining operations on at least one selected image by inputting attributes of a second location into a generative AI model. For example, based on lighting information about the second location, shadows or lighting effects can be further added to the object image.

[0156] Figure 12a and Figure 12b A method for determining the location of at least one selected object by a wearable electronic device, according to an embodiment, is shown.

[0157] refer to Figure 12a According to embodiments, wearable electronic devices (e.g., Figure 4 Wearable electronic devices 401 can be accessed via a display (e.g., Figure 4 The display 460) shows a first screen 1210 including augmented reality images. For example, the wearable electronic device 401 can use a camera (e.g., Figure 4 Images obtained by the camera are used to display the first screen 1210, which includes augmented reality images.

[0158] According to an embodiment, in response to recognizing a fourth gesture, the wearable electronic device 401 can change a first screen 1210 including augmented reality images to a second screen 1220 including virtual reality images. Furthermore, in response to recognizing a fourth gesture, the wearable electronic device 401 can change the second screen 1220 including virtual reality images to the first screen 1210 including augmented reality images.

[0159] According to an embodiment, after selecting at least one object to be copied from the objects included in the first screen 1210, the wearable electronic device 401 can change the first screen 1210 to a second screen 1220 when a fourth gesture is recognized. In this case, the wearable electronic device 401 can identify a second location on the second screen 1220 for displaying (or pasting) the copied at least one object. When identifying the attributes of the second location, the wearable electronic device 401 can consider a virtual reality image corresponding to the virtual reality environment.

[0160] refer to Figure 12bAccording to an embodiment, the wearable electronic device 401 can recognize a fourth gesture and a second gesture. The wearable electronic device 401 can determine a second location for displaying at least one copied object based on the second gesture, and can change the environment based on the fourth gesture to display the corresponding object. In other words, at least one copied object can be displayed (or pasted) on a screen corresponding to another environment (e.g., a second screen 1220). For example, the second gesture may include a gesture of the user's one hand (e.g., right hand) 1250 making a fist. For example, the fourth gesture may be a gesture of a hand different from the hand of the second gesture (e.g., left hand) 1230. For example, the fourth gesture may be different from the second gesture.

[0161] refer to Figure 12b (a) According to an embodiment, the fourth gesture may include a gesture of flipping the user's left hand 1230 when the user's left hand 1230 is open. Reference Figure 12b (b) According to another embodiment, the fourth gesture may include a gesture of waving or sweeping the user's left hand 1230 while the user's left hand 1230 is open.

[0162] According to an embodiment, the wearable electronic device 401 can easily change the environment in which the copied object is to be displayed. Furthermore, the wearable electronic device 401 can generate an object image taking into account the environment in which the copied object is to be displayed, and can display the generated object image to match the corresponding environment.

[0163] Figure 13 This is a flowchart illustrating a method for editing an object image by a wearable electronic device according to an embodiment.

[0164] refer to Figure 13 According to an embodiment, in operation 1301, a wearable electronic device (e.g., Figure 4 Wearable electronic devices 401 can be accessed via a display (e.g., Figure 4 The display (460) shows a second screen including an image of an object positioned at a second location, and control objects for the object image can then be displayed on the second screen. For example, the control objects may include at least one object for adjusting (or editing) at least one of the following: position, orientation, size, or scale of the object image. For example, the control objects may be displayed around the object image.

[0165] According to an embodiment, in operation 1303, the wearable electronic device 401 can adjust or edit at least one of the position, orientation, size, or scale of an object image based on user input to the controlled object. For example, when user input for adjusting at least one of the position, orientation, size, or scale of the object image is recognized, the wearable electronic device 401 can render the adjusted object image and display the rendered object image on a second screen.

[0166] Figure 14 A method for editing an object image by a wearable electronic device according to an embodiment is shown.

[0167] refer to Figure 14 Wearable electronic devices (e.g.) Figure 4 Wearable electronic devices 401 can be accessed via a display (e.g., Figure 4 The display (460) shows an object image 1410 corresponding to the copied first object. For example, the object image 1410 can be implemented as a 3D image.

[0168] According to an embodiment, the wearable electronic device 401 can display a 3D object 1420 corresponding to the object image 1410. The wearable electronic device 401 can change the position of the object image 1410 based on user input to the 3D object 1420.

[0169] According to an embodiment, the control object 1430 may be displayed around the object image 1410. For example, the control object 1430 may include an object for redirecting, enlarging (or scaling up) and reducing the size of the object image. For example, the wearable electronic device 401 may change the orientation and size (or scale) of the object image 1410 based on user input to the control object 1430. For example, when user input for enlarging an object in the control object 1430 is recognized, the wearable electronic device 401 may display a magnified object image 1450 of the object image 1410.

[0170] After displaying the object image 1401 at the second position using the above method, the wearable electronic device 401 can provide a user interface that allows easy adjustment of the size, scale, orientation, or position of the corresponding object image.

[0171] Figure 15 This is a flowchart illustrating a method for displaying an image of an object by a wearable electronic device according to an embodiment.

[0172] refer to Figure 15 According to an embodiment, in operation 1501, a wearable electronic device (e.g., Figure 4 The wearable electronic device 401 can recognize a third gesture. For example, the third gesture could be a gesture used to display a copied object at a location desired by the user (e.g., a second location).

[0173] According to an embodiment, in operation 1503, the wearable electronic device 401 can identify whether there is another object overlapping the object image at the second location.

[0174] According to an embodiment, when it is identified that there are no other objects overlapping with the object image at the second location (No in operation 1503), in operation 1511, the wearable electronic device 401 can display the object image at the second location.

[0175] According to an embodiment, when another object overlapping with the object image is identified at a second location ("Yes" in operation 1503), in operation 1505, the wearable electronic device 401 can identify whether the object overlapping the object image is a real object. For example, a real object may refer to a real object identified by camera 410. For example, an object that is not a real object may include a virtual object, such as a virtual object generated by wearable electronic device 401.

[0176] According to an embodiment, when it is identified that the object overlapping with the object image is not a real object ("No" in operation 1505), in operation 1507, the wearable electronic device 401 can display the object image so that it does not overlap with the corresponding object. For example, when a virtual object exists at a second location, the wearable electronic device 401 can set and display the object image such that the virtual object and the object image do not overlap. According to an implementation, the wearable electronic device 401 can set and display virtual objects to overlap based on user input or preset conditions. When virtual objects are set to overlap, and one virtual object is displayed in front of another virtual object, the wearable electronic device 401 may not display a portion of the other virtual object (the portion obscured by one virtual object). The wearable electronic device 401 can display a specific virtual object in front of or behind another virtual object according to preset conditions.

[0177] According to an embodiment, when it is identified that the object overlapping with the object image is a real object ("Yes" in operation 1505), in operation 1509, the wearable electronic device 401 can display the object image as if it at least partially overlaps with the object. For example, when a real object is present at a second location, the wearable electronic device 401 can set and display the object image such that the image representing the real object and the object image overlap each other. For example, the wearable electronic device 401 can display at least a portion of the object image in front of or behind the image representing the real object.

[0178] Figure 16 This is a view illustrating a method for displaying an image of an object by a wearable electronic device according to an embodiment.

[0179] refer to Figure 16 According to embodiments, wearable electronic devices (e.g., Figure 4The wearable electronic device 401 can select and copy an object 1615 displayed at a first location included in a first screen 1610 corresponding to an augmented reality (AR) environment based on a user's first gesture. According to an embodiment, the wearable electronic device 401 can display or paste the copied object 1615 at a second location in a virtual reality (VR) environment. The wearable electronic device 401 can display an object image 1625 corresponding to the object 1615, such that the virtual object set at the second location and the object image 1625 corresponding to the copied object 1615 do not overlap. The wearable electronic device 401 can display a second screen 1625 in the virtual reality environment, in which the object image 1625 is set not to overlap with another virtual object.

[0180] According to another embodiment, the wearable electronic device 401 can select and copy a virtual object 1625 displayed at a second location on a second screen 1620 corresponding to a virtual reality environment. According to the embodiment, the wearable electronic device 401 can display or paste the copied virtual object 1625 at a first location in the augmented reality (AR) environment. The wearable electronic device 401 can display an object image 1615 corresponding to the virtual object 1625, such that a real object (e.g., a monitor) positioned at the first location and the object image 1615 corresponding to the copied virtual object 1625 overlap each other. The wearable electronic device 401 can display a first screen 1615 in which the object image 1615 is configured not to overlap with another real object in the augmented reality environment.

[0181] Based on the above method, wearable electronic devices can adjust the arrangement of the corresponding objects by taking into account the environment in which the copied objects are pasted.

[0182] According to an embodiment, a wearable electronic device 401, worn on a user's head, may include a camera 410, a display 460, at least one processor 420, and a memory 430 storing instructions. The instructions, when executed by at least one processor, enable the wearable electronic device to display a first screen, including a plurality of objects arranged at a first position, based on images captured using the camera. The instructions, when executed by at least one processor, enable the wearable electronic device to select at least one object to be copied from among the plurality of objects based on a first gesture of the user. The instructions, when executed by at least one processor, enable the wearable electronic device to determine a second position on the first screen for displaying at least one object based on a second gesture of the user. The instructions, when executed by at least one processor, enable the wearable electronic device to perform at least one of in-line drawing of the portion of the at least one object occluded by at least one other object or out-line drawing of the at least one object based on the attributes of the second position, by providing attributes of the second position and information about the at least one object to a generative AI model stored in memory, and to obtain an object image corresponding to the at least one object by performing at least one of in-line drawing or out-line drawing. The instructions, when executed by at least one processor, enable the wearable electronic device to display a second screen, including an image of an object located at the second position, based on a third gesture of the user.

[0183] According to an embodiment, when executed by at least one processor, the instructions enable a wearable electronic device to display an object image at a second location based on a copy of an image corresponding to at least one object.

[0184] According to an embodiment, when executed by at least one processor, the instructions can enable a wearable electronic device to hold or remove at least one object displayed at a first position on a second screen based on a third gesture.

[0185] According to an embodiment, when instructions are executed by at least one processor, the instructions can cause a wearable electronic device to in-draw a portion of at least one object that is occluded by at least one other object, in order to obtain an object image that includes an image corresponding to the occluded portion.

[0186] According to an embodiment, when the instructions are executed by at least one processor, the instructions may enable a wearable electronic device to perform outlining on at least one object based on at least one of lighting information or texture information at a second location to obtain an object image, the object image including an image corresponding to at least one of shadow or lighting associated with the at least one object.

[0187] According to an embodiment, when executed by at least one processor, the instructions enable a wearable electronic device to change the augmented reality environment of a first screen into a virtual reality environment based on a user's fourth gesture. According to an embodiment, when executed by at least one processor, the instructions enable the wearable electronic device to display an object image that does not overlap with another object when it recognizes that another object exists at a second location in the virtual reality environment.

[0188] According to an embodiment, when executed by at least one processor, the instructions can cause a wearable electronic device to change the virtual reality environment of a first screen into an augmented reality environment based on a user's fourth gesture. According to an embodiment, when executed by at least one processor, the instructions can cause the wearable electronic device to display an object image overlapping another object when it recognizes the presence of another object at a second location in the augmented reality environment.

[0189] According to an embodiment, when executed by at least one processor, the instructions enable the wearable electronic device to display a control object on a second screen for adjusting at least one of the position, orientation, size, or scale of the object image after displaying an object image at a second location. According to an embodiment, when executed by at least one processor, the instructions enable the wearable electronic device to adjust the object image based on user input regarding the control object.

[0190] According to an embodiment, when an instruction is executed by at least one processor, the instruction can cause a wearable electronic device to select at least one object from a plurality of objects that is touched or indicated by a first gesture.

[0191] According to an embodiment, when executed by at least one processor, the instructions enable a wearable electronic device to display at least one preview image representing at least one of a plurality of objects on a first screen based on a first gesture.

[0192] According to an embodiment, a method for operating a wearable electronic device 401 worn on a user's head may include: displaying a first screen, comprising a plurality of objects arranged at a first location, via a display 460 included in the wearable electronic device, based on an image captured using a camera 410 included in the wearable electronic device. According to an embodiment, a method for operating the wearable electronic device may include: selecting at least one object to be copied from the plurality of objects based on a first gesture of the user. According to an embodiment, a method for operating the wearable electronic device may include: determining a second location on the first screen for displaying at least one object based on a second gesture of the user. According to an embodiment, a method for operating the wearable electronic device may include: providing attributes of the second location and information about at least one object stored in a memory 430 included in the wearable electronic device; performing at least one of in-painting or out-painting of the at least one object based on the attributes of the second location; and obtaining an object image corresponding to the at least one object by performing at least one of in-painting or out-painting. According to an embodiment, a method for operating the wearable electronic device may include: displaying a second screen, comprising an image of an object located at the second location, via a display based on a third gesture of the user.

[0193] According to an embodiment, displaying a second screen may include: displaying an object image at a second location based on a copy of an image corresponding to at least one object.

[0194] According to an embodiment, displaying a second screen may include: based on a third gesture, maintaining or removing at least one object displayed at a first position on the second screen.

[0195] According to an embodiment, obtaining an object image may include drawing the portion of at least one object that is occluded by at least one other object to obtain an object image that includes an image corresponding to the occluded portion.

[0196] According to an embodiment, obtaining an object image may include outlining at least one object based on at least one of lighting information or texture information at a second location to obtain an object image including an image corresponding to at least one of shadows or lighting associated with at least one object.

[0197] According to an embodiment, the method for operating a wearable electronic device may further include: changing the augmented reality environment of a first screen to a virtual reality environment based on a user's fourth gesture. According to an embodiment, the method for operating a wearable electronic device may further include: when another object is detected at a second location in the virtual reality environment, displaying an image of the object so that it does not overlap with the other object.

[0198] According to an embodiment, the method for operating a wearable electronic device may further include: changing the virtual reality environment of a first screen to an augmented reality environment based on a user's fourth gesture. According to an embodiment, the method for operating a wearable electronic device may further include: when another object is detected at a second location in the augmented reality environment, displaying an object image as overlapping with the other object.

[0199] According to an embodiment, the method for operating a wearable electronic device may further include: after displaying an object image at a second location, displaying a control object on a second screen for adjusting at least one of the position, orientation, size, or scale of the object image. According to an embodiment, the method for operating a wearable electronic device may further include: adjusting the object image based on user input to the control object.

[0200] According to an embodiment, the method for operating a wearable electronic device may further include: displaying at least one preview image representing at least one of a plurality of objects on a first screen based on a first gesture.

[0201] According to an embodiment, the non-transitory recording medium 130 or 430 may store instructions to execute: displaying a first screen including a plurality of objects arranged at a first position via a display 460 included in the wearable electronic device 401, based on an image captured using an image captured by a camera 410 included in the wearable electronic device 401 worn on a user's head; selecting at least one object to be copied from the plurality of objects based on a first gesture of the user; and determining a second position on the first screen for displaying at least one object based on a second gesture of the user. By providing the attributes of the second position and information about at least one object to a generative AI model stored in a memory 430 included in the wearable electronic device, performing at least one of in-line drawing of the portion of at least one object occluded by at least one other object or out-line drawing of at least one object based on the attributes of the second position, and obtaining an object image corresponding to at least one object by performing at least one of in-line drawing or out-line drawing, and displaying a second screen including an object image located at the second position via a display based on a third gesture of the user.

[0202] The electronic device according to various embodiments can be one of a variety of types of electronic devices. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer equipment, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. According to embodiments of this disclosure, the electronic device is not limited to those described above.

[0203] It should be understood that the various embodiments of this disclosure and the terminology used therein are not intended to limit the technical features set forth herein to the specific embodiments, but rather to include various changes, equivalents, or substitutions to the respective embodiments. In the description of the drawings, similar reference numerals may be used to refer to similar or related elements. It will be understood that nouns in the singular form corresponding to terms may include one or more things unless the relevant context clearly indicates otherwise. As used herein, each of the phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B, or C,” “at least one of A, B, and C,” and “at least one of A, B, or C” may include any one or all possible combinations of the items enumerated together with the corresponding phrase among the plurality of phrases. As used herein, terms such as “first” and “second” or “first” and “second” may be used to simply distinguish the respective component from another component and do not limit the components in other respects (e.g., importance or order). It will be understood that, whether the terms “operably” or “communically” are used or not, if an element (e.g., a first element) is referred to as “combined with another element (e.g., a second element),” “combined to another element (e.g., a second element),” “connected to another element (e.g., a second element),” or “connected to another element (e.g., a second element)”, it means that the element can be directly (e.g., wiredly) connected to the other element, wirelessly connected to the other element, or connected to the other element via a third element.

[0204] As used in connection with various embodiments of this disclosure, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with other terms such as "logic," "logic block," "part," or "circuit." A module may be a single integrated component adapted to perform one or more functions, or the smallest unit or part of such a single integrated component. For example, according to embodiments, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0205] The various embodiments set forth herein can be implemented as software (e.g., program 1440) including one or more instructions readable by a machine (e.g., electronic device 1401) stored in a storage medium (e.g., internal memory 1436 or external memory 1438). For example, under the control of a processor, the processor (e.g., processor 1420) of the machine (e.g., electronic device 1401) can invoke and execute at least one of the one or more instructions stored in the storage medium, with or without the use of one or more other components. This enables the machine to operate to perform at least one function according to the invoked at least one instruction. The one or more instructions may include code generated by a compiler or code that can be run by an interpreter. Machine-readable storage media may be provided in the form of non-transitory storage media. The term "non-transitory" means only that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but this term does not distinguish between data being stored semi-permanently in the storage medium and data being temporarily stored in the storage medium.

[0206] According to embodiments, methods according to various embodiments of this disclosure may be included and provided in a computer program product. The computer program product can be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)) or via an app store (e.g., the Play Store). TM The computer program product may be published online (e.g., downloaded or uploaded), or may be distributed directly between two user devices (e.g., smartphones) (e.g., downloaded or uploaded). If published online, at least a portion of the computer program product may be temporarily generated, or at least a portion of the computer program product may be temporarily stored in a machine-readable storage medium (such as the memory of a manufacturer's server, an app store's server, or a forwarding server).

[0207] According to various embodiments, each of the above-described components (e.g., a module or program) may include a single entity or multiple entities, and some of the multiple entities may be separately disposed in different components. According to various embodiments, one or more of the above-described components may be omitted, or one or more other components may be added. Optionally or additionally, multiple components (e.g., modules or programs) may be integrated into a single component. In this case, according to various embodiments, the integrated component may still perform the one or more functions of each of the multiple components in the same or similar manner as a corresponding component of the multiple components performed one or more functions prior to integration. According to various embodiments, the operations performed by a module, program, or other component may be performed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations may be run in a different order or omitted, or one or more other operations may be added.

Claims

1. A wearable electronic device (401) that can be worn on a user's head, the wearable electronic device comprising: Camera (410); Display (460); At least one processor (420); and A memory (430) storing instructions that, when executed by the at least one processor, enable the wearable electronic device to: Based on images captured using the camera, a first screen is displayed on the monitor, comprising a plurality of objects arranged at a first location; Based on the user's first gesture, select at least one object to be copied from the plurality of objects; Based on the user's second gesture, a second position on the first screen for displaying the at least one object is determined; By providing the attributes of the second location and information about the at least one object to the generative AI model stored in the memory, at least one of in-line drawing of the portion of the at least one object occluded by at least one other object or out-line drawing of the at least one object is performed based on the attributes of the second location, and an object image corresponding to the at least one object is obtained by performing at least one of the in-line drawing or the out-line drawing. and Based on the user's third gesture, a second screen is displayed via the display, including an image of an object located at the second position.

2. The wearable electronic device according to claim 1, wherein, When executed by the at least one processor, the instruction enables the wearable electronic device to: The object image is displayed at the second location by copying the image corresponding to the at least one object.

3. The wearable electronic device according to claim 1 or 2, wherein, When executed by the at least one processor, the instruction enables the wearable electronic device to: Based on the third gesture, the at least one object displayed at the first position on the second screen is maintained or removed.

4. The wearable electronic device according to any one of claims 1 to 3, wherein, When executed by the at least one processor, the instruction enables the wearable electronic device to: The in-drawing is performed on the portion of the at least one object that is occluded by the at least one other object to obtain an image of the object that includes an image corresponding to the occluded portion.

5. The wearable electronic device according to any one of claims 1 to 4, wherein, When executed by the at least one processor, the instruction enables the wearable electronic device to: Based on at least one of the lighting information or texture information at the second location, outlining is performed on the at least one object to obtain an object image, the object image including an image corresponding to at least one of the shadows or lighting associated with the at least one object.

6. The wearable electronic device according to any one of claims 1 to 5, wherein, When executed by the at least one processor, the instruction enables the wearable electronic device to: Based on the user's fourth gesture, the augmented reality environment of the first screen is changed into a virtual reality environment; and When another object is identified as existing at the second location in the virtual reality environment, the image of the object is displayed so as not to overlap with the other object.

7. The wearable electronic device according to any one of claims 1 to 6, wherein, When executed by the at least one processor, the instruction enables the wearable electronic device to: Based on the user's fourth gesture, the virtual reality environment of the first screen is changed into an augmented reality environment; and When another object is identified as existing at the second location in the augmented reality environment, the image of the object is displayed as overlapping with the other object.

8. The wearable electronic device according to any one of claims 1 to 7, wherein, When executed by the at least one processor, the instruction enables the wearable electronic device to: After displaying the object image at the second location, a control object for adjusting at least one of the position, orientation, size, or scale of the object image is displayed on the second screen; and The image of the object is adjusted based on user input regarding the controlled object.

9. The wearable electronic device according to any one of claims 1 to 8, wherein, When executed by the at least one processor, the instruction enables the wearable electronic device to: Select at least one of the plurality of objects that is touched by the first gesture or indicated by the first gesture.

10. The wearable electronic device according to any one of claims 1 to 9, wherein, When executed by the at least one processor, the instruction enables the wearable electronic device to: Based on the first gesture, at least one preview image representing at least one of the plurality of objects is displayed on the first screen.

11. A method for operating a wearable electronic device (401) worn on a user's head, the method comprising: Based on images captured using a camera (410) included in the wearable electronic device, a first screen comprising a plurality of objects arranged at a first position is displayed via a display (460) included in the wearable electronic device; Based on the user's first gesture, select at least one object to be copied from the plurality of objects; Based on the user's second gesture, a second position on the first screen for displaying the at least one object is determined; By providing the attributes of the second location and information about the at least one object to a generative AI model stored in a memory (430) included in the wearable electronic device, at least one of in-line drawing of the portion of the at least one object occluded by at least one other object or out-line drawing of the at least one object is performed based on the attributes of the second location, and an object image corresponding to the at least one object is obtained by performing at least one of the in-line drawing or the out-line drawing. and Based on the user's third gesture, a second screen is displayed via the display, including an image of an object located at the second position.

12. The method according to claim 11, wherein, Displaying the second screen includes: The object image is displayed at the second location by copying the image corresponding to the at least one object.

13. The method according to claim 11 or 12, wherein, Displaying the second screen includes: Based on the third gesture, the at least one object displayed at the first position on the second screen is maintained or removed.

14. The method according to any one of claims 11 to 13, wherein, Obtaining the object image includes: The in-drawing is performed on the portion of the at least one object that is occluded by the at least one other object to obtain an image of the object that includes an image corresponding to the occluded portion.

15. A non-transitory recording medium (430) for storing instructions, said instructions causing said at least one processor, when executed by said at least one processor, to: Based on images captured using a camera (410) included in a wearable electronic device (401) worn on a user's head, a first screen comprising a plurality of objects arranged at a first location is displayed via a display (460) included in the wearable electronic device. Based on the user's first gesture, select at least one object to be copied from the plurality of objects; Based on the user's second gesture, a second position on the first screen for displaying the at least one object is determined; By providing the attributes of the second location and information about the at least one object to a generative AI model stored in a memory (430) included in the wearable electronic device, at least one of in-line drawing of the portion of the at least one object occluded by at least one other object or out-line drawing of the at least one object is performed based on the attributes of the second location, and an object image corresponding to the at least one object is obtained by performing at least one of the in-line drawing or the out-line drawing. and Based on the user's third gesture, a second screen is displayed via the display, including an image of an object located at the second position.

Images