Methods and devices for obtaining additional object on basis of source object and target object

Abstract

This electronic device comprises a display, at least one processor including a processing circuit, and a memory comprising one or more storage media storing instructions, wherein the instructions, when executed individually or collectively by the at least one processor, instruct the electronic device to: select a source object and a target object from among objects placed in a space; extract a prompt representing an attribute of the source object and occupancy information of the target object within the space; obtain, on the basis of the prompt and the occupancy information, an additional object that has the attribute of the source object and a shape that matches at least part of the outline of the target object; and display the additional object at a position aligned along the at least part of the outline of the target object through the display.

Description

Method and device for obtaining additional objects based on source objects and target objects

[0001] Hereinafter, a technique for obtaining additional objects based on source objects and target objects is disclosed.

[0002] Recently, virtual reality (VR), augmented reality (AR), and mixed reality (MR) technologies that utilize computer graphics technology are being developed. VR technology uses computers to create virtual spaces that do not exist in the real world and then make them feel real, while AR or MR technologies superimpose computer-generated information on top of the real world. In other words, they combine the real and virtual worlds to enable real-time user interaction.

[0003] Among these, AR and MR technologies are being integrated and utilized in various fields (e.g., broadcasting, medical, and gaming). Representative examples of AR being integrated into broadcasting include the natural shifting of a weather map in front of a weathercaster giving a weather forecast on TV, or the insertion of non-existent advertising images onto the screen during a sports broadcast, as if they were actually present in the stadium.

[0004] A representative service that provides users with augmented reality or mixed reality is the metaverse. This metaverse, a portmanteau of "meta," meaning "fictional" or "abstract," and "universe," signifying the real world, refers to a three-dimensional virtual world. A more advanced concept than the existing term "virtual reality environment," the metaverse offers an augmented reality environment where the virtual world of the web and the internet is integrated into the real world.

[0005] An electronic device may include a display. The electronic device may include at least one processor including a processing circuit. The electronic device may include a memory including one or more storage media storing instructions. When the instructions are individually or collectively executed by the at least one processor, the instructions may cause the electronic device to select a source object and a target object from among objects arranged in a space. When the instructions are executed by the processor, the electronic device may cause the electronic device to extract a prompt indicating an attribute of the source object and occupancy information of the target object within the space. When the instructions are executed by the processor, the electronic device may obtain an additional object having a shape according to the attribute while matching at least a portion of an outline of the target object based on the prompt and the occupancy information. When the above instructions are executed by the processor, the electronic device may cause the display to display the additional object at a location aligned along at least a portion of the outline of the target object.

[0006] A method performed by an electronic device may include an operation of selecting a source object and a target object from among objects arranged in a space. The method performed by the electronic device may include an operation of extracting a prompt indicating an attribute of the source object and occupancy information of the target object within the space. The method performed by the electronic device may include an operation of obtaining an additional object having a shape according to the attribute while matching at least a portion of an outline of the target object based on the prompt and the occupancy information. The method performed by the electronic device may include an operation of displaying the additional object at a position aligned along at least a portion of the outline of the target object through a display.

[0007] FIG. 1 is a block diagram illustrating an exemplary configuration of an electronic device according to various embodiments.

[0008] FIG. 2 illustrates an example of an optical see-through device according to various embodiments.

[0009] FIG. 3 illustrates examples of optical systems for an eye tracking camera, a transparent member, and a display according to various embodiments.

[0010] FIGS. 4A and 4B are drawings showing examples of the front and back of an electronic device according to various embodiments.

[0011] FIG. 5 illustrates examples of construction of a virtual space, input from a user within the virtual space, and output to the user according to various embodiments.

[0012] FIG. 6 is a diagram illustrating an example of an operation of an electronic device providing space to a user according to various embodiments.

[0013] FIG. 7 is a flowchart illustrating an example of a method for an electronic device to acquire an additional object according to various embodiments.

[0014] FIG. 8 is a diagram illustrating an example of an operation of an electronic device according to various embodiments to obtain and display an additional object fitted to a target object.

[0015] FIG. 9 is a diagram illustrating an example of an operation in which an electronic device according to various embodiments displays different additional objects depending on the operational state of a target object.

[0016] FIG. 10 is a diagram illustrating an example of an operation of an electronic device according to various embodiments to obtain and display a replacement object that replaces a target object.

[0017] FIG. 11 is a diagram illustrating example outputs of an object creation model according to various embodiments.

[0018] FIGS. 12A and 12B are diagrams illustrating examples of operations in which an electronic device, according to various embodiments, acquires an object selected from among additional objects and replacement objects based on user input.

[0019] FIG. 13 is a diagram illustrating an example of an operation of an electronic device according to various embodiments to stop or re-display an additional object or a replacement object based on a distance.

[0020] FIG. 14 is a diagram illustrating an example of an operation of an electronic device according to various embodiments to stop or re-display an additional object or a replacement object based on a layout.

[0021] FIG. 15 is a diagram illustrating an example of an operation of an electronic device selecting multiple target objects according to various embodiments.

[0022] FIG. 16 is a block diagram illustrating an example configuration of an electronic device according to various embodiments.

[0023] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are assigned the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted.

[0024] FIG. 1 is a block diagram illustrating an exemplary configuration of an electronic device according to various embodiments.

[0025] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0026] The processor (120) may control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing, for example, software (e.g., program (140)), and may perform various data processing or operations. The processor (120) may include at least one processor including a processing circuit. According to one embodiment, as at least a part of the data processing or operation, the processor (120) may store a command or data received from another component (e.g., sensor module (176) or communication module (190)) in a volatile memory (132), process the command or data stored in the volatile memory (132), and store the resulting data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0027] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0028] The memory (130) can store various data used by at least one component (e.g., the processor (120) or the sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., the program (140)) and input data or output data for commands related thereto. The memory (130) can include a volatile memory (132) or a non-volatile memory (134). According to one embodiment, the memory (130) can include one or more storage media that store instructions that cause the operation of the electronic device (101).

[0029] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0030] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0031] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0032] A display module (160) (e.g., a display) can visually provide information to an external device (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0033] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0034] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0035] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0036] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0037] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0038] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0039] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0040] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0041] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data relation (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0042] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0043] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0044] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0045] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0046] According to one embodiment, commands or data may be transmitted or received between an electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199).

[0047] Each of the external electronic devices (102, 103) and the server (108) may be the same type of device as or different from the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 103) or the server (108). For example, when the electronic device (101) needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of executing the function or service itself or in addition, request one or more external electronic devices to execute at least a part of the function or service. The one or more external electronic devices that receive the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a part of a response to the request. In this specification, an example is mainly described in which an electronic device (101) is an augmented reality device (e.g., an electronic device (201) of FIG. 2, an electronic device (301) of FIG. 3, or an electronic device (401) of FIG. 4), and an external electronic device (102, 103) or a server (108) among the servers transmits the results of executing a virtual space and additional functions or services related to the virtual space to the electronic device (101).

[0048] The server (108) may include a processor (181), a communication module (182), and a memory (183). The processor (181), the communication module (182), and the memory (183) may be configured similarly to the processor (120), the communication module (190), and the memory (130) of the electronic device (101). For example, the processor (181) may provide a virtual space and interaction between users within the virtual space by executing instructions stored in the memory (183). The processor (181) may generate at least one of visual information, auditory information, or tactile information of the virtual space and objects within the virtual space. For example, as visual information, the processor (181) may generate rendering data (e.g., visual rendering data) that renders the appearance (e.g., shape, size, color, or texture) of the virtual space and the appearance (e.g., shape, size, color, or texture) of objects positioned within the virtual space. In addition, the processor (181) may generate rendering data that renders a change (e.g., a change in the appearance of an object, a sound generation, or a tactile generation) based on at least one of an interaction between objects (e.g., a physical object, a virtual object, or an avatar object) in a virtual space, or a user's input to an object (e.g., a physical object, a virtual object, or an avatar object). The communication module (182) may establish communication with a first electronic device (e.g., an electronic device (101)) of a user and a second electronic device (e.g., an electronic device (102)) of another user. The communication module (182) may transmit at least one of the visual information, the tactile information, or the auditory information described above to the first electronic device and the second electronic device. For example, the communication module (182) may transmit rendering data.

[0049] For example, the server (108) renders content data executed in an application and transmits it to the electronic device (101), and the electronic device (101) that receives the data can output the content data to the display module (160). For example, when the electronic device (101) detects user movement through an IMU sensor, the processor (120) of the electronic device (101) can correct the rendering data received from the external electronic device (102) based on the movement information and output it to the display module (160). Alternatively, the movement information can be transmitted to the server (108) to request rendering so that the screen data is updated accordingly. However, the present invention is not limited thereto, and the aforementioned rendering can be performed by various forms of external electronic devices (102, 103), such as a smartphone or a case device that can store and charge the electronic device (101). Rendering data corresponding to the aforementioned virtual space generated by the external electronic device (102, 103) can be provided to the electronic device (101). For example, the electronic device (101) may receive virtual space information (e.g., vertex coordinates, textures, and / or colors defining the virtual space) and object information (e.g., vertex coordinates, textures, and / or colors defining the appearance of an object) from the server (108), and perform rendering on its own based on the received data.

[0050] However, in various embodiments of the present disclosure, the server (108) is not limited to transmitting the result of executing a virtual space and an additional function or service related to the virtual space to the electronic device (101), and the electronic device (101), which is an augmented reality device (e.g., the electronic device (201) of FIG. 2, the electronic device (301) of FIG. 3, or the electronic device (401) of FIG. 4), may also execute a virtual space and an additional function or service related to the virtual space.

[0051] FIG. 2 illustrates an example of an optical see-through device according to various embodiments.

[0052] The electronic device (201) may include at least one of a display (e.g., a display module (160) of FIG. 1), a vision sensor, a light source (230a, 230b), an optical element, or a substrate. An electronic device (201) that has a transparent display and provides an image through the transparent display may be referred to as an optical see-through device (OST device).

[0053] The display may include, for example, a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED).

[0054] In one embodiment, when the display is formed of one of a liquid crystal display (LCD), a digital mirror display (DMD), or a silicon liquid crystal display (SLCD), the electronic device (201) may include a light source (230a, 230b) that irradiates light to a screen output area of ​​the display (e.g., a screen display unit (215a, 215b)). In another embodiment, when the display can generate light on its own, for example, when it is formed of one of an organic light emitting diode (OLED) or a micro LED, the electronic device (201) may provide a good quality virtual image to the user even without including a separate light source (230a, 230b). In one embodiment, if the display is formed of an organic light emitting diode (OLED) or a micro LED, the light source (230a, 230b) is unnecessary, and thus the electronic device (201) may be lightweight.

[0055] Referring to FIG. 2, the electronic device (201) may include a display, a first transparent member (225a) and / or a second transparent member (225b), and a user may use the electronic device (201) while wearing it on his or her face. The first transparent member (225a) and / or the second transparent member (225b) may be formed of a glass plate, a plastic plate, or a polymer, and may be manufactured to be transparent or translucent. According to one embodiment, the first transparent member (225a) may be arranged to face the user's right eye, and the second transparent member (225b) may be arranged to face the user's left eye. The display may include a first display (205) that outputs a first image (e.g., a right image) corresponding to the first transparent member (225a), and a second display (210) that outputs a second image (e.g., a left image) corresponding to the second transparent member (225b). In one embodiment, if each display is transparent, each display and transparent member may be positioned at a position facing the user's eyes to form a screen display unit (215a, 215b).

[0056] In one embodiment, light emitted from the display (205, 210) may be guided along an optical path through an input optical member (220a, 220b) into a waveguide. Light traveling within the waveguide may be guided toward a user's eyes through an output optical member (e.g., the output optical member (340) of FIG. 3). The screen display units (215a, 215b) may be determined based on the light emitted toward the user's eyes.

[0057] For example, light emitted from the display (205, 210) may be reflected in the grating region of the waveguide formed in the input optical member (220a, 220b) and the screen display portions (215a, 215b) and transmitted to the user's eyes.

[0058] The optical element may include at least one of a lens or an optical waveguide.

[0059] The lens can adjust the focus so that the screen output to the display can be viewed by the user. The lens may include, for example, at least one of a Fresnel lens, a pancake lens, or a multi-channel lens.

[0060] An optical waveguide can transmit image light generated from a display to a user's eyes. For example, the image light may represent light emitted by a light source (230a, 230b) that passes through a screen output area of ​​the display. The optical waveguide can be made of glass, plastic, or polymer. The optical waveguide can include a nano-pattern formed on a portion of an internal or external surface, for example, a grating structure having a polygonal or curved shape. An exemplary structure of the optical waveguide is described below in FIG. 3.

[0061] The vision sensor may include at least one of a camera sensor or a depth sensor.

[0062] The first camera (265a, 265b) is a recognition camera, and may be used for 3DoF, 6DoF head tracking, hand detection, hand tracking, and spatial recognition. The first camera (265a, 265b) may mainly include a GS (global shutter) camera. Since a stereo camera is required for head tracking and spatial recognition, the first camera (265a, 265b) may include two or more GS cameras. The GS camera may have superior performance compared to the RS (rolling shutter) camera in terms of detecting rapid hand movements and fine finger movements and tracking movements. For example, the GS camera may have low image blur. The first camera (265a, 265b) may capture image data used for spatial recognition for 6DoF and SLAM function through depth shooting. Additionally, a user gesture recognition function can be performed based on image data captured by the first camera (265a, 265b).

[0063] The second camera (270a, 270b) is an ET (eye tracking) camera that can be used to capture image data for detecting and tracking the user's pupils. The second camera (270a, 270b) is described later in FIG. 3.

[0064] The third camera (245) may be a camera for photography. The third camera (245) may include a high-resolution camera for capturing HR (high resolution) or PV (photo video) images. The third camera (245) may include a color camera equipped with functions for obtaining high-quality images, such as an autofocus (AF) function and optical image stabilization (OIS). The third camera (245) may be a GS camera or an RS camera.

[0065] The fourth camera unit (e.g., the face recognition camera (428) of FIG. 4 below) is a face recognition camera, and the FT (face tracking) camera can be used to detect and track the user's facial expression.

[0066] A depth sensor (not shown) may be a sensor that senses information for determining the distance to an object, such as Time of Flight (TOF). TOF is a technology that measures the distance to an object using signals (e.g., near-infrared, ultrasound, or laser). A depth sensor based on TOF technology can measure the time of flight of a signal by transmitting a signal and measuring the signal at a receiver.

[0067] The light source (230a, 230b) (e.g., illumination module) may include a device (e.g., light emitting diode (LED)) that irradiates light of various wavelengths. The illumination module may be attached to various locations depending on the application. As an example, a first illumination module (e.g., LED device) attached around the frame of the augmented reality glasses device may emit light to assist in gaze detection when tracking eye movements with an ET camera. The first illumination module may include, for example, an IR LED of an infrared wavelength. As another example, a second illumination module (e.g., LED device) may be attached adjacent to a camera mounted around a hinge (240a, 240b) connecting the frame and the temple, or around a bridge connecting the frames. The second illumination module may emit light to supplement the ambient brightness when the camera is taking pictures. If it is difficult to detect a subject in a dark environment, the second illumination module may emit light.

[0068] A substrate (235a, 235b) (e.g., a printed circuit board (PCB)) can support the aforementioned components.

[0069] A printed circuit board (PCB) may be disposed on the temple portion of the glasses. The FPCB may transmit electrical signals to each module (e.g., a camera, a display, an audio module, a sensor module) and other printed circuit boards. In one embodiment, at least one printed circuit board may include a first substrate, a second substrate, and an interposer disposed between the first substrate and the second substrate. Electrical signals may be transmitted to each module and other printed circuit boards.

[0070] Other components may include, for example, at least one of a plurality of microphones (e.g., a first microphone (250a), a second microphone (250b), a third microphone (250c)), a plurality of speakers (e.g., a first speaker (255a), a second speaker (255b)), a battery (260), an antenna, or a sensor (e.g., an acceleration sensor, a gyro sensor, or a touch sensor).

[0071] FIG. 3 illustrates examples of optical systems for an eye tracking camera, a transparent member, and a display according to various embodiments.

[0072] FIG. 3 is a diagram for explaining the operation of an eye-tracking camera included in an electronic device according to one embodiment. Referring to FIG. 3, a process is illustrated in which an eye-tracking camera (310) of an electronic device (301) according to one embodiment (e.g., the second camera (270a, 270b) of FIG. 2) tracks a user's eye (309), or in other words, the user's gaze, using light (e.g., infrared light) output from a display (320) (e.g., the first display (205) of FIG. 2, the second display (210) of FIG. 2).

[0073] The eye tracking camera (310) (e.g., the second camera (270a, 270b) of FIG. 2) may be a camera that collects information for positioning the center of a virtual image projected onto the electronic device (301) according to the direction in which the pupil of the wearer of the electronic device (301) is looking. The eye tracking camera (310) (e.g., the second camera (270a, 270b) of FIG. 2) may include a GS camera to detect the pupil and track rapid eye movement. The eye tracking camera (310) (e.g., the second camera (270a, 270b) of FIG. 2) may be installed for the left eye and the right eye, respectively, and the same camera performance and specifications may be used for each. The eye tracking camera (310) may include an eye tracking sensor (315). The eye tracking sensor (315) may be included inside the eye tracking camera (310). Infrared light output from the display (320) can be transmitted to the user's eye (309) as infrared reflected light (303) by the half mirror. The eye tracking sensor (315) can detect the infrared transmitted light (305) reflected from the user's eye (309) by the infrared reflected light (303). The eye tracking camera (310) can track the user's eye (309), or in other words, the user's gaze, based on the detection result of the eye tracking sensor (315).

[0074] The display (320) may include a plurality of visible light pixels and a plurality of infrared pixels. The visible light pixels may include R, G, and B pixels. The visible light pixels may output visible light corresponding to a virtual object image. The infrared pixels may output infrared light. The display (320) may include, for example, micro light emitting diodes (LEDs) or organic light emitting diodes (OLEDs).

[0075] The display optical waveguide (350) and the eye-tracking optical waveguide (360) may be included inside a transparent member (370) (e.g., the first transparent member (225a) and the second transparent member (225b) of FIG. 2). The transparent member (370) may be formed of a glass plate, a plastic plate, or a polymer, and may be manufactured to be transparent or translucent. The transparent member (370) may be positioned to face the user's eyes. At this time, the distance between the transparent member (370) and the user's eyes (309) may be referred to as 'eye relief' (380).

[0076] The transparent member (370) may include optical waveguides (350, 360). The transparent member (370) may include an input optical member (330) and an output optical member (340). In addition, the transparent member (370) may include an eye-tracking splitter (375) that separates input light into multiple waveguides.

[0077] According to one embodiment, light incident on one end of the display light pipe (350) can be propagated inside the display light pipe (350) by the nano-pattern and provided to the user. In addition, the display light pipe (350) composed of a free-form prism can provide image light to the user through the reflected mirror by reflecting the incident light. The display light pipe (350) can include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) or at least one reflective element (e.g., a reflective mirror). The display light pipe (350) can guide display light (e.g., image light) emitted from a light source to the user's eyes by using at least one diffractive element or reflective element included in the display light pipe (350). For reference, also, in FIG. 3, the output optical member (340) is expressed as being separate from the eye-tracking optical waveguide (360), but the output optical member (340) may be included inside the eye-tracking optical waveguide (360).

[0078] According to various embodiments, the diffractive element may include an input optical member (330) and an output optical member (340). For example, the input optical member (330) may mean an input grating region. The output optical member (340) may mean an output grating region. The input grating region may serve as an input terminal that diffracts (or reflects) light output from (e.g., a Micro LED) to transmit the light to a transparent member (e.g., a first transparent member, a second transparent member) of a screen display unit. The output grating region may serve as an outlet that diffracts (or reflects) light transmitted to a transparent member (e.g., a first transparent member, a second transparent member) of a waveguide to a user's eye.

[0079] According to various embodiments, the reflective element may include a total internal reflection (TIR) ​​optical element or a total internal reflection waveguide. For example, total internal reflection may refer to a method of guiding light such that light (e.g., a virtual image) entering through an input grating region is 100% reflected from one surface (e.g., a specific surface) of the waveguide, thereby transmitting 100% of the light to the output grating region.

[0080] In one embodiment, light emitted from the display (320) may be guided along an optical path through an input optical element (330) into a waveguide. Light traveling within the waveguide may be guided toward the user's eyes through an output optical element (340). The screen display may be determined based on the light emitted toward the user's eyes.

[0081] FIGS. 4A and 4B are diagrams illustrating examples of the front and back of an electronic device according to various embodiments. FIG. 4A may be an external appearance of the electronic device (401) when viewed in a first direction (①), and FIG. 4B may be an external appearance of the electronic device (401) when viewed in a second direction (②). When a user wears the electronic device (401), the external appearance viewed by the user's eyes may be FIG. 4B.

[0082] Referring to FIG. 4A, according to various embodiments, an electronic device (401) (e.g., the electronic device (101) of FIG. 1, the electronic device (201) of FIG. 2, and the electronic device (301) of FIG. 3) may provide a service that provides an extended reality (XR) experience to a user. For example, XR or an XR service may be defined as a service that collectively refers to virtual reality (VR), augmented reality (AR), and / or mixed reality (MR).

[0083] According to one embodiment, the electronic device (401) may refer to a head-mounted device or a head-mounted display worn on the user's head, but may also be configured in the form of at least one of glasses, goggles, a helmet, or a hat. The electronic device (401) may include an OST (optical see-through) type configured to allow external light to reach the user's eyes through the glasses when worn, or a VST (video see-through) type configured to allow light emitted from the display to reach the user's eyes but block external light so that external light does not reach the user's eyes when worn.

[0084] According to one embodiment, the electronic device (401) may be worn on the user's head and may provide the user with an image related to an extended reality (XR) service. For example, the electronic device (401) may provide XR content (hereinafter, also referred to as 'XR content image') that outputs at least one virtual object to be superimposed on a display area or an area determined to be the user's field of view (FoV). According to one embodiment, the XR content may refer to an image or image in which at least one virtual object is superimposed on an image related to a real space acquired through a camera (e.g., a camera for taking pictures) or a virtual space. According to one embodiment, the electronic device (401) may provide XR content based on a function being performed by the electronic device (401) and / or a function being performed by one or more external electronic devices (e.g., the electronic devices (102, 104) of FIG. 1, the server (108) of FIG. 1).

[0085] According to one embodiment, the electronic device (401) is at least partially controlled by an external electronic device (e.g., electronic devices (102 or 104) of FIG. 1), and may perform at least one function under the control of the external electronic device, but may also perform at least one function independently.

[0086] Referring to FIG. 4A, a vision sensor may be placed on a first surface of a housing of a main body (410) of an electronic device (401). The vision sensor may include cameras (e.g., cameras for second functions (411, 412), cameras for first functions (415)) and / or a depth sensor (417) for obtaining information related to the surrounding environment of the electronic device (401).

[0087] In one embodiment, the second function cameras (411, 412) can obtain images related to the surrounding environment of the electronic device (401). The first function cameras (415) can obtain images when the wearable electronic device is worn by the user. The first function cameras (415) can be used for hand detection, hand tracking, and / or user gesture (e.g., hand movement) recognition. The first function cameras (415) can be used for 3DoF, 6DoF head tracking, position (spatial, environmental) recognition, and / or movement recognition. In one embodiment, the second function cameras (411, 412) can also be used for hand detection and tracking, and user gesture.

[0088] In one embodiment, the depth sensor (417) may be configured to transmit a signal and receive a signal reflected from a subject, and may be used for purposes such as time of flight (TOF) to determine the distance to an object. Instead of or in addition to the depth sensor (417), cameras (411, 412, 415) may determine the distance to an object.

[0089] Referring to FIG. 4b, a camera (428) and / or a display (421) (and / or a lens) for facial recognition may be placed on the second surface (420) of the housing of the main body (410).

[0090] In one embodiment, a facial recognition camera (428) adjacent to the display may be used to recognize a user's face, or may recognize and / or track both eyes of the user.

[0091] In one embodiment, the display (421) (and / or lens) may be disposed on the second side (420) of the electronic device (401). In one embodiment, the electronic device (401) may not include some of the plurality of cameras (415). Although not shown in FIGS. 4A and 4B , the electronic device (401) may further include at least one of the configurations illustrated in FIG. 2 .

[0092] According to one embodiment, the electronic device (401) may include a main body (410) that mounts at least some of the components of FIG. 1, a display (421) (e.g., a display module (160) of FIG. 1) disposed in a first direction (①) of the main body (410), a first function camera (e.g., a recognition camera) (415) disposed in a second direction (②) of the main body (410), a second function camera (e.g., a shooting camera) (411, 412) disposed in a second direction (②), a third function camera (e.g., an eye tracking camera) (425, 426) disposed in the first direction (①), a fourth function camera (e.g., a face recognition camera) (428) disposed in the first direction (①), a depth sensor (417) disposed in the second direction (②), and a touch sensor (413) disposed in the second direction (②). Although not shown in the drawing, the main body (410) includes a memory (e.g., memory (130) of FIG. 1) and a processor (e.g., processor (120) of FIG. 1), and may further include other components shown in FIG. 1.

[0093] According to one embodiment, the display (421) may include a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED).

[0094] In one embodiment, when the display (421) is formed of one of a liquid crystal display (LCD), a digital mirror display (DMD), or a silicon liquid crystal display (SiLCD), the electronic device (401) may include a light source that irradiates light to a screen output area of ​​the display (421). In another embodiment, when the display (421) can generate light on its own, for example, when the electronic device (401) is formed of one of an organic light emitting diode (OLED) or a micro LED, the electronic device (401) may provide a good quality XR content image to the user even without including a separate light source. In one embodiment, when the display (421) is formed of an organic light emitting diode (OLED) or a micro LED, a light source is unnecessary, and thus the electronic device (401) may be lightweight.

[0095] According to one embodiment, the display (421) may include a first transparent member (421a) and / or a second transparent member (421b). The user may use the electronic device (401) while wearing it on his or her face. The first transparent member (421a) and / or the second transparent member (421b) may be formed of a glass plate, a plastic plate, or a polymer, and may be manufactured to be transparent or translucent. According to one embodiment, the first transparent member (421a) may be arranged to face the user's left eye in the fourth direction (④), and the second transparent member (421b) may be arranged to face the user's right eye in the third direction (③). According to various embodiments, when the display (421) is transparent, it may be arranged at a position facing the user's eyes to form a display area.

[0096] According to one embodiment, the display (421) may include a lens including a transparent waveguide. The lens may serve to adjust the focus so that the screen (e.g., XR content image) output to the display (421) can be viewed by the user's eyes. For example, light emitted from the display panel may pass through the lens and be transmitted to the user through a waveguide formed within the lens. The lens may be configured as a Fresnel lens, a pancake lens, or a multi-channel lens.

[0097] An optical waveguide (e.g., a waveguide) may serve to transmit light generated by the display (421) to the user's eyes. The optical waveguide may be made of glass, plastic, or a polymer, and may include nano-patterns formed on a portion of the inner or outer surface, for example, a grating structure having a polygonal or curved shape. According to one embodiment, light incident on one end of the optical waveguide, that is, an output image of the display (421), may be propagated within the optical waveguide and provided to the user. In addition, an optical waveguide composed of a free-form prism may provide the incident light to the user through a reflective mirror. The optical waveguide may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) or at least one reflective element (e.g., a reflective mirror). The optical waveguide may guide an image output from the display (421) to the user's eyes by using at least one diffractive element or reflective element included in the optical waveguide.

[0098] According to one embodiment, the diffractive element may include an input optical member / output optical member (not shown). For example, the input optical member may mean an input grating region, and the output optical member (not shown) may mean an output grating region. The input grating region may serve as an input terminal that diffracts (or reflects) light output from a light source (e.g., a Micro LED) to transmit the light to a transparent member (e.g., a first transparent member (421a), a second transparent member (421b)) of the display area. The output grating region may serve as an outlet that diffracts (or reflects) light transmitted to a transparent member (e.g., a first transparent member, a second transparent member) of the optical waveguide to a user's eye.

[0099] In some embodiments, the reflective element may comprise a total internal reflection (TIR) ​​optical element or waveguide for total internal reflection. For example, total internal reflection may refer to a method of directing light such that light (e.g., a virtual image) entering through an input grating region is reflected substantially 100% from one surface (e.g., a specific surface) of the optical waveguide, thereby transmitting substantially 100% of the light to the output grating region at an angle of incidence.

[0100] In one embodiment, light emitted from the display (421) may be guided along an optical path through an input optical element into a waveguide. Light traveling within the optical waveguide may be guided toward the user's eyes through an output optical element. The display area may be determined based on the light emitted toward the user's eyes.

[0101] According to one embodiment, the electronic device (401) may include a plurality of cameras. For example, the cameras may include a first function camera (e.g., a recognition camera) (415) disposed in the second direction (②) of the main body (410), a second function camera (e.g., a shooting camera) (411, 412) disposed in the second direction (②), a third function camera (e.g., an eye tracking camera) (425, 426) disposed in the first direction (①), and / or a fourth function camera (e.g., a face recognition camera) (428) disposed in the first direction (①), but may further include cameras for other functions not shown.

[0102] The first function camera (e.g., recognition camera) (415) can be used for the purpose of detecting user movement or recognizing user gestures. The first function camera (415) can support at least one of head tracking, hand detection and hand tracking, and spatial recognition. For example, the first function camera (415) mainly uses a GS (global shutter) camera, which has superior performance compared to an RS (rolling shutter) camera, to detect and track fine movements of hand movements and fingers, and can be configured as a stereo camera including two or more GS cameras for head tracking and spatial recognition. The first function camera (415) can perform a SLAM (simultaneous localization and mapping) function to recognize information (e.g., location and / or direction) related to the surrounding space through spatial recognition for 6DoF and depth shooting.

[0103] A second function camera (e.g., a camera for shooting) (411, 412) can be used to capture the outside and generate an image or video corresponding to the outside and transmit it to a processor (e.g., a processor (120) of FIG. 1). The processor can display the image provided from the second function camera (411, 412) on a display (421). The second function camera (411, 412) may be referred to as HR (high resolution) or PV (photo video) and may include a high-resolution camera. For example, the second function camera (411, 412) may include a color camera equipped with functions for obtaining high-quality images, such as an AF (auto focus) function and an OIS (optical image stabilizer), but is not limited thereto, and the second function camera (411, 412) may also include a GS camera or an RS camera.

[0104] A third function camera (e.g., an eye tracking camera) (425, 426) may be positioned on the display (421) (or inside the main body) so that the camera lens faces the user's eyes when the user wears the electronic device (401). The third function camera (425, 426) may be used for the purpose of detecting and tracking pupils (ET: eye tracking). The processor may track the movements of the user's left and right eyes in the images received from the third function camera (425, 426) to determine the gaze direction. By tracking the position of the pupil in the images, the processor may ensure that the center of the XR content image displayed in the display area is positioned according to the direction in which the pupil is looking. As an example, a GS camera may be used as the third function camera (425, 426) to detect the pupil and track the movement of the pupil. The third function camera (425, 426) can be installed for the left eye and the right eye, respectively, and cameras with the same performance and specifications can be used.

[0105] A fourth functional camera (e.g., a camera for facial recognition) (428) can be used to detect and track (FT: face tracking) a user's facial expression when the user wears an electronic device (401).

[0106] According to one embodiment, the electronic device (401) may include a lighting unit (e.g., LED) (not shown) as an auxiliary means for the cameras. For example, the third function camera (425, 426) may use the lighting included in the display so that the emitted light (e.g., IR LED of infrared wavelength) is directed toward the user's both eyes as an auxiliary means to facilitate gaze detection when tracking eye movement. As another example, the second function camera (411, 412) may further include a lighting unit (e.g., flash) as an auxiliary means to supplement the surrounding brightness when taking external pictures.

[0107] According to one embodiment, a depth sensor (or depth camera) (417) can be used for the purpose of checking the distance to an object (e.g., an object), such as time of flight (TOF). TOF (time of flight) is a technology that measures the distance to an object using a signal (e.g., near-infrared, ultrasound, or laser). After a signal is transmitted from a transmitter, a signal is measured at a receiver, and the distance to an object can be measured based on the flight time of the signal.

[0108] According to one embodiment, the touch sensor (413) may be arranged in the second direction (②) of the main body (410). For example, when a user wears the electronic device (401), the user's eyes may look in the first direction (①) of the main body. The touch sensor (413) may be implemented as a single type or a left / right separated type depending on the shape of the main body (410), but is not limited thereto. For example, when the touch sensor (413) is implemented as a left / right separated type as illustrated in FIG. 4A, when a user wears the electronic device (401), the first touch sensor (413a) may be arranged at the user's left eye position, such as in the fourth direction (④), and the second touch sensor (413b) may be arranged at the user's right eye position, such as in the third direction (③).

[0109] The touch sensor (413) can recognize a touch input using at least one of, for example, a capacitive, pressure-sensitive, infrared, or ultrasonic method. For example, the capacitive touch sensor (413) can recognize a physical touch (or contact) input or a hovering input (or proximity) of an external object. According to some embodiments, the electronic device (401) may utilize a proximity sensor (not shown) to recognize proximity of an external object.

[0110] According to one embodiment, the touch sensor (413) has a two-dimensional surface and can transmit touch data (e.g., touch coordinates) of an external object (e.g., a user's finger) that comes into contact with the touch sensor (413) to a processor (e.g., the processor (120) of FIG. 1). The touch sensor (413) can detect a hovering input for an external object (e.g., a user's finger) that approaches within a first distance from the touch sensor (413), or detect a touch input that touches the touch sensor (413).

[0111] According to one embodiment, the touch sensor (413) may provide two-dimensional information about the point of contact as “touch data” to the processor (120) when an external object touches the touch sensor (413). The touch data may be described as a “touch mode.” The touch sensor (413) may provide hovering data about the time or location of hovering around the touch sensor (413) to the processor (120) when an external object is located within a first distance from the touch sensor (or in proximity, hovering above the touch sensor). The hovering data may be described as a “hovering mode / proximity mode.”

[0112] According to one embodiment, the electronic device (401) may obtain hovering data using at least one of a touch sensor (413), a proximity sensor (not shown), or / and a depth sensor (417) to generate information about a distance, location, or time point between the touch sensor (413) and an external object.

[0113] According to one embodiment, the interior of the main body (410) may include a processor (e.g., processor (120) of FIG. 1) and memory (e.g., memory (130) of FIG. 1).

[0114] Memory can store various instructions that can be executed by the processor. Instructions can include arithmetic and logical operations, data transfer, or control commands such as input / output that can be recognized by the processor. Memory can temporarily or permanently store various data, including volatile memory (e.g., volatile memory (132) of FIG. 1) and non-volatile memory (e.g., non-volatile memory (134) of FIG. 1).

[0115] The processor may be a configuration that is operatively, functionally, and / or electrically connected to each component of the electronic device (401) and can perform calculations or data processing related to control and / or communication of each component. The operations performed by the processor may be stored in memory and, when executed, executed by instructions that cause the processor to operate.

[0116] Hereinafter, the computational and data processing functions that the processor can implement on the electronic device (401) are not limited, but a series of operations related to the XR content service function will be described. The operations of the processor described below can be performed by executing instructions stored in memory.

[0117] According to one embodiment, the processor may generate a virtual object based on virtual information based on image information. The processor may output a virtual object related to an XR service together with background space information through the display (421). For example, the processor may capture an image related to a real space corresponding to the field of view of a user wearing the electronic device (401) through a second function camera (411, 412) to obtain image information or generate a virtual space for a virtual environment. For example, the processor may control the display (421) to display XR content (hereinafter referred to as an XR content screen) in which at least one virtual object is output so as to be overlapped in an area determined to be a field of view or a field of view (FoV) of the user.

[0118] According to one embodiment, the electronic device (401) may have a form factor for being worn on a user's head. The electronic device (401) may further include a strap and / or a wearable member for being secured on a body part of the user. The electronic device (401) may provide a user experience based on augmented reality, virtual reality, and / or mixed reality while being worn on the user's head.

[0119] FIG. 5 illustrates examples of construction of a virtual space, input from a user within the virtual space, and output to the user according to various embodiments.

[0120] An electronic device (501) (e.g., the electronic device (101) of FIG. 1, the electronic device (201) of FIG. 2, the electronic device (301) of FIG. 3, and the electronic device (401) of FIG. 4) can obtain spatial information about a physical space in which the sensor is located using a sensor. The spatial information may include a geographical location of the physical space in which the sensor is located, a size of the space, an appearance of the space, a location of a physical object (551) arranged in the space, a size of the physical object (551), an appearance of the physical object (551), and / or illuminant information. The appearance of the space and the physical object (551) may include at least one of a shape, a texture, or a color of the space and the physical object (551). The illuminant information is information about a light source that emits light acting within the physical space, and may include at least one of an intensity, a direction, or a color of the illumination. The aforementioned sensor may collect information for providing augmented reality. For example, referring to the augmented reality device illustrated in FIGS. 2 to 4, the sensor may include a camera and a depth sensor. However, the present invention is not limited thereto, and the sensor may further include at least one of an infrared sensor, a depth sensor (e.g., a lidar sensor, a radar sensor, or a stereo camera), a gyro sensor, an acceleration sensor, or a geomagnetic sensor.

[0121] The electronic device (501) can collect spatial information over multiple time frames. For example, in each time frame, the electronic device (501) can collect spatial information about a portion of a scene within a sensing range (e.g., field of view (FOV)) of a sensor at the location of the electronic device (501) in physical space. By analyzing the spatial information of multiple time frames, the electronic device (501) can track changes in an object (e.g., movement of position or change of state) over time. The electronic device (501) can also obtain integrated spatial information (e.g., an image that spatially stitches scenes around the electronic device (501) in physical space) for the integrated sensing range of the multiple sensors by comprehensively analyzing the spatial information collected through the multiple sensors.

[0122] An electronic device (501) according to one embodiment can analyze a physical space into three-dimensional information by utilizing various input signals of sensors (e.g., sensing data from an RGB camera, an infrared sensor, a depth sensor, or a stereo camera). For example, the electronic device (501) can analyze at least one of the shape, size, and position of a physical space, and the shape, size, or position of a physical object (551).

[0123] For example, the electronic device (501) can detect an object captured within a scene corresponding to the camera's field of view using the camera's sensing data (e.g., a captured image). The electronic device (501) can determine a label of a physical object (551) from the camera's two-dimensional scene image (e.g., information indicating the classification of the object, including a value indicating a chair, a monitor, or a plant) and an area (e.g., a bounding box) occupied by the physical object (551) within the two-dimensional scene. Accordingly, the electronic device (501) can obtain two-dimensional scene information at a position viewed by the user (590). In addition, the electronic device (501) can also calculate the physical space location of the electronic device (501) based on the camera's sensing data.

[0124] The electronic device (501) can obtain location information of the user (590) and depth information of the actual space in the viewing direction using sensing data (e.g., depth data) of the depth sensor. The depth information is information indicating the distance from the depth sensor to each point and can be expressed in the form of a depth map. The electronic device (501) can analyze the distance of each pixel in the three-dimensional position viewed by the user (590).

[0125] The electronic device (501) can acquire information including a 3D point cloud and mesh using various sensing data. The electronic device (501) can analyze a physical space to acquire a surface, mesh, or 3D coordinate point cluster that constitutes the space. Based on the information acquired as described above, the electronic device (501) can acquire a 3D point cloud representing physical objects.

[0126] The electronic device (501) can analyze a physical space to obtain information including at least one of a three-dimensional position coordinate, a three-dimensional shape, or a three-dimensional size (e.g., a three-dimensional bounding box) of physical objects placed in the physical space.

[0127] Accordingly, the electronic device (501) can obtain information on a physical object detected in a three-dimensional space and semantic segmentation information on the three-dimensional space. The physical object information may include at least one of the position, appearance (e.g., shape, texture, and color), or size of the physical object (551) in the three-dimensional space. The semantic segmentation information is information that semantically divides the three-dimensional space into subspaces, and may include, for example, information indicating that the three-dimensional space is divided into an object and a background, and information indicating that the background is divided into a wall, a floor, and a ceiling. The electronic device (501) can obtain and store three-dimensional information (e.g., spatial information) on the physical object (551) and the physical space as described above. The electronic device (501) can store three-dimensional position information of the user (590) in the space together with the spatial information.

[0128] An electronic device (501) according to one embodiment can construct a virtual space (500) based on the physical location of the electronic device (501) and / or the user (590). The electronic device (501) can create the virtual space (500) by referring to the spatial information described above. The electronic device (501) can create a virtual space (500) of the same scale as the physical space based on the spatial information and place objects within the created virtual space (500). The electronic device (501) can provide a complete virtual reality to the user (590) by outputting an image that represents the entire physical space. The electronic device (501) can provide mixed reality (MR) or augmented reality (AR) by outputting an image that represents a part of the physical space. However, although the construction of a virtual space (500) based on spatial information acquired through analysis of the aforementioned physical space is described, the electronic device (501) may also construct a virtual space (500) regardless of the physical location of the user (590). In this specification, the virtual space (500) is a space corresponding to augmented reality or virtual reality, and may also be referred to as a metaverse space.

[0129] For example, the electronic device (501) may provide a virtual graphic representation that replaces at least a portion of a physical space. An electronic device (501) based on optical see-through may output a virtual graphic representation by overlaying the virtual graphic representation on a screen area corresponding to at least a portion of the space on a screen display unit. An electronic device (501) based on video see-through may output an image generated by replacing an image area corresponding to at least a portion of a space image corresponding to a physical space rendered based on spatial information with a virtual graphic representation. The electronic device (501) may replace at least a portion of a background in a physical space with a virtual graphic representation, but is not limited thereto. The electronic device (501) may also perform only additional placement of a virtual object (552) within a virtual space (500) based on spatial information without changing the background.

[0130] The electronic device (501) can place and output a virtual object (552) in a virtual space (500). The electronic device (501) can set an operation area of ​​the virtual object (552) in a space occupied by the virtual object (552) (e.g., a volume corresponding to the appearance of the virtual object (552). The operation area may indicate an area where manipulation of the virtual object (552) occurs. In addition, the electronic device (501) can output a physical object (551) by replacing it with the virtual object (552). The virtual object (552) corresponding to the physical object (551) may have a shape identical to or similar to that of the physical object (551). However, the present invention is not limited thereto, and the electronic device (501) may also set only an operation area in a space occupied by the physical object (551) or in a location corresponding to the physical object (551) without outputting a virtual object (552) that replaces the physical object (551). In other words, the electronic device (501) can transmit visual information representing a physical object (551) (e.g., light reflected from the physical object (551) or an image captured of the physical object (551)) to the user (590) without change, and set a manipulation area on the physical object (551). The manipulation area can be set to the same shape and volume as the space occupied by the virtual object (552) or the physical object (551), but is not limited thereto. The electronic device (501) can also set a manipulation area smaller than the space occupied by the virtual object (552) or the space occupied by the physical object (551).

[0131] According to one embodiment, the electronic device (501) may place a virtual object (552) representing a user (590) (e.g., an avatar object) within the virtual space (500). When the avatar object is provided in a first-person view, the electronic device (501) may visualize a graphical representation corresponding to a part (e.g., a hand, a torso, or a leg) of the avatar object to the user (590) through the aforementioned display (e.g., an optical see-through display or a video see-through display). However, the present invention is not limited thereto, and when the avatar object is provided in a third-person view, the electronic device (501) may also visualize a graphical representation corresponding to the entire shape (e.g., a back view) of the avatar object to the user (590) through the aforementioned display. The electronic device (501) may provide the user (590) with an experience integrated with the avatar object.

[0132] In addition, the electronic device (501) can provide an avatar object of another user who has entered the same virtual space (500). The electronic device (501) can receive feedback information that is the same or similar to feedback information (e.g., information based on at least one of visual, auditory, or tactile senses) provided to another electronic device (501) who has entered the same virtual space (500). For example, when an object is placed in a certain virtual space (500) and multiple users access the virtual space (500), the electronic devices (e.g., electronic devices (501)) of the multiple users can receive feedback information (e.g., graphical representation, sound signal, or haptic feedback) of the same object placed in the virtual space (500) and provide the feedback information to each user (590).

[0133] The electronic device (501) can detect input to an avatar object of another electronic device and can also receive feedback information from the avatar object of the other electronic device. The exchange of input and feedback between each electronic device in the virtual space (500) can be performed by a server (e.g., server (108) of FIG. 1). For example, a server (e.g., a server providing a metaverse space) can transmit input and feedback between an avatar object of a user (590) and an avatar object of another user to the user (590) and the other user. However, the present invention is not limited thereto, and the electronic device (501) can establish direct communication with another electronic device without going through a server to provide input or receive feedback based on an avatar object.

[0134] For example, the electronic device (501) may determine that a physical object (551) corresponding to the selected manipulation area has been selected by the user (590) based on detecting a user input selecting an manipulation area. The user's (590) input may include at least one of a gesture input using a part of the body (e.g., a hand or an eye), an input using a separate virtual reality accessory device, or a user's voice input.

[0135] A gesture input is an input corresponding to a gesture identified based on tracking a body part (510) of a user (590), and may include, for example, an input for pointing or selecting an object. The gesture input may include at least one of a gesture in which a part of the body (e.g., a hand) faces an object for a predetermined period of time or longer, a gesture in which a part of the body (e.g., a finger, an eye, a head) points to an object, or a gesture in which a part of the body and an object make spatial contact. A gesture in which the eye points to an object can be identified based on eye tracking. A gesture in which the head points to an object can be identified based on head tracking.

[0136] Tracking of a body part (510) of a user (590) may be performed primarily based on a camera of the electronic device (501), but is not limited thereto. The electronic device (501) may also track the body part (510) based on the cooperation of sensing data of a vision sensor (e.g., image data of a camera and depth data of a depth sensor) and information collected by an accessory device described below (e.g., controller tracking or finger tracking within the controller). Finger tracking may be performed by sensing the distance or contact between an individual finger and the controller based on a sensor built into the controller (e.g., an infrared sensor).

[0137] The virtual reality accessory device may include a ride-on device, a wearable device, a controller device (520), or other sensor-based device. The ride-on device is a device that a user (590) rides on and operates, and may include, for example, at least one of a treadmill-type device or a chair-type device. The wearable device is a manipulation device that is worn on at least a part of the user's (590) body, and may include, for example, at least one of a full-body and half-body suit-type controller, a vest-type controller, a shoe-type controller, a bag-type controller, a glove-type controller (e.g., a haptic glove), or a face mask-type controller. The controller device (520) may include, for example, an input device (e.g., a stick-type controller or a gun) that is operated by a hand, foot, toe, or other body part (510).

[0138] The electronic device (501) may establish direct communication with the accessory device to track at least one of the location or motion of the accessory device, but is not limited thereto. The electronic device (501) may also communicate with the accessory device via a base station for virtual reality.

[0139] For example, the electronic device (501) may determine that the virtual object (552) has been selected based on detecting an action of gazing at the virtual object (552) for a predetermined period of time or longer through the aforementioned eye gaze tracking technology. For example, the electronic device (501) may recognize a gesture indicating the virtual object (552) through hand tracking technology. The electronic device (501) may determine that the virtual object (552) has been selected based on whether the direction in which the tracked hand points indicates the virtual object (552) for a predetermined period of time or longer, or whether the hand of the user (590) contacts or enters an area occupied by the virtual object (552) within the virtual space (500).

[0140] The user's voice input is an input corresponding to the user's voice acquired by the electronic device (501), and may include, for example, voice data sensed by an input module (e.g., a microphone) of the electronic device (501) or received from an external electronic device of the electronic device (501). The electronic device (501) may determine that a physical object (551) or a virtual object (552) has been selected by analyzing the user's voice input. For example, the electronic device (501) may determine that at least one of the physical object (551) or the virtual object (552) corresponding to the detected keyword has been selected based on detecting a keyword indicating at least one of the physical object (551) or the virtual object (552) from the user's voice input.

[0141] The electronic device (501) may provide feedback as described below in response to the user (590) input described above.

[0142] Feedback may include visual feedback, auditory feedback, tactile feedback, olfactory feedback, or gustatory feedback. The feedback may be rendered by a server (108), an electronic device (101), or an external electronic device (102), as described above in FIG. 1.

[0143] Visual feedback may include outputting an image through a display (e.g., a transparent display or an opaque display) of the electronic device (501).

[0144] Auditory feedback may include outputting sound through a speaker of the electronic device (501).

[0145] Tactile feedback may include force feedback that simulates weight, shape, texture, dimension, and dynamics. For example, a haptic glove may include haptic elements (e.g., electrical muscles) that can simulate the sensation of touch by tensing and relaxing the body of a user (590). The haptic elements within the haptic glove may act as tendons. The haptic glove may provide haptic feedback to the entire hand of the user (590). The electronic device (501) may provide feedback indicating the shape, size, and stiffness of an object through the haptic glove. For example, the haptic glove may generate forces that simulate the shape, size, and stiffness of an object. The exoskeleton of the haptic glove (or suit-like device) may include sensors and finger motion measurement devices, and may transmit tactile information to the body by transmitting a force (e.g., an electromagnetic, DC motor, or pneumatic-based force) that pulls a cable to the fingers of the user (590). Hardware providing tactile feedback may include sensors, actuators, power sources, and wireless transmission circuitry. Haptic gloves may operate by inflating and deflating inflatable air bladders on the glove surface.

[0146] However, haptic feedback is not limited to being provided through haptic gloves. In one embodiment, the electronic device may include a haptic module (e.g., a motor) capable of generating vibration, and may output haptic feedback including vibration using the haptic module. In one embodiment, the electronic device may include a controller or may establish communication with a controller external to the electronic device, and the controller may include a haptic module capable of generating vibration. The controller may output haptic feedback including vibration using the haptic module.

[0147] The electronic device (501) may provide feedback to the user (590) based on the selection of an object within the virtual space (500). For example, the electronic device (501) may output a graphical representation indicating the selected object (e.g., an expression highlighting the selected object) through the display. For example, the electronic device (501) may output a sound (e.g., a voice) guiding the selected object through the speaker. For example, the electronic device (501) may provide the user (590) with a haptic movement that simulates the tactile sensation of the corresponding object by transmitting an electrical signal to a haptic-enabled accessory device (e.g., a haptic glove).

[0148] FIG. 6 is a diagram illustrating an example of an operation of an electronic device providing space to a user according to various embodiments.

[0149] An electronic device according to one embodiment (e.g., an electronic device (101) of FIG. 1, an electronic device (201) of FIG. 2, an electronic device (301) of FIG. 3, an electronic device (401) of FIG. 4, an electronic device (501) of FIG. 5) may display an image rendered of objects (621, 622, 623, 624, 625, 626) arranged in a space. The space according to one embodiment may include at least one of a physical space and a virtual space.

[0150] For example, the space may include a virtual space constructed based on a physical space. For example, an electronic device may obtain spatial information about a physical space around the electronic device, and construct and provide a virtual space based on attributes (e.g., scale) of the obtained physical space. In various embodiments of the present disclosure, a space including a virtual space constructed based on a physical space around the electronic device may also be expressed as a video see-through space (VST space). As described above with reference to FIG. 5, information about the physical space around the electronic device may be collected using a sensor (e.g., a camera sensor, a depth sensor) of the electronic device. The electronic device may construct a virtual space based on information about the physical space around the electronic device.

[0151] For example, the space may include a virtual space constructed independently of the physical space surrounding the electronic device. For example, the space may be a virtual space constructed based on a physical space different from the physical space surrounding the electronic device. For example, the space may be a virtual space constructed by a server (e.g., server 108 of FIG. 1 ). In various embodiments of the present disclosure, a space including a virtual space constructed independently of the physical space surrounding the electronic device may also be expressed as a VR space (virtual reality space).

[0152] For example, the space may include both a physical space around the electronic device and a virtual space in which virtual objects are placed. The virtual space in which virtual objects are placed may be constructed based on the physical space around the electronic device. In other words, the space may include a space in which the physical space around the electronic device and the virtual space in which virtual objects are placed are combined. For example, when the electronic device is an optical see-through (OST)-based electronic device (e.g., the electronic device (201) of FIG. 2), at least a part of the physical space around the electronic device may be directly recognized by the user (610) through a transparent member, and at least a part of the virtual space may be provided to the user (610) by displaying elements of the virtual space (e.g., visual effects, virtual objects) on the display. In other words, light from the outside (e.g., the background of the physical space around the electronic device or a physical object) reaches the user's (610) eyes through the transparent member, so that the user (610) can recognize the physical space around the user (610), and elements of the virtual space are displayed through the display, so that the user (610) can recognize the virtual space overlaid on the physical space around the user (610). For example, the electronic device can provide the user (610) with a space in which elements of the virtual space are additionally arranged against the background of the physical space around the user (610). In various embodiments of the present disclosure, a space including the physical space around the electronic device and the virtual space can also be expressed as an optical see-through space (OST space) or an augmented reality space (AR space).

[0153] Objects placed in space may include physical objects and virtual objects.

[0154] For example, a virtual object may include a virtual object for executing an application (e.g., an icon of the application), a virtual object that provides information (e.g., a virtual object that displays weather information, or a virtual object that displays a memo), or a virtual object that includes a running screen of the application (e.g., a window that displays a running screen of the application). A virtual object may include, for example, a widget, an icon, a visualization anchor, or a running screen of the application (or a window that displays a running screen). For example, a virtual object may include a virtual object that occupies a three-dimensional space (e.g., a three-dimensional area) within a space for decoration and / or providing information.

[0155] For example, a virtual object may be related to a physical object. A virtual object based on a physical object may include a virtual object that replaces the physical object and / or a virtual object related to the physical object (e.g., a virtual controller for controlling the physical object).

[0156] Referring to FIG. 6, when a user (610) wearing the electronic device enters a space, the electronic device can display a screen (620) for the space. The electronic device determines a subspace to be displayed to the user (610) from the space based on the line of sight of the user (610), and can display the screen (620) based on rendering data that renders objects (621, 622, 623, 624, 625, 626) arranged in the subspace. In FIG. 6, the objects (621, 622, 623, 624, 625, 626) on the screen (620) are depicted as being arranged in a curved area, but the present invention is not limited thereto, and the objects may be arranged on a surface other than the curved area.

[0157] In the screen (620) of FIG. 6, at least a portion of the background of the space may be displayed in the remaining portion other than the portion where objects (621, 622, 623, 624, 625, 626) are displayed. For example, if the space is a VST space, the remaining portion other than the portion where objects (621, 622, 623, 624, 625, 626) are displayed may be a virtual space in which the physical space around the electronic device is reconstructed. For example, if the space is an AR space, the remaining portion other than the portion where objects (621, 622, 623, 624, 625, 626) are displayed may be a physical space around the electronic device. For example, if the space is a VR space, the remaining part other than the part where the objects (621, 622, 623, 624, 625, 626) are displayed may be a virtual space (e.g., a virtual space constructed independently from the surroundings of the electronic device).

[0158] FIG. 7 is a flowchart illustrating an example of a method for an electronic device to acquire an additional object according to various embodiments.

[0159] An electronic device according to one embodiment (e.g., electronic device (101) of FIG. 1, electronic device (201) of FIG. 2, electronic device (301) of FIG. 3, electronic device (401) of FIG. 4, electronic device (501) of FIG. 5) can acquire and display additional objects based on objects arranged in space.

[0160] An electronic device can provide a user with a space in which multiple objects are arranged. For example, the electronic device can display a stereoscopic image of the space through at least one display. The stereoscopic image can include a pair of images, a left image corresponding to the user's left eye and a right image corresponding to the user's right eye. The pair of images included in the stereoscopic image can include a pair of images that reflect binocular disparity corresponding to depth information of each point (or object) in the image. In various embodiments of the present disclosure, the stereoscopic image can also be expressed as a three-dimensional image and / or a space image.

[0161] In operation (710), the electronic device may select a source object and a target object from among objects arranged in a space. According to one embodiment, the source object may refer to an object to be used to determine attributes of an additional object to be acquired. Attributes of objects (e.g., source object, target object) may be used to determine visual attributes, auditory attributes, motion attributes related to the motion of the object, and / or tactile attributes. The target object may refer to an object to be used to determine an area to be occupied by the additional object to be acquired. The source object and the target object may be physical object types or virtual object types. The additional object may refer to an object generated based on the source object and the target object.

[0162] An electronic device may detect a user input (e.g., an object selection input) for selecting at least one object among objects arranged in a space, either a source object or a target object. For example, the electronic device may detect a user input for selecting a source object and a user input for selecting a target object, and the electronic device may select the source object and the target object based on the detected inputs. For example, the electronic device may detect a user input for selecting a source object and select an object that intersects the movement trajectory of the user input or the source object moved by the input as a target object.

[0163] According to one embodiment, the user's input (e.g., object selection input) may include at least one of a button input, a motion input, a gesture input, or a voice input.

[0164] A button input may refer to an input detected through a physical button included in an electronic device or an external device (e.g., an input module (150) of FIG. 1 or a controller device (520) of FIG. 5) that is connected to the electronic device through communication. For example, the electronic device or the external device may include a button assigned to a function of selecting an object, and an object selection input may be detected when the button is detected to be pressed.

[0165] Motion input may refer to a motion applied to an electronic device or an external device (e.g., an input module (150) of FIG. 1 or a controller device (520) of FIG. 5) that is connected to the electronic device by communication. The motion input may include an input detected when a user rotates, tilts, and / or moves the electronic device or the external device. For example, a specific motion may be assigned to a function of selecting an object, and an object selection input may be detected when the electronic device or the external device detects that a specific motion has occurred. For example, the specific motion may be determined as a motion that moves along at least a portion of a determined movement trajectory (e.g., a circle, a closed curve, or a check mark (e.g., a V mark)).

[0166] Gesture input may refer to an input corresponding to a gesture identified based on tracking of a user's body part (e.g., a finger, an eye, or a head). Gesture input may include a gesture input corresponding to a gesture of pointing and / or dragging an object with a user's body part. For example, a specific gesture may be assigned to a function of selecting an object, and an object selection input may be detected when an electronic device or an external device detects that a specific gesture has occurred. For example, a specific gesture may be determined as a gesture of pointing to an area corresponding to an object with the user's eyes and / or grabbing an area corresponding to an object with the user's hands.

[0167] Voice input can refer to input corresponding to a user's voice acquired through an electronic device or an external device connected to the electronic device through communication. The electronic device can analyze the voice input to determine that the user's voice input requests the selection of an object. If the user's intent, as determined based on the user's voice input, is to select an object, the electronic device can acquire the voice input as an object selection input.

[0168] According to one embodiment, an electronic device can obtain voice data corresponding to a user's voice. The electronic device can analyze the voice data using a natural language platform. For example, the natural language platform may include an automatic speech recognition module (ASR module) and / or a natural language understanding module (NLU module).

[0169] For example, an automatic speech recognition module can convert a user's speech input into text data. A natural language understanding module can use the text data of the speech input to determine the user's intent. For example, the natural language understanding module can perform syntactic analysis or semantic analysis to determine the user's intent. In one embodiment, the natural language understanding module can use linguistic features (e.g., grammatical elements) of morphemes or phrases to determine the meaning of words extracted from the speech input, and can match the meaning of the determined words to the intent to determine the user's intent.

[0170] User input (e.g., object selection input) is not limited to being composed of one of button input, motion input, gesture input, or voice input, and may be a user input that combines two or more of button input, motion input, gesture input, voice input, or biosignal input. User input that combines two or more of button input, motion input, gesture input, voice input, or biosignal input may also be expressed as 'multimodal input'.

[0171] In operation (720), the electronic device can extract a prompt indicating an attribute of the source object and occupancy information of the target object within the space.

[0172] A prompt may refer to text that represents a property of a source object. For example, the prompt may include a keyword representing at least one of the shape, color, brightness, pattern, visual texture, haptic texture, or semantic information of the source object.

[0173] According to one embodiment, the electronic device may extract a prompt from a partial image corresponding to a source object. For example, if the space is a virtual space constructed based on a physical space (e.g., a VST space), the electronic device may segment a partial image corresponding to the source object from a stereoscopic image and extract a prompt from the partial image corresponding to the segmented source object. For example, if the space is a space that combines a physical space around the electronic device and a virtual space in which virtual objects are placed (e.g., an AR space), the electronic device may acquire a partial image corresponding to the source object through a camera and extract a prompt from the partial image.

[0174] An electronic device may extract a prompt from a partial image using a prompt extraction model. The prompt extraction model may refer to a model generated and / or trained to output output data corresponding to a prompt indicating an attribute of an object from input data corresponding to the object. For example, the input data may include an image corresponding to the source object and / or information regarding a selection result (or recognition result) of the source object. The prompt may include text corresponding to an analysis result of the source object or text describing the source object. According to one embodiment, the prompt extraction model may include a model based on a machine learning model (e.g., a neural network and / or a convolutional neural network (CNN)).

[0175] However, the electronic device according to various embodiments of the present disclosure is not limited to extracting a prompt from a partial image corresponding to a source object. For example, the electronic device may extract a prompt based on metadata corresponding to the source object. When the source object is detected within a user's field of view (FOV) or a display area of ​​a display, the electronic device may store metadata of the source object based on a result of determining (e.g., classifying) the class of the source object and / or a signal received from the source object. The electronic device may extract a prompt from the metadata of the source object.

[0176] Occupancy information may refer to information indicating a three-dimensional area (e.g., a three-dimensional area) occupied by a specific object within a space. For example, the occupancy information may include a plurality of voxels representing a space provided to a user, and the voxel value of each voxel may indicate whether the voxel is occupied by a specific object. For example, the voxel value may have a first value corresponding to unknown (e.g., -1), a second value corresponding to unoccupied (e.g., 0), and a third value corresponding to occupied (e.g., a positive number). The third value corresponding to occupied may include a value indicating an object that occupies an area corresponding to the voxel (e.g., an identifier of the object).

[0177] Occupancy information of a target object may refer to information indicating the area occupied by the target object within space. For example, occupancy information of a target object may include the coordinates of voxels occupied by the target object.

[0178] According to one embodiment, if the type of the target object is a virtual object type, the target object is an object constructed by the electronic device to occupy a specific area within a space, and therefore the electronic device can obtain occupancy information of the target object.

[0179] According to one embodiment, when a space is an AR space and the type of the target object is a physical object type, the electronic device can determine (e.g., estimate, analyze) an area occupied by the target object within the space using sensing data corresponding to the target object (e.g., an image captured by a camera, depth data from a depth sensor). The electronic device can obtain occupancy information of the target object based on the area determined to be occupied by the target object.

[0180] In operation (730), the electronic device may obtain an additional object having properties of the source object and having a shape that matches at least a portion of the outline of the target object, based on the prompt and occupancy information.

[0181] An electronic device can acquire additional objects based on prompts and occupancy information. The additional objects may refer to objects generated based on the source object and the target object. The additional objects may have a shape based on the area occupied by the target object and may have the properties of the source object. The shape of the additional object may match at least a portion of the outline of the target object.

[0182] In one embodiment, the prompt and occupancy information may be implemented in an integrated manner. For example, the prompt may include text corresponding to the occupancy information. For example, the prompt may include, "Create a 10x10 white cat at specific coordinates (e.g., (x1, y1, z1)) and reflect its movement." The "specific coordinates" in the prompt may be at least a portion of the occupancy information, or may be coordinates based on the occupancy information.

[0183] The outline of an object may refer to the outer face (e.g., shell) of the object. For example, based on the occupancy information of the target object, the electronic device may determine a set of voxels adjacent to voxels not occupied by the target object among voxels occupied by the target object as the outline of the target object.

[0184] Matching the shape of the additional object to the outline of the target object may include that the outline of the additional object and the outline of the target object have at least partially the same shape. Matching the shape of the additional object to the outline of the target object may include that the additional object has a shape that surrounds at least a portion of the outline of the target object. Matching the shape of the additional object to the outline of the target object may include that the shape of the additional object can cover (e.g., occupy, replace) an internal area of ​​the outline of the target object (e.g., an area occupied by the target object). An additional object according to various embodiments of the present disclosure may also be expressed as a skin object, an outfit object, or a wrap object.

[0185] According to one embodiment, an electronic device may acquire additional objects using an object generation model. The object generation model may refer to a model generated and / or trained to output output data containing information corresponding to the additional objects by being applied to input data including information about a source object (e.g., a prompt) and information about a target object (e.g., occupancy information of the target object). The object generation model may include a model based on a machine learning model (e.g., a neural network, a large language model (LLM), or a generative artificial intelligence).

[0186] In one embodiment, the input data is not limited to including prompts and / or occupancy information. For example, the input data may further include additional properties (e.g., size, shape, texture, and / or semantic information) of the additional object (or substitute object) to be generated. The additional properties may refer to properties of the additional object (or substitute object) that are independent of the properties of the source object, and the additional properties may be specified by user input or predetermined by the server. In one embodiment, the prompt may further include text indicating the additional properties.

[0187] The object creation model may be stored internally in the electronic device or stored in an external device accessible to the electronic device (e.g., another electronic device, a server, a cloud). Additional objects may be created using the object creation model. For example, the electronic device may directly create additional objects using the object creation model (e.g., on-device), or the electronic device may transmit information about a source object (e.g., a prompt) and information about a target object (e.g., occupancy information of the target object) to another electronic device that stores the object creation model and receive information about the additional objects created (e.g., on-cloud).

[0188] In operation (740), the electronic device may display an additional object via a display at a location aligned along at least a portion of the outline of the target object. The location aligned along at least a portion of the outline of the target object may mean a location corresponding to at least a portion of the outline of the target object that matches the shape of the additional object.

[0189] For example, if the outline of the additional object and the outline of the target object have a partially identical shape, the additional object may be attached to the target object at a portion of the outline of the additional object that is identical to the shape of the target object. For example, if the additional object has a shape that surrounds at least a portion of the outline of the target object, the additional object may be positioned to surround at least a portion of the outline of the target object. For example, if the additional object can cover an inner area of ​​the outline of the target object, the additional object may be positioned to occupy at least a portion of the inner area of ​​the outline of the target object.

[0190] Various embodiments of the present disclosure primarily describe, but are not limited to, an additional object being aligned along at least a portion of the outline of a target object. The additional object may be aligned with the target object or a reference point determined based on the target object. The reference point determined based on the target object may include a center point of the target object. The electronic device may align the additional object with the center point of the target object, and the electronic device may replace the target object with the additional object. For example, the electronic device may delete the target object and display the additional object in its place.

[0191] In one embodiment, the electronic device can transmit information about an additional object to another electronic device. The other electronic device may be an electronic device distinct from the electronic device and implemented in the same or similar manner as an augmented reality device (e.g., electronic device 201 of FIG. 2 , electronic device 301 of FIG. 3 , electronic device 401 of FIG. 4 , or electronic device 501 of FIG. 5 ). The other electronic device can receive information about the additional object from the electronic device. When the other electronic device provides (e.g., displays) a target object within a space, the other electronic device can display the additional object at a location aligned along at least a portion of the outline of the target object. Consequently, the additional object acquired by the electronic device can be displayed by the electronic device and / or by the other electronic device.

[0192] FIG. 8 is a diagram illustrating an example of an operation of an electronic device according to various embodiments to obtain and display an additional object fitted to a target object.

[0193] An electronic device according to one embodiment (e.g., electronic device (101) of FIG. 1, electronic device (201) of FIG. 2, electronic device (301) of FIG. 3, electronic device (401) of FIG. 4, electronic device (501) of FIG. 5) may generate an additional object (830) matching at least a portion of an outline of a target object (812) based on a source object (811) and a target object (812), and display the additional object (830) together with the target object (812).

[0194] The electronic device can select a source object (811) and a target object (812) from among a plurality of objects placed in the space (801) while providing the space (801) to the user.

[0195] In FIG. 8, for example, the source object (811) may be an object corresponding to a cat. The source object (811) may be a physically existing cat, a virtual object type cat object (e.g., a virtual cat object), or an object indicating a cat (e.g., a photo of a cat, a drawing of a cat, a symbol indicating a cat, or text indicating a cat).

[0196] In FIG. 8, for example, the target object (812) may be an object corresponding to a television (TV). The target object (812) may be a physically existing television, or may include a TV object of a virtual object type (e.g., a virtual TV object).

[0197] The electronic device may extract a prompt indicating an attribute of the source object (811) based on the selected source object (811). For example, the electronic device may extract at least one of the text 'cat', the text 'black', the text 'short-haired', or the text 'long-tailed' from the source object (811).

[0198] The electronic device can extract occupancy information of the target object (812) within the space (801) based on the selected target object (812). For example, if the target object (812) is a TV object of an actual physical object type, the electronic device can determine (e.g., estimate) the area occupied by the TV object within the space (801). If the target object (812) is a virtual TV object, the electronic device can obtain the area occupied by the virtual TV object within the space (801).

[0199] The electronic device can acquire additional objects (830) based on prompt and occupancy information. The object creation model (820) can be applied to input data including the prompt and occupancy information to output information about the additional objects (830).

[0200] According to one embodiment, the input data may further include information directing the creation of an additional object (830) having a shape matching at least a portion of the outline of the target object (812). As will be described in more detail later in FIG. 10, the object creation model (820) may also be used in an operation of obtaining a substitute object. The substitute object may refer to an object that can replace the target object. The substitute object may include an object that can occupy an area occupied by the target object in space. The object creation model (820) may optionally output output data including information about one of the additional object (830) or the substitute object. For example, if the input data includes information directing the creation of the additional object (830), information about the additional object (830) may be output from the object creation model (820), and if the input data includes information directing the creation of the substitute object, information about the substitute object may be output from the object creation model (820).

[0201] In one embodiment, the input data may further include information regarding additional properties (e.g., size and / or shape) of the replacement object (or additional object) to be created, together with the prompt and / or occupancy information. However, the additional properties of the replacement object (or additional object) to be created are not limited to being included in the input data separately from the prompt and / or occupancy information, and the prompt may include text indicating the additional properties of the replacement object (or additional object) to be created.

[0202] In FIG. 8, for example, the additional object (830) may include partial objects representing cat ears that can be placed (e.g., attached) on the top surface of the TV, and partial objects representing cat tails that can be placed (e.g., attached) on the back surface of the TV. In FIG. 8, the outline of the target object (812) is illustrated together with the additional object (830) for explanation, but the additional object (830) generated by the object generation model (820) may not include the target object (812).

[0203] The electronic device may display an additional object (830) at a location corresponding to the target object (812). In FIG. 8, the electronic device may display the additional object (830) (e.g., partial objects representing a cat's ears and a cat's tail) at a location aligned with at least a portion of the outline of the target object (812). The electronic device may display a form (840) in which the additional object (830) is combined with the target object (812). In various embodiments of the present disclosure, displaying the additional object (830) at a location aligned with the outline of the target object (812) may also be expressed as applying the additional object (830) to the target object (812).

[0204] FIG. 9 is a diagram illustrating an example of an operation in which an electronic device according to various embodiments displays different additional objects depending on the operational state of a target object.

[0205] An electronic device according to one embodiment (e.g., electronic device (101) of FIG. 1, electronic device (201) of FIG. 2, electronic device (301) of FIG. 3, electronic device (401) of FIG. 4, electronic device (501) of FIG. 5) may acquire and display additional objects for each operation state when a target object (910) can operate in one of a plurality of operation states.

[0206] According to one embodiment, the electronic device may obtain a plurality of candidate additional objects corresponding to the plurality of operation states based on the target object (910) being another electronic device capable of operating in one of the plurality of operation states. The electronic device may select an additional object corresponding to the operation state in which the target object (910) is operating from among the plurality of candidate additional objects. For example, the electronic device may receive a signal from the other electronic device, which is the target object (910), indicating the operation state in which the other electronic device is operating. The electronic device may determine the operation state of the other electronic device based on the signal received from the other electronic device. For example, the electronic device may obtain an image corresponding to the other electronic device, which is the target object (910). The electronic device may determine the operation state of the other electronic device based on the image corresponding to the other electronic device. The electronic device may select one additional object from among the plurality of candidate additional objects based on the determined operation state of the other electronic device. The electronic device may display the selected additional object at a position aligned along the outline of the other electronic device, which is the target object (910).

[0207] Other electronic devices may indicate the type of operation being performed by the other electronic device. For example, if the other electronic device is an air conditioner, the air conditioner may perform at least one of the following operations: cooling, dehumidifying, air purifying, or heating. Each operation state may correspond to a type of operation of the other electronic device. The electronic device may display additional candidate objects corresponding to a specific type of operation while the other electronic device is performing a specific type of operation.

[0208] According to one embodiment, the other electronic device, which is the target object (910), may operate at least a portion of the outline of the target object (910) as a working area depending on the operating state. For example, if the other electronic device, which is the target object (910), includes a display area (930), the operating states of the other electronic device may include operating states corresponding to activation and deactivation of the display area (930). The other electronic device may display a screen in the display area (930) while the display area (930) is activated. The other electronic device may not display a screen in the display area (930) while the display area (930) is deactivated.

[0209] According to one embodiment, the other electronic device, which is the target object (910), may have a different area to which an additional object can be applied depending on the operating state. The area to which the additional object can be applied may refer to an area in the outline of the target object (910) where the additional object can be displayed (e.g., attached). The area to which the additional object can be applied may include a portion of the outline of the target object (910) that has little or no effect on the operation of the target object (910) even if the outline of the target object (910) is obscured by the additional object. For example, if the other electronic device, which is the target object (910), includes a display area (930), the area to which the additional object can be applied in an operating state where the display area (930) is activated may be another area excluding the display area (930). The area to which the additional object can be applied in an operating state where the display area (930) is deactivated may be an area including the display area (930) and another area.

[0210] The plurality of candidate additional objects may have a shape that matches at least a portion of the outline of the target object (910) corresponding to the area to which the additional objects are applicable in the corresponding operation state. For example, if another electronic device, which is the target object (910), includes a display area (930), the electronic device may obtain a first candidate additional object that has a shape that matches a portion of the outline of the target object (910) excluding the display area (930) as a candidate additional object corresponding to an operation state in which the display area (930) is activated. The electronic device may obtain a second candidate additional object that has a shape that matches at least a portion of the outline of the target object (910) including the display area (930) as a candidate additional object corresponding to an operation state in which the display area (930) is deactivated.

[0211] However, the electronic device according to various embodiments of the present disclosure is not limited to acquiring a plurality of candidate additional objects corresponding to the operational states of another electronic device, which is the target object (910). The electronic device according to one embodiment may limit the display of additional objects at a location corresponding to the work area (e.g., the display area (930)) depending on the operational state of another electronic device, which is the target object (910).

[0212] For example, the electronic device may restrict displaying additional objects at a location corresponding to the display area (930) while the display area (930) is activated based on the target object (910) including another electronic device and the outline of the target object (910) including the display area (930). The electronic device may display at least a portion of the additional objects at a location corresponding to the display area (930) while the display area (930) is deactivated.

[0213] Referring to FIG. 9, for example, a target object (910) may be a TV object including a display area (930). A situation (901) may represent a situation in which the display area (930) of the target object (910) is activated, and a situation (902) may represent a situation in which the display area (930) of the target object (910) is deactivated.

[0214] In situation (901), the electronic device may receive a signal from another electronic device, which is a target object (910), indicating that the other electronic device is operating in a first operation state. The first operation state may, for example, mean an operation state in which the display area (930) is activated. The first operation state may mean a state in which the other electronic device is turned on (or a state in which the other electronic device is operating at a power higher than a threshold power). The electronic device may restrict displaying an additional object at a location corresponding to the display area (930). Alternatively, the electronic device may obtain and display an additional object corresponding to the first operation state, which is positioned at a location different from the location corresponding to the display area (930). According to one embodiment, the additional object may include multiple partial objects (e.g., multiple partial additional objects). In FIG. 9, in a situation (901), the electronic device can display an additional object (e.g., a partial object corresponding to a cat's ear (921), a partial object corresponding to a cat's tail (922)) in a different area (e.g., a part of the top surface of a TV object, a part of the back surface) from the display area (930) of the target object (910), and display the screen without displaying the additional object in the display area (930) of the target object (910).

[0215] In situation (902), the electronic device may receive a signal from another electronic device, which is a target object (910), indicating that the other electronic device is operating in a second operating state. The second operating state may, for example, mean an operating state in which the display area (930) is disabled. The second operating state may mean a state in which the other electronic device is turned off (or is operating at a power below a threshold power). The electronic device may display at least a portion of an additional object at a location corresponding to the display area (930). Alternatively, the electronic device may obtain and display an additional object corresponding to the second operating state, which is positioned at a location corresponding to the display area (930). In FIG. 9, in a situation (902), the electronic device can display an additional object (e.g., a partial object (921) corresponding to a cat's ear, a partial object (922) corresponding to a cat's tail) in a different area (e.g., a part of the top surface, a part of the back surface of a TV object) than the display area (930) of the target object (910), and can display an additional object (e.g., a partial object (923) corresponding to a cat's face) in the display area (930) of the target object (910).

[0216] In various embodiments of the present disclosure, the electronic device primarily, but not limited to, receiving a signal from another electronic device, which is a target object (910), indicating an operational state in which the other electronic device is operating. For example, the electronic device may determine that the other electronic device, which is a target object (910), is in a specific operational state (e.g., turned off) based on the absence of a signal from the other electronic device, which is a target object (910) (e.g., when no signal is received from the other electronic device for a threshold period of time).

[0217] The electronic device can acquire an image corresponding to another electronic device, which is a target object (910), and determine the operating state of the other electronic device based on the acquired image. For example, the electronic device can determine whether a screen is being output in a working area (e.g., a display area) of the other electronic device, which is a target object (910). If the screen is being output, the electronic device can determine that the other electronic device is operating in a first operating state (e.g., a turned-on state). If the screen is not being output (e.g., a display area is deactivated), the electronic device can determine that the other electronic device is operating in a second operating state (e.g., a turned-off state).

[0218] For example, the operation state of the target object (910) may be determined based on the operation state of another electronic device external to the target object (910). For example, the other electronic device may be a beam projector, and the target object (910) may be an area (e.g., a cloth, a wall, a screen) onto which light output from the beam projector is projected. While the operation state of the other electronic device, the beam projector, is in a first operation state (e.g., a turned-on state), the other electronic device, the beam projector, may project light onto the target object (910), and the operation state of the target object (910) may be determined as the first operation state (e.g., an activated state). While the operation state of the other electronic device, the beam projector, is in a second operation state (e.g., a turned-off state), the other electronic device, the beam projector, may not project light, and the operation state of the target object (910) may be determined as the second operation state (e.g., a deactivated state).

[0219] FIG. 10 is a diagram illustrating an example of an operation of an electronic device according to various embodiments to obtain and display a replacement object that replaces a target object.

[0220] An electronic device according to one embodiment (e.g., electronic device (101) of FIG. 1, electronic device (201) of FIG. 2, electronic device (301) of FIG. 3, electronic device (401) of FIG. 4, electronic device (501) of FIG. 5) may generate a replacement object (1030) of a target object (1012) based on a source object (1011) and a target object (1012), and display the replacement object (1030) instead of the target object (1012).

[0221] The electronic device can select a source object (1011) and a target object (1012) from among a plurality of objects placed in the space (1001) while providing the space (1001) to the user.

[0222] In FIG. 10, for example, the source object (1011) may be an object corresponding to a cat. The source object (1011) may be a physically existing cat, a virtual object type cat object (e.g., a virtual cat object), or an object indicating a cat (e.g., a photo of a cat, a drawing of a cat, a symbol indicating a cat, or text indicating a cat).

[0223] In FIG. 10, for example, the target object (1012) may be an object corresponding to a pillow. The target object (1012) may be a physically existing pillow, or may include a pillow object of a virtual object type (e.g., a virtual pillow object).

[0224] The electronic device may extract a prompt indicating an attribute of the source object (1011) based on the selected source object (1011). For example, the electronic device may extract at least one of the text 'cat', the text 'black', the text 'short-haired', or the text 'long-tailed' from the source object (1011).

[0225] The electronic device can extract occupancy information of the target object (1012) within the space (1001) based on the selected target object (1012). For example, if the target object (1012) is a pillow object of an actual physical object type, the electronic device can determine (e.g., estimate) the area occupied by the pillow object within the space (1001). If the target object (1012) is a virtual pillow object, the electronic device can obtain the area occupied by the virtual pillow object within the space (1001).

[0226] The electronic device can obtain a substitute object (1030) that can occupy an area occupied by the target object (1012) in space from the prompt and occupancy information. The substitute object may refer to an object that can occupy an area that has a difference below a threshold from the area occupied by the target object in space. The object creation model (1020) can output information about an additional object by being applied to input data including the prompt and occupancy information. The input data may further include information that instructs the creation of the substitute object (1030).

[0227] In FIG. 10, by way of example, the replacement object (1030) may include a cat object of a virtual object type that may occupy an area corresponding to a pillow (e.g., an area occupied by a pillow).

[0228] The electronic device may display a substitute object (1030) instead of the target object (1012) at a location corresponding to the target object (1012) through a display. In FIG. 10, the electronic device may display a substitute object (1030) (e.g., a cat object of the virtual object type) at a location where the target object (1012) is placed. In various embodiments of the present disclosure, displaying the substitute object (1030) instead of the target object (1012) may also be expressed as applying the substitute object (1030) to the target object (1012).

[0229] According to one embodiment, the electronic device can transmit information regarding the substitute object (1030) to another electronic device. The other electronic device may be an electronic device distinct from the electronic device and implemented in the same or similar manner as an augmented reality device (e.g., the electronic device (201) of FIG. 2 , the electronic device (301) of FIG. 3 , the electronic device (401) of FIG. 4 , or the electronic device (501) of FIG. 5 ). The other electronic device may receive information regarding the substitute object (1030) from the electronic device. When the other electronic device provides (e.g., displays) a target object (1012) within a space, the other electronic device may display the substitute object (1030) at a location aligned along at least a portion of the outline of the target object (1012). Consequently, the substitute object (1030) acquired by the electronic device may be displayed by the electronic device and / or by the electronic device with the other electronic device.

[0230] FIG. 11 is a diagram illustrating example outputs of an object creation model according to various embodiments.

[0231] An electronic device according to one embodiment (e.g., electronic device (101) of FIG. 1, electronic device (201) of FIG. 2, electronic device (301) of FIG. 3, electronic device (401) of FIG. 4, electronic device (501) of FIG. 5) may acquire at least one object among additional objects or substitute objects based on a source object (1110) and a target object, and may further acquire additional information about the at least one object.

[0232] The additional information may include information about the motion (1131) of the additional object or replacement object, or information about the haptic properties (1133) of the additional object or replacement object.

[0233] In one embodiment, the electronic device may acquire motion (1131) of an additional object based on prompt and occupancy information. By applying the acquired motion (1131) of the additional object to the additional object, the electronic device may display an additional object moving at a position aligned along the outline of the target object.

[0234] For example, as shown in FIG. 9, a partial object corresponding to a cat's ear, a partial object corresponding to a cat's tail, and a partial object corresponding to a cat's face may be created as additional objects. The motion of the partial object corresponding to the cat's ear may include the cat's ear twitching. The motion of the partial object corresponding to the cat's tail may include the cat's tail wagging. The motion of the partial object corresponding to the cat's face may include the cat's eye blinking, the cat yawning, or the cat's whiskers wagging. When the additional object includes multiple partial objects, the motion of the additional object may include the motion of one or a combination of two or more of the multiple partial objects.

[0235] In various embodiments of the present disclosure, the object to which motion (1131) can be applied is not limited to an additional object. An electronic device according to one embodiment can acquire the motion (1131) of a substitute object. For example, the electronic device can acquire the motion (1131) of the substitute object based on prompt and occupancy information. By applying the acquired motion (1131) of the substitute object to the substitute object, the electronic device can display a moving substitute object instead of the target object at a location corresponding to the target object.

[0236] The electronic device can obtain motion (1131) of an additional object (or substitute object) using an object generation model (1120). The object generation model (1120) can be generated and / or trained to output output data that further includes information about the motion (1131) of the additional object (or substitute object) together with information about the additional object (or substitute object) by being applied to input data including prompt and occupancy information.

[0237] An electronic device according to one embodiment may apply a motion selected from among a plurality of candidate motions based on a user input to an additional object (or a replacement object).

[0238] For example, the electronic device can obtain a plurality of candidate motions of an additional object from the prompt and occupancy information, and a correspondence relationship (1132) between the plurality of candidate motions and the candidate inputs of the user. The electronic device can display an additional object moving at a location where the additional object is displayed by applying a motion corresponding to the input detected among the plurality of candidate motions to the additional object based on detecting a user input for the additional object.

[0239] For example, the electronic device can obtain a plurality of candidate motions of the substitute object from the prompt and occupancy information, and a correspondence relationship (1132) between the plurality of candidate motions and the candidate inputs of the user. The electronic device can display a substitute object moving at a location where the substitute object is displayed by applying a motion corresponding to the input detected among the plurality of candidate motions to the substitute object based on detecting a user input for the substitute object.

[0240] The electronic device can obtain a plurality of candidate motions of an additional object (or a substitute object) and a correspondence relationship (1132) between the plurality of candidate motions and candidate inputs using an object generation model (1120). The object generation model (1120) can be generated and / or trained to output output data that further includes information about the plurality of candidate motions and the correspondence relationship (1132) between the plurality of candidate motions and candidate inputs, together with information about the additional object (or substitute object), by being applied to input data including prompt and occupancy information.

[0241] For example, as in FIG. 10, a cat object of a virtual object type may be generated as a substitute object. The electronic device may obtain a plurality of candidate motions and a correspondence relationship (1132) between the plurality of candidate motions and candidate inputs. For example, the candidate inputs may include a first candidate input corresponding to a gaze looking at the substitute object cat object, a second candidate input corresponding to a gesture of stroking the substitute object cat object with a user's hand, and a third candidate input corresponding to a gesture of lifting the substitute object cat object. The candidate motions may include a first candidate motion corresponding to the substitute object cat object yawning, a second candidate motion corresponding to the substitute object cat object rubbing against the user's hand, and a third candidate motion corresponding to the substitute object cat object flounding. The correspondence relationship (1132) between candidate motions and candidate inputs may include a first candidate motion corresponding to a first candidate input, a second candidate motion corresponding to a second candidate input, and a third candidate motion corresponding to a third candidate input.

[0242] When the electronic device detects a gaze input (e.g., a first candidate input) by the user looking at the cat object as a substitute object, the electronic device can display a yawning cat object by applying a first candidate motion to the cat object as a substitute object. When the electronic device detects a gesture input (e.g., a second candidate input) by the user stroking the cat object as a substitute object with the user's hand, the electronic device can display a cat object rubbing against the user's hand by applying a second candidate motion to the cat object as a substitute object. When the electronic device detects a gesture input (e.g., a third candidate input) by the user lifting the cat object as a substitute object, the electronic device can display a struggling cat object by applying a third candidate motion to the cat object as a substitute object.

[0243] An electronic device according to one embodiment may acquire a haptic attribute (1133) of haptic feedback for an additional object (or a substitute object). The electronic device may detect a user's gesture input for the additional object (or the substitute object) (e.g., a gesture input corresponding to a user's gesture of contacting an outline of the additional object or a target object). Based on the user's gesture input for the additional object (or the substitute object), the electronic device may output haptic feedback having the haptic attribute (1133) of the additional object (or the substitute object). For example, the electronic device may provide the haptic feedback having the haptic attribute (1133) to the user through a haptic feedback output device (e.g., a smart glove) included in or separate from the electronic device.

[0244] Haptic properties (1133) may include, for example, temperature, elasticity, degree of hardness (or degree of softness), degree of smooth (or degree of roughness), or degree of slipperiness.

[0245] For example, the electronic device can acquire haptic properties (1133) of haptic feedback for an additional object based on prompt and occupancy information. The electronic device can output haptic feedback having the acquired haptic properties (1133) based on detecting a user input for the additional object.

[0246] For example, an electronic device may acquire haptic properties (1133) of haptic feedback for a substitute object based on prompt and occupancy information. The electronic device may output haptic feedback having the acquired haptic properties (1133) based on detecting a user input for the substitute object.

[0247] The electronic device can obtain haptic properties (1133) of haptic feedback for an additional object (or a substitute object) using an object generation model (1120). The object generation model (1120) can be generated and / or trained to output output data further including haptic properties (1133) of haptic feedback for the additional object (or substitute object) together with information about the additional object (or substitute object) by being applied to input data including prompt and occupancy information.

[0248] Although not explicitly shown in FIG. 11, the object generation model (1120) is not limited to outputting information related to visual feedback (e.g., motion (1131), correspondence (1132)) and information related to haptic feedback (e.g., haptic properties (1133)). For example, the object generation model (1120) may output properties of feedback for an additional object (or a substitute object). The feedback for the additional object (or substitute object) may include at least one of visual feedback, auditory feedback, and / or haptic feedback. For example, the object generation model (1120) may be generated and / or trained to output information regarding auditory properties, including the tone of voice feedback to be provided to the user.

[0249] FIGS. 12A and 12B are diagrams illustrating examples of operations in which an electronic device, according to various embodiments, acquires an object selected from among additional objects and replacement objects based on user input.

[0250] An electronic device according to one embodiment (e.g., electronic device (101) of FIG. 1, electronic device (201) of FIG. 2, electronic device (301) of FIG. 3, electronic device (401) of FIG. 4, electronic device (501) of FIG. 5) may select one of an additional object or an alternative object based on a user's input.

[0251] The electronic device may provide a user with a menu object (1220a, 1220b) for selecting either the creation of an additional object or the creation of a replacement object. The menu object (1220a, 1220b) may include, as an example of a virtual object, a first button (1221a, 1221b) for the creation of an additional object and a second button (1222a, 1222b) for the creation of a replacement object.

[0252] For example, the electronic device may display a menu object (1220a, 1220b) after selecting at least one of a source object (1211a, 1211b) or a target object (1212a, 1212b). As illustrated in FIGS. 12a and 12b , the menu object (1220a, 1220b) may include a first button (1221a, 1221b) for creating an additional object, a second button (1222a, 1222b) for creating a replacement object, and a third button (1223a, 1223b) for saving the source object (1211a, 1211b).

[0253] In FIG. 12A, for example, the electronic device can detect a user input from a first button (1221a) among menu objects (1220a). Based on the user input, the electronic device can perform acquisition of an additional object. For example, the electronic device can add information indicating the creation of an additional object to the input data of the object creation model.

[0254] In FIG. 12b, for example, the electronic device can detect a user input from the second button (1222b) among the menu objects (1220b). Based on the user input, the electronic device can perform acquisition of a substitute object. For example, the electronic device can add information indicating the creation of a substitute object to the input data of the object creation model.

[0255] Storage of the source object (1211a, 1211b) may mean an operation of transmitting information about the source object (1211a, 1211b) to the target object (1212a, 1212b), if the target object (1212a, 1212b) is another electronic device, and enabling the target object (1212a, 1212b) to provide information about the source object (1211a, 1211b) to a user.

[0256] For example, the target object (1212a, 1212b) may be another electronic device including a display area. The electronic device may detect a user input selecting to store the source object (1211a, 1211b) from a menu object. The electronic device may transmit information about the source object (1211a, 1211b) (e.g., information for visualizing the source object (1211a, 1211b)) to the other electronic device that is the target object (1212a, 1212b). The other electronic device that is the target object (1212a, 1212b) may receive and store information about the source object (1211a, 1211b). Another electronic device, which is a target object (1212a, 1212b), can visualize the source object (1211a, 1211b) through the display area when a condition is met (e.g., when receiving a user input requesting visualization of the source object (1211a, 1211b)).

[0257] In one embodiment, information about the source object (1211a, 1211b) may include information about an additional object (or substitute object) generated based on the source object (1211a, 1211b). Another electronic device, which is a target object (1212a, 1212b), may visualize the additional object (or substitute object) through the display area when a condition is met (e.g., when receiving a user input requesting visualization of the additional object or substitute object).

[0258] FIG. 13 is a diagram illustrating an example of an operation of an electronic device according to various embodiments to stop or re-display an additional object or a replacement object based on a distance.

[0259] An electronic device according to one embodiment (e.g., electronic device (101) of FIG. 1, electronic device (201) of FIG. 2, electronic device (301) of FIG. 3, electronic device (401) of FIG. 4, electronic device (501) of FIG. 5) may stop displaying and / or re-displaying an additional object or a replacement object based on a distance between the electronic device and a target object in space.

[0260] The electronic device may display an additional object at a location aligned with at least a portion of the outline of the target object while the distance from the target object to the electronic device in the space is less than a threshold distance. The electronic device may restrict displaying of the additional object while the distance from the target object to the electronic device in the space is greater than or equal to the threshold distance.

[0261] For example, the electronic device may stop displaying additional objects based on a distance from the target object within the space to the electronic device being greater than or equal to a threshold distance. The electronic device may then re-display additional objects at locations aligned with at least a portion of the outline of the target object based on a distance from the target object within the space to the electronic device being less than the threshold distance.

[0262] Referring to FIG. 13, in a situation (1301), the electronic device may display an additional object within a space at a location aligned with at least a portion of an outline of a target object. In a situation (1301), a distance between the electronic device and the target object within the space may be a first distance less than a threshold distance. The electronic device may display the additional object based on the distance between the target object and the electronic device within the space being less than the threshold distance.

[0263] An electronic device can change its position or heading direction within a space. For example, the electronic device can change the position or heading direction of the electronic device (or an avatar object corresponding to the user) within a space based on user input or movement. The electronic device can move away from a target object within the space.

[0264] In situation (1302), the electronic device may be positioned at a greater distance from the target object than in situation (1301) due to a change in the position of the electronic device within the space. In situation (1302), the distance between the electronic device and the target object within the space may be a second distance that is less than the threshold distance and greater than the first distance. The electronic device may continue to display additional objects based on the distance between the target object and the electronic device within the space being less than the threshold distance. Thereafter, the electronic device may be positioned further away from the target object within the space.

[0265] In situation (1303), the electronic device may be located at a greater distance from the target object than in situation (1302) due to a change in the position of the electronic device within the space. In situation (1303), the distance between the electronic device and the target object within the space may be a third distance greater than the threshold distance. The electronic device may stop displaying additional objects based on the distance between the target object and the electronic device within the space being greater than the threshold distance.

[0266] Although not explicitly illustrated in FIG. 13, the electronic device may then move closer to the target object within the space. Based on the distance between the electronic device and the target object being less than a threshold distance, the electronic device may display additional objects again.

[0267] In FIG. 13, displaying an additional object is mainly described, but is not limited thereto. According to one embodiment, an electronic device can control the display of a substitute object based on the distance between the electronic device and a target object (or substitute object) within a space. For example, the electronic device can display a substitute object instead of the target object if the distance between the electronic device and the target object (or substitute object) is less than a threshold distance. If the distance between the electronic device and the target object (or substitute object) is greater than the threshold distance, the electronic device can restrict the display of the substitute object and display the target object. Since the substitute object can be displayed in place of the target object, the distance between the electronic device and the target object can be interpreted as being substantially the same as the distance between the electronic device and the substitute object.

[0268] FIG. 14 is a diagram illustrating an example of an operation of an electronic device according to various embodiments to stop or re-display an additional object or a replacement object based on a layout.

[0269] An electronic device according to one embodiment (e.g., electronic device (101) of FIG. 1, electronic device (201) of FIG. 2, electronic device (301) of FIG. 3, electronic device (401) of FIG. 4, electronic device (501) of FIG. 5) may, when a target object includes multiple partial objects (e.g., multiple partial target objects), stop displaying and / or re-display an additional object (1420) or a replacement object based on the relative arrangement between the multiple partial objects.

[0270] An electronic device may obtain a user input for selecting a target object including a plurality of partial objects when the relative arrangement between the plurality of partial objects is a first arrangement. The relative arrangement between the plurality of partial objects may refer to a distance and a direction in which the remaining partial objects are located with respect to at least one partial object among the plurality of partial objects. For example, in FIG. 14, the target object may include a first partial object (1411) and a second partial object (1412). The relative arrangement between the first partial object (1411) and the second partial object (1412) illustrated in the situation (1401) may be the first arrangement.

[0271] The electronic device may display an additional object (1420) at a location aligned along at least a portion of an outline of a target object including a plurality of partial objects arranged in a first arrangement. In situation (1401), the electronic device may display the additional object (1420) at a location aligned along at least a portion of an outline of a target object including a first partial object (1411) and a second partial object (1412) arranged in a first arrangement.

[0272] The electronic device may stop displaying the additional object (1420) based on a change in the relative arrangement of the plurality of partial objects from a first arrangement to a second arrangement. For example, the position and / or orientation of one partial object, either the first partial object (1411) or the second partial object (1412), within the space may change independently (or relative) to the position and / or orientation of the other partial object. In the situation (1402), the relative arrangement between the first partial object (1411) and the second partial object (1412) may be a second arrangement different from the first arrangement. The relative arrangement of the partial objects may change from the situation (1401) in which the first arrangement is the relative arrangement to the situation (1402) in which the second arrangement is the relative arrangement. In the situation (1402), the electronic device may limit the display of the additional object (1420) based on a change in the relative arrangement between the partial objects from the first arrangement.

[0273] The electronic device can display the additional object (1420) again at a position aligned along at least a portion of an outline of a target object including the plurality of partial objects arranged in the first arrangement, based on the return of the relative arrangement of the plurality of partial objects from the second arrangement to the first arrangement. The relative arrangement of the partial objects can be changed from a situation (1402) in which the relative arrangement of the partial objects is the second arrangement to a situation (1401) in which the relative arrangement of the partial objects is the first arrangement. In situation (1401), the electronic device can display the additional object (1420) again together with the target object including the first partial object (1411) and the second partial object (1412) arranged in the first arrangement, based on the return of the relative arrangement between the first partial object (1411) and the second partial object (1412) to the first arrangement.

[0274] In Fig. 14, the display of an additional object (1420) is mainly described, but is not limited thereto. An electronic device according to one embodiment can control the display of a substitute object based on the relative arrangement between a plurality of partial objects included in a target object within a space.

[0275] For example, an electronic device may obtain a user input for selecting a target object including a plurality of partial objects when the relative arrangement among the plurality of partial objects is a first arrangement. The electronic device may display a substitute object instead of the target object including the plurality of partial objects arranged in the first arrangement. The electronic device may stop displaying the substitute object based on a change in the relative arrangement of the plurality of partial objects from the first arrangement to the second arrangement. The electronic device may display the target object including the plurality of partial objects arranged in the second arrangement again instead of the substitute object. The electronic device may display the substitute object again instead of the target object based on a change in the relative arrangement of the plurality of partial objects from the second arrangement to the first arrangement.

[0276] In FIG. 14, for a target object including a plurality of partial objects, when the relative arrangement between the plurality of partial objects is a specific arrangement (e.g., a first arrangement), an additional object (or a substitute object) is displayed, and when the relative arrangement is another arrangement (e.g., a second arrangement), the additional object (or substitute object) is not displayed, but is not limited thereto. The electronic device may generate a plurality of candidate objects including the additional object and / or the substitute object, and display an object corresponding to the relative arrangement of the plurality of partial objects of the target objects among the plurality of candidate objects. For example, when the relative arrangement of the plurality of partial objects of the target object is the first arrangement, the electronic device may display an additional object or a substitute object corresponding to the first arrangement. When the relative arrangement of the plurality of partial objects of the target object is the second arrangement, the electronic device may display an additional object or a substitute object corresponding to the second arrangement.

[0277] FIG. 15 is a diagram illustrating an example of an operation of an electronic device selecting multiple target objects according to various embodiments.

[0278] An electronic device according to one embodiment (e.g., electronic device (101) of FIG. 1, electronic device (201) of FIG. 2, electronic device (301) of FIG. 3, electronic device (401) of FIG. 4, electronic device (501) of FIG. 5) can select a plurality of target objects.

[0279] The electronic device may detect a user input selecting a source object (1510). In FIG. 15, for example, in situation (1501), the electronic device may display a button (1511) corresponding to a cat object placed in space at a location corresponding to the cat object. When the electronic device detects a user input to the button (1511), the electronic device may select the cat object as the source object (1510).

[0280] An electronic device may select multiple target objects from among objects arranged in a space. For example, the electronic device may select an object to which an additional object or a replacement object based on a source object (1510) can be applied as a target object from among objects arranged in a space (or objects detected in a user's field of view). Whether an additional object or a replacement object can be applied to each object may be determined based on metadata of the object or properties of the object. In FIG. 15, as an example, the electronic device may select a first target object (1521) and a second target object (1522).

[0281] The electronic device may acquire additional objects for each of the plurality of target objects based on occupancy information and prompts of the plurality of target objects. For example, the electronic device may acquire a first additional object (1531) for a first target object (1521) and a second additional object (1532) for a second target object (1522). For the plurality of target objects, the electronic device may repeat the acquisition of input data including occupancy information of each target object and a prompt of the source object (1510), acquisition of output data including information about the additional object, and acquisition of the additional object.

[0282] The electronic device may display, for each of a plurality of target objects, an additional object corresponding to each of the plurality of target objects at a location aligned along the outlines of the plurality of target objects. In FIG. 15, in situation (1502), for example, the electronic device may display a first additional object (1531) at a location aligned along the outline of a first target object (1521). The electronic device may display a second additional object (1532) at a location aligned along the outline of a second target object (1522).

[0283] FIG. 16 is a block diagram illustrating an example configuration of an electronic device according to various embodiments.

[0284] An electronic device according to one embodiment (e.g., an electronic device (101) of FIG. 1, an electronic device (201) of FIG. 2, an electronic device (301) of FIG. 3, an electronic device (401) of FIG. 4, an electronic device (501) of FIG. 5) includes an image sensing device (1610) (e.g., a camera module (180) of FIG. 1), a user input sensing device (1620) (e.g., an input module (150) of FIG. 1, a sensor module (176) of FIG. 1), a processing device (1630) (e.g., a processor (120) of FIG. 1), an object extraction device (1640) (e.g., a processor (120) of FIG. 1), an object generation device (1650) (e.g., a processor (120) of FIG. 1), a display device (1660) (e.g., a display module (160) of FIG. 1), and a storage device (1670) (e.g., a memory (130) of FIG. 1). May include.

[0285] The image sensing device (1610) can sense image data regarding physical space and / or physical objects surrounding the electronic device. For example, the image sensing device (1610) can include a camera sensor and / or a depth sensor. The image sensing device (1610) can transmit the image data to the processing device (1630).

[0286] The user input sensing device (1620) can sense (e.g., detect) a user's input. The user input sensing device (1620) can transmit information about the detected user's input to the processing device (1630). The information about the user's input can include information indicating an object selected by the user's input. For example, the user input sensing device (1620) can detect a user's input selecting at least one of a source object and a target object.

[0287] The processing device (1630) can generate a stereoscopic image of a space to be provided to a user based on image data acquired from the image sensing device (1610). The processing device (1630) can receive information about an object of a virtual object type from an external device or acquire it from a storage device (1670). The processing device (1630) can generate and / or display a stereoscopic image including an object of a virtual object type.

[0288] The object extraction device (1640) can extract objects from image data and / or stereoscopic images.

[0289] The object generation device (1650) can generate additional objects or replacement objects based on source objects and target objects. The object generation device (1650) can transmit information about the generated additional objects or replacement objects to a storage device (1670). In FIG. 16, the object generation device (1650) is illustrated as being included as a component of an electronic device (e.g., on-device), but is not limited thereto, and the object generation device (1650) can be included in another electronic device external to the electronic device (e.g., on-cloud).

[0290] The display device (1660) can display the generated additional object together with the target object, or display the generated replacement object instead of the target object.

[0291] The storage device (1670) can store information about the generated additional object or target object.

[0292] The electronic device (101; 201; 301; 401; 501) may include a display (160; 205; 210; 320; 421). The electronic device (101; 201; 301; 401; 501) may include a processor (120). The electronic device (101; 201; 301; 401; 501) may include a memory (130) that stores instructions. When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) may be enabled to select a source object (811; 1011) and a target object (812; 910; 1012) from among objects arranged in a space (801; 1001). When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) may be caused to extract a prompt indicating an attribute of the source object (811; 1011) and occupancy information of the target object (812; 910; 1012) within the space (801; 1001). When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) may obtain, based on the prompt and the occupancy information, an additional object (830; 1420) having a shape matching at least a portion of an outline of the target object (812; 910; 1012) and having the properties of the source object (811; 1011).When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) may display the additional object (830; 1420) at a position aligned along at least a portion of the outline of the target object (812; 910; 1012) through the display (160; 205; 210; 320; 421).

[0293] When the instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) may be caused to restrict displaying the additional object (830; 1420) at a location corresponding to the display area (930) while the display area (930) is activated, based on the target object (812; 910; 1012) including another electronic device and the outline of the target object (812; 910; 1012) including the display area (930). When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) may display at least a portion of the additional object (830; 1420) at a location corresponding to the display area (930) while the display area (930) is inactive.

[0294] When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) may obtain a plurality of candidate additional objects corresponding to the plurality of operating states based on the target object (812; 910; 1012) being another electronic device (101; 201; 301; 401; 501) capable of operating in one of the plurality of operating states. When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) may select the additional object (830; 1420) corresponding to the operating state in which the target object (812; 910; 1012) is operating from among the plurality of candidate additional objects.

[0295] When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) may obtain motion (1131) of the additional object (830; 1420) based on the prompt and the occupancy information. When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) may display an additional object (830; 1420) moving at a position aligned along the outline of the target object (812; 910; 1012) by applying the motion (1131) of the acquired additional object (830; 1420) to the additional object (830; 1420).

[0296] When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) can detect a user input for selecting the source object (811; 1011). When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) can select a plurality of target objects from among the objects arranged in the space (801; 1001). When the instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) may be caused to, for each of the plurality of target objects, acquire an additional object for the corresponding target object based on the occupancy information of the corresponding target object and the prompt. When the instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) may be caused to, for each of the plurality of target objects, display an additional object corresponding to the corresponding target object at a position aligned along the outline of the corresponding target object.

[0297] When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) can obtain a user input for selecting the target object (812; 910; 1012) including a plurality of partial objects when the relative arrangement between the plurality of partial objects is a first arrangement. When the instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) may be caused to display the additional object (830; 1420) at the location aligned along the at least a portion of the outline of the target object (812; 910; 1012) including the plurality of partial objects arranged in the first arrangement. When the instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) may be caused to stop displaying the additional object (830; 1420) based on a change in the relative arrangement of the plurality of partial objects from the first arrangement to the second arrangement. When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) may be caused to redisplay the additional object (830; 1420) at the location based on the relative arrangement of the plurality of partial objects returning from the second arrangement to the first arrangement.

[0298] When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) may be caused to stop displaying the additional object (830; 1420) based on a distance from the target object (812; 910; 1012) to the electronic device (101; 201; 301; 401; 501) within the space (801; 1001) being greater than or equal to a threshold distance. When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) may be caused to display the additional object (830; 1420) again at the location based on a distance from the target object (812; 910; 1012) to the electronic device (101; 201; 301; 401; 501) within the space (801; 1001) being less than the threshold distance.

[0299] When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) may obtain, from the prompt and the occupancy information, a replacement object (1030) that can occupy the area occupied by the target object (812; 910; 1012) in the space (801; 1001). When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) may be caused to display the substitute object (1030) instead of the target object (812; 910; 1012) at a location corresponding to the target object (812; 910; 1012) through the display (160; 205; 210; 320; 421).

[0300] When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) may obtain a plurality of candidate motions of the additional object (830; 1420) from the prompt and the occupancy information, and a correspondence relationship (1132) between the plurality of candidate motions and candidate inputs of the user. When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) may cause the additional object (830; 1420) to display an additional object (830; 1420) moving at the location by applying a motion (1131) corresponding to the detected input among the plurality of candidate motions to the additional object (830; 1420) based on detecting a user input for the additional object (830; 1420).

[0301] When the instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) may obtain a haptic attribute (1133) of haptic feedback for the additional object (830; 1420) based on the prompt and the occupancy information. When the instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) may output a haptic feedback having the obtained haptic attribute (1133) based on detecting a user input for the additional object (830; 1420).

[0302] A method performed by an electronic device (101; 201; 301; 401; 501) may include an operation of selecting a source object (811; 1011) and a target object (812; 910; 1012) from among objects arranged in a space (801; 1001). The method performed by the electronic device (101; 201; 301; 401; 501) may include an operation of extracting a prompt indicating an attribute of the source object (811; 1011) and occupancy information of the target object (812; 910; 1012) within the space (801; 1001). The method performed by the electronic device (101; 201; 301; 401; 501) may include an operation of obtaining, based on the prompt and the occupancy information, an additional object (830; 1420) having a shape that matches at least a portion of an outline of the target object (812; 910; 1012) and having the properties of the source object (811; 1011). The method performed by the electronic device (101; 201; 301; 401; 501) may include an operation of displaying, through a display (160; 205; 210; 320; 421), the additional object (830; 1420) at a position aligned along at least a portion of the outline of the target object (812; 910; 1012).

[0303] The operation of displaying the additional object may include an operation of limiting display of the additional object (830; 1420) at a location corresponding to the display area (930) while the display area (930) is activated, based on the fact that the target object (812; 910; 1012) includes another electronic device (101; 201; 301; 401; 501) and the outline of the target object (812; 910; 1012) includes the display area (930). The operation of displaying the additional object may include an operation of displaying at least a part of the additional object (830; 1420) at a location corresponding to the display area (930) while the display area (930) is deactivated.

[0304] The operation of acquiring the additional object (830; 1420) may include an operation of acquiring a plurality of candidate additional objects corresponding to the plurality of operation states, based on the fact that the target object (812; 910; 1012) is another electronic device (101; 201; 301; 401; 501) capable of operating in one of the plurality of operation states. The operation of acquiring the additional object (830; 1420) may include an operation of selecting the additional object (830; 1420) corresponding to the operation state in which the target object (812; 910; 1012) is operating from among the plurality of candidate additional objects.

[0305] The operation of acquiring the additional object (830; 1420) may include an operation of acquiring a motion (1131) of the additional object (830; 1420) based on the prompt and the occupancy information. The operation of displaying the additional object (830; 1420) may include an operation of displaying the additional object (830; 1420) moving at a position aligned along the outline of the target object (812; 910; 1012) by applying the motion (1131) of the acquired additional object (830; 1420) to the additional object (830; 1420).

[0306] The operation of selecting the source object (811; 1011) and the target object (812; 910; 1012) may include an operation of detecting a user's input for selecting the source object (811; 1011). The operation of selecting the source object (811; 1011) and the target object (812; 910; 1012) may include an operation of selecting a plurality of target objects from among the objects arranged in the space (801; 1001). The operation of acquiring the additional object (830; 1420) may include an operation of acquiring an additional object for each of the plurality of target objects based on occupancy information of the corresponding target object and the prompt. The operation of displaying the additional object (830; 1420) may include an operation of displaying, for each of the plurality of target objects, an additional object corresponding to the target object at a position aligned along the outline of the target object.

[0307] The method may include an operation of obtaining a user input for selecting the target object (812; 910; 1012) when the relative arrangement between the plurality of partial objects is a first arrangement, for the target object (812; 910; 1012) including a plurality of partial objects. The method may include an operation of displaying the additional object (830; 1420) at the position aligned along the at least a portion of the outline of the target object (812; 910; 1012) including the plurality of partial objects arranged in the first arrangement. The method may include an operation of stopping displaying the additional object (830; 1420) based on a change in the relative arrangement of the plurality of partial objects from the first arrangement to the second arrangement. The method may include an action of re-displaying the additional object (830; 1420) at the location based on the relative arrangement of the plurality of partial objects returning from the second arrangement to the first arrangement.

[0308] The method may include an action of stopping displaying the additional object (830; 1420) based on a distance from the target object (812; 910; 1012) to the electronic device (101; 201; 301; 401; 501) within the space (801; 1001) being greater than or equal to a threshold distance. The method may include an action of re-displaying the additional object (830; 1420) at the location based on a distance from the target object (812; 910; 1012) to the electronic device (101; 201; 301; 401; 501) within the space (801; 1001) being less than the threshold distance.

[0309] The method may include an operation of obtaining a replacement object (1030) that can occupy an area occupied by the target object (812; 910; 1012) in the space (801; 1001) from the prompt and the occupancy information. The method may include an operation of displaying the replacement object (1030) instead of the target object (812; 910; 1012) at a location corresponding to the target object (812; 910; 1012) through the display (160; 205; 210; 320; 421).

[0310] The operation of obtaining the additional object (830; 1420) may include an operation of obtaining a plurality of candidate motions of the additional object (830; 1420) from the prompt and the occupancy information, and a correspondence relationship (1132) between the plurality of candidate motions and candidate inputs of the user. The operation of displaying the additional object (830; 1420) may include an operation of displaying the additional object (830; 1420) moving at the location by applying a motion (1131) corresponding to the detected input among the plurality of candidate motions to the additional object (830; 1420) based on detecting a user input for the additional object (830; 1420).

[0311] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0312] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0313] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0314] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0315] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0316] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0317] The embodiments described above may be implemented using hardware components, software components, and / or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and software applications running on the OS. Furthermore, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0318] Software may include computer programs, codes, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, or computer storage medium or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on a computer-readable recording medium.

[0319] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, and / or data structures, alone or in combination, and the program instructions recorded on the medium may be those specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROMs, RAMs, or flash memories. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter.

[0320] The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Claims

1. In an electronic device (101; 201; 301; 401; 501), display(160; 205; 210; 320; 421); At least one processor (120) comprising a processing circuit; and A memory (130) comprising one or more storage media storing instructions, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) Among the objects placed in the space (801; 1001), select the source object (811; 1011) and the target object (812; 910; 1012), Extracting a prompt indicating an attribute of the source object (811; 1011) and occupancy information of the target object (812; 910; 1012) within the space (801; 1001), Based on the above prompt and the occupancy information, an additional object (830; 1420) is obtained having a shape matching at least a part of an outline of the target object (812; 910; 1012) and having the above properties of the source object (811; 1011), Through the above display (160; 205; 210; 320; 421), the additional object (830; 1420) is displayed at a position aligned along at least a portion of the outline of the target object (812; 910; 1012). To do, Electronic devices (101; 201; 301; 401; 501).

2. In paragraph 1, When the above instructions are individually or collectively executed by at least one processor (120), Based on the fact that the target object (812; 910; 1012) includes another electronic device and the outline of the target object (812; 910; 1012) includes a display area (930), while the display area (930) is activated, the display of the additional object (830; 1420) is restricted at a location corresponding to the display area (930). While the above display area (930) is deactivated, display at least a portion of the additional object (830; 1420) at a location corresponding to the above display area (930). To do, Electronic devices (101; 201; 301; 401; 501).

3. In any one of paragraphs 1 and 2, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) Based on the fact that the target object (812; 910; 1012) is another electronic device (101; 201; 301; 401; 501) capable of operating in one of the plurality of operating states, a plurality of candidate additional objects corresponding to the plurality of operating states are obtained, Among the above multiple candidate additional objects, the additional object (830; 1420) corresponding to the operating state in which the target object (812; 910; 1012) is operating is selected. To do, Electronic devices (101; 201; 301; 401; 501).

4. In any one of paragraphs 1 to 3, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) Obtaining the motion (1131) of the additional object (830; 1420) based on the above prompt and the above occupancy information, By applying the motion (1131) of the acquired additional object (830; 1420) to the additional object (830; 1420), the additional object (830; 1420) is displayed moving at a position aligned along the outline of the target object (812; 910; 1012). To do, Electronic devices (101; 201; 301; 401; 501).

5. In any one of paragraphs 1 to 4, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) Detecting user input selecting the above source object (811; 1011), Among the objects placed in the above space (801; 1001), multiple target objects are selected, For each of the plurality of target objects, an additional object for the target object is obtained based on the occupancy information of the target object and the prompt. For each of the above multiple target objects, display an additional object corresponding to the target object at a location aligned along the outline of the target object. To do, Electronic devices (101; 201; 301; 401; 501).

6. In any one of paragraphs 1 to 5, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) For the target object (812; 910; 1012) including a plurality of partial objects, when the relative arrangement between the plurality of partial objects is a first arrangement, a user's input is obtained for selecting the target object (812; 910; 1012), Displaying the additional object (830; 1420) at the position aligned along at least a portion of the outline of the target object (812; 910; 1012) including the plurality of partial objects arranged in the first arrangement, Based on the change in the relative arrangement of the plurality of partial objects from the first arrangement to the second arrangement, the display of the additional object (830; 1420) is stopped, Based on the relative arrangement of the plurality of partial objects returning from the second arrangement to the first arrangement, the additional object (830; 1420) is displayed again at the location. To do, Electronic devices (101; 201; 301; 401; 501).

7. In any one of paragraphs 1 to 6, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) Based on the distance from the target object (812; 910; 1012) to the electronic device (101; 201; 301; 401; 501) within the space (801; 1001) being greater than or equal to a threshold distance, the display of the additional object (830; 1420) is stopped, Based on the distance from the target object (812; 910; 1012) to the electronic device (101; 201; 301; 401; 501) within the space (801; 1001) being less than the threshold distance, the additional object (830; 1420) is displayed again at the location. To do, Electronic devices (101; 201; 301; 401; 501).

8. In any one of paragraphs 1 to 7, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) From the above prompt and the occupancy information, a replacement object (1030) that can occupy the area occupied by the target object (812; 910; 1012) in the space (801; 1001) is obtained, Through the above display (160; 205; 210; 320; 421), the replacement object (1030) is displayed instead of the target object (812; 910; 1012) at a location corresponding to the target object (812; 910; 1012). To do, Electronic devices (101; 201; 301; 401; 501).

9. In any one of paragraphs 1 to 8, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) Obtaining a plurality of candidate motions of the additional object (830; 1420) from the above prompt and the above occupancy information, and a correspondence relationship (1132) between the plurality of candidate motions and the user's candidate inputs, Based on detecting a user's input for the additional object (830; 1420), by applying a motion (1131) corresponding to the detected input among the plurality of candidate motions to the additional object (830; 1420), the additional object (830; 1420) moving at the location is displayed. To do, Electronic devices (101; 201; 301; 401; 501).

10. In any one of paragraphs 1 to 9, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) Obtaining a haptic attribute (1133) of haptic feedback for the additional object (830; 1420) based on the above prompt and the above occupancy information, Based on detecting the user's input to the above additional object (830; 1420), outputting a haptic feedback having the acquired haptic property (1133). To do, Electronic devices (101; 201; 301; 401; 501).

11. In a method performed by an electronic device (101; 201; 301; 401; 501), An action of selecting a source object (811; 1011) and a target object (812; 910; 1012) among objects placed in a space (801; 1001); An action of extracting a prompt indicating an attribute of the source object (811; 1011) and occupancy information of the target object (812; 910; 1012) within the space (801; 1001); An operation of obtaining an additional object (830; 1420) having a shape matching at least a part of an outline of the target object (812; 910; 1012) and having the properties of the source object (811; 1011), based on the above prompt and the above occupancy information; and An action of displaying said additional object (830; 1420) at a location aligned along at least a portion of said outline of said target object (812; 910; 1012) via a display (160; 205; 210; 320; 421). A method including:

12. In paragraph 11, The action of displaying the above additional objects is: An operation of limiting displaying the additional object (830; 1420) at a location corresponding to the display area (930) while the display area (930) is activated, based on the target object (812; 910; 1012) including another electronic device (101; 201; 301; 401; 501) and the outline of the target object (812; 910; 1012) including the display area (930); and An operation comprising displaying at least a portion of the additional object (830; 1420) at a location corresponding to the display area (930) while the display area (930) is inactive. method.

13. In any one of paragraphs 11 to 12, The operation of obtaining the above additional object (830; 1420) is as follows: An operation of obtaining a plurality of candidate additional objects corresponding to the plurality of operation states based on the target object (812; 910; 1012) being another electronic device (101; 201; 301; 401; 501) capable of operating in one of the plurality of operation states; and An operation including selecting the additional object (830; 1420) corresponding to the operating state in which the target object (812; 910; 1012) is operating among the plurality of candidate additional objects. method.

14. In any one of paragraphs 11 to 13, The operation of obtaining the above additional object (830; 1420) is as follows: An operation for obtaining motion (1131) of the additional object (830; 1420) based on the above prompt and the above occupancy information, The action of displaying the above additional object (830; 1420) is: An operation of displaying an additional object (830; 1420) moving at a position aligned along the outline of the target object (812; 910; 1012) by applying the motion (1131) of the acquired additional object (830; 1420) to the additional object (830; 1420), method.

15. A computer-readable recording medium storing one or more computer programs including commands for performing the method of any one of claims 11 to 14.

Images