Electronic device for generating virtual object and operation method thereof

Abstract

An electronic device according to one or more embodiments comprises: a display; a camera for acquiring an image; a memory for storing at least one instruction; and at least one processor for executing the at least one instruction, wherein the at least one processor executes the at least one instruction to: acquire space information about a real space on the basis of the image acquired by the camera; acquire a user input on the basis of the image acquired by the camera; acquire object characteristic information from the user input; acquire object generation information for generating a virtual object on the basis of the space information and the object characteristic information; generate a virtual object related to object generation information by inputting the object generation information to a generative artificial intelligence model trained to generate a three-dimensional virtual object on the basis of information about a space and an object; and control the display to display the virtual object.

Description

Electronic device for generating virtual objects and method for operating the same

[0001] The present disclosure relates to an electronic device for generating a three-dimensional virtual object in real space and an operating method thereof. Specifically, the present disclosure discloses an electronic device for generating a three-dimensional virtual object using various modality information, and for providing the generated three-dimensional virtual object, and an operating method thereof.

[0002] Augmented reality (AR) is a technology that overlays virtual objects onto the physical environment or objects of the real world, presenting them alongside one another. Its advantage lies in the fusion of virtual objects and information within the real world. AR devices (e.g., smart glasses) utilizing AR technology are being used in everyday life for various purposes, such as information retrieval, route guidance, and camera photography. In particular, smart glasses are often worn as fashion accessories and are primarily used for outdoor activities.

[0003] Augmented reality technologies currently in use and widespread use can display virtual objects by placing them in real space, but the placement of virtual objects is limited to placing pre-modeled 3D virtual objects. For example, 3D object modeling technology creates 3D virtual objects using depth information, pose information, and texture information, but has the technical limitation of not being able to create objects that do not exist in reality. For another example, a modeling method using a deep neural network model (Deep Learning) performs object modeling by optimizing 2D images and position data of objects taken from various locations using a trained deep neural network model, thereby creating 3D virtual objects. However, this method also has the limitation of not being able to be used when creating objects that do not exist in reality. Furthermore, the virtual object creation technologies currently in use and widespread cannot modify the created 3D virtual objects.

[0004] As mentioned above, current augmented reality technology cannot create virtual objects with non-existent shapes, and also cannot allow users to modify created virtual objects, thus providing only a limited augmented reality experience, which may hinder the expansion of augmented reality services.

[0005] An electronic device according to one or more exemplary embodiments includes a display; a camera for acquiring an image; a memory for storing at least one instruction; and at least one processor for executing the at least one instruction, wherein the at least one processor executes the at least one instruction, such that the electronic device acquires spatial information about a real space based on an image acquired through the camera, acquires a user input based on the image acquired through the camera, acquires object characteristic information from the user input, acquires object generation information for generating a virtual object based on the spatial information and the object characteristic information, inputs the object generation information into a generative artificial intelligence model trained to generate a three-dimensional virtual object based on information about space and an object, generates a virtual object for the object generation information, and controls the display to display the virtual object.

[0006] A method for generating a virtual object based on multi-modality information, performed in an electronic device according to one or more exemplary embodiments, may include: acquiring an image through a camera; acquiring a user input based on the image acquired through the camera; acquiring spatial information about a real space from the image; acquiring object characteristic information from the user input; acquiring object generation information for generating a virtual object based on the spatial information and the object characteristic information; inputting the acquired object generation information into a generative artificial intelligence model trained to generate a three-dimensional virtual object based on information about a space and an object, thereby generating a virtual object for the object generation information; and displaying the virtual object on a display.

[0007] A computer program product according to one or more exemplary embodiments includes a computer-readable storage medium, wherein the storage medium may include instructions readable by an electronic device, such that the electronic device performs the following operations: acquiring an image through a camera; acquiring a user input based on the image acquired through the camera; acquiring spatial information about a real space from the image; acquiring object characteristic information from the user input; acquiring object generation information for generating a virtual object based on the spatial information and the object characteristic information; inputting the object generation information into a generative artificial intelligence model trained to generate a three-dimensional virtual object based on information about a space and an object, thereby generating a virtual object for the object generation information; and displaying the virtual object on a display.

[0008] A head-mounted display (HMD) according to one or more exemplary embodiments comprises: a display; a camera configured to acquire an image; a memory storing at least one instruction; and by executing the at least one instruction, acquire spatial information about a real space based on an image acquired through the camera; acquire user input based on the image acquired through the camera; acquire object characteristic information based on the user input, the object characteristic information indicating an object to occupy a real-world space; acquire object generation information for generating a virtual object corresponding to the object to occupy the real-world space based on the spatial information and the object characteristic information; input the object generation information into a generative artificial intelligence model to generate a virtual object for the object generation information; and control the display to display the virtual object in the real space.

[0009] The embodiments may be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings.

[0010] FIG. 1 is a conceptual diagram illustrating an operation of an electronic device of the present disclosure to create and display a three-dimensional virtual object based on multi-modality information.

[0011] FIG. 2 is a flowchart illustrating a method of operating an electronic device according to one or more embodiments of the present disclosure.

[0012] FIG. 3 is a block diagram illustrating components of an electronic device according to one or more embodiments of the present disclosure.

[0013] FIG. 4 is a diagram illustrating data input / output between a software module stored in a memory and a camera, a microphone, and a display unit of an electronic device according to one or more embodiments of the present disclosure.

[0014] FIG. 5 is a flowchart illustrating a method by which an electronic device according to one or more embodiments of the present disclosure obtains a vector representing object creation information.

[0015] FIG. 6 is a diagram illustrating an operation of an electronic device according to one or more embodiments of the present disclosure to obtain feature data representing object creation information by embedding multi-modality information.

[0016] FIG. 7 is a diagram illustrating an operation of an electronic device according to one or more embodiments of the present disclosure to assign a missing vector value to feature data representing object creation information.

[0017] FIG. 8 is a flowchart illustrating a method for an electronic device to generate a three-dimensional virtual object by modifying object generation information based on user input according to one or more embodiments of the present disclosure.

[0018] FIG. 9 is a diagram illustrating components of an electronic device that creates and displays a three-dimensional virtual object by modifying object creation information according to one or more embodiments of the present disclosure.

[0019] FIG. 10 is a diagram illustrating an operation of an electronic device according to one or more embodiments of the present disclosure to modify object creation information by receiving a user input for adjusting a weight of multi-modality information.

[0020] FIG. 11 is a diagram illustrating an operation of an electronic device according to one or more embodiments of the present disclosure to modify object creation information by receiving a user input for modifying the shape, size, position, etc. of an object.

[0021] FIG. 12A is a diagram illustrating an operation of an electronic device according to one or more embodiments of the present disclosure to generate and display a three-dimensional virtual object based on multi-modality information.

[0022] FIG. 12B is a diagram illustrating an operation of an electronic device according to one or more embodiments of the present disclosure to generate and display a three-dimensional virtual object based on multi-modality information.

[0023] FIG. 12c is a diagram illustrating an operation of an electronic device according to one or more embodiments of the present disclosure to generate and display a three-dimensional virtual object based on multi-modality information.

[0024] FIG. 12d is a diagram illustrating an operation of an electronic device according to one or more embodiments of the present disclosure to generate and display a three-dimensional virtual object based on multi-modality information.

[0025] FIG. 12E is a diagram illustrating an operation of an electronic device according to one or more embodiments of the present disclosure to generate and display a three-dimensional virtual object based on multi-modality information.

[0026] The terms used in the embodiments of this specification have been selected from widely used, current terms, taking into account the functions of the present disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant embodiments. Therefore, the terms used in this specification should not be defined simply as names of terms, but rather based on their meanings and the overall content of the present disclosure.

[0027] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art described herein.

[0028] Throughout this disclosure, when a part is said to "include" a component, this does not exclude other components, but rather implies the inclusion of other components, unless otherwise specifically stated. Furthermore, terms such as "part," "module," etc., used herein refer to a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software.

[0029] As used herein, the expression "configured to" can be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" does not necessarily mean something is "specifically designed to" in terms of hardware. Instead, in some contexts, the expression "a system configured to" can mean that the system is "capable of" in conjunction with other devices or components. For example, the phrase "a processor configured to perform A, B, and C" can mean a dedicated processor for performing the operations (e.g., an embedded processor), or a general-purpose processor (e.g., a CPU or an application processor) that can perform the operations by executing one or more software programs stored in memory.

[0030] Additionally, when a component is referred to as being "connected" or "connected" to another component in the present disclosure, it should be understood that the component may be directly connected or connected to the other component, but may also be connected or connected via another component in between, unless otherwise specifically stated.

[0031] In the present disclosure, an "electronic device" may be a mobile device, a smart phone, or a tablet PC. However, the electronic device is not limited thereto, and may be implemented as various electronic devices such as a laptop computer, a desktop, an e-book reader, a digital broadcasting terminal, a PDA (Personal Digital Assistant), a PMP (Portable Multimedia Player), a navigation device, an MP3 player, a camcorder, an IPTV (Internet Protocol Television), a DTV (Digital Television), a wearable device, etc.

[0032] In one or more embodiments of the present disclosure, the electronic device may be an augmented reality device. An 'augmented reality device' is a device capable of expressing augmented reality, and may be implemented as, for example, augmented reality glasses in the shape of glasses worn by a user on the face. However, the augmented reality device is not limited thereto, and may also be implemented as a head-mounted display apparatus (HMD) worn on the user's head, an augmented reality helmet, or the like.

[0033] In this disclosure, 'Augmented Reality' means displaying a virtual image together within a physical environment space of the real world or displaying a real object and a virtual image together.

[0034] In this disclosure, "real world space" refers to a space in the real world viewed by a user through an augmented reality device. In one or more embodiments of the present disclosure, "real world space" may refer to an indoor space. Real world objects may be placed within the real world.

[0035] In the present disclosure, a "virtual object" is an image generated by an optical engine, and may include both static and dynamic images. Such a virtual object is observed along with a real scene, and may be an image representing information about a real object in the real scene, information about the operation of an augmented reality device, a control menu, or the like. In one or more embodiments of the present disclosure, the "virtual object" may include a user interface (UI) provided through an application or program executed by an augmented reality device.

[0036] In the present disclosure, a 'three-dimensional (3D) virtual object' means a virtual object implemented as a three-dimensional stereoscopic shape.

[0037] In the present disclosure, functions related to 'artificial intelligence' are operated through a processor and memory. The processor may be composed of one or more processors. In this case, one or more processors may be a general-purpose processor such as a CPU, AP, or DSP (Digital Signal Processor), a graphics-only processor such as a GPU or VPU (Vision Processing Unit), or an artificial intelligence-only processor such as an NPU. One or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in memory. Alternatively, if one or more processors are artificial intelligence-only processors, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0038] The predefined operation rules or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that the basic artificial intelligence model is trained using a learning algorithm using a plurality of learning data, thereby creating a predefined operation rules or artificial intelligence model set to perform a desired characteristic (or purpose). This learning may be performed on the device itself on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and / or system. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0039] In the present disclosure, the 'artificial intelligence model' may be composed of a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and performs neural network operations through operations between the operation results of the previous layer and the plurality of weights. The plurality of weights of the plurality of neural network layers may be optimized based on the learning results of the artificial intelligence model. For example, the plurality of weights may be updated so that the loss value or cost value obtained from the artificial intelligence model during the learning process is reduced or minimized. The artificial neural network model may include a deep neural network (DNN), and examples thereof include, but are not limited to, a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or deep Q-networks.

[0040] In the present disclosure, 'vision recognition' refers to image signal processing that inputs an image into an artificial intelligence model and, through inference using the artificial intelligence model, recognizes (detection) an object from the input image, classifies the object into a specific category, or segments the object. In one or more embodiments of the present disclosure, vision recognition may refer to image processing that uses an artificial intelligence model to recognize an object from an image captured by a camera and obtains classification information or type information of the object.

[0041] Below, embodiments of the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

[0042] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0043] FIG. 1 is a conceptual diagram illustrating an operation of an electronic device (100) of the present disclosure to create and display a three-dimensional virtual object (70) based on multi-modality input.

[0044] Referring to FIG. 1, the electronic device (100) may be an augmented reality glasses device in the shape of glasses worn on the user's face. However, the present invention is not limited thereto, and in one or more embodiments of the present disclosure, the electronic device (100) may be implemented as various electronic devices such as a mobile device, a smart phone, a laptop computer, a desktop, a tablet PC, an e-book reader, a digital broadcasting terminal, a PDA (Personal Digital Assistants), a PMP (Portable Multimedia Player), a navigation device, an MP3 player, a camcorder, an IPTV (Internet Protocol Television), a DTV (Digital Television), a wearable device, etc.

[0045] An electronic device (100) may include a camera (110). Referring to FIG. 1, the electronic device (100) may capture a real space (10) using the camera (110) to obtain a space image (20), receive a user's gesture input (30) and voice input (40), and obtain a two-dimensional (2D) guide image (50) (operation ①).

[0046] The electronic device (100) can obtain spatial information from a spatial image (20) and object characteristic information from a gesture input (30), a voice input (40), and a two-dimensional guide image (50) (operation ②).

[0047] The electronic device (100) can obtain object creation information (60) by mixing the acquired spatial information and object characteristic information (operation ③).

[0048] An electronic device (100) can generate a three-dimensional (3D) virtual object (70) by performing inferencing to input object creation information into a generative artificial intelligence model (146) (operation ④).

[0049] The electronic device (100) can display the generated three-dimensional virtual object (70) (operation ⑤).

[0050] Hereinafter, the function and / or operation of the electronic device (100) will be described in detail with reference to FIG. 1 and FIG. 2 together.

[0051] FIG. 2 is a flowchart illustrating a method of operating an electronic device (100) according to one or more embodiments of the present disclosure.

[0052] In step S210, the electronic device (100) acquires an image through a camera. Referring to FIG. 1, the electronic device (100) can acquire a spatial image (20) by photographing a real space (10) using a camera (110).

[0053] In step S220, the electronic device (100) acquires a user input based on an image acquired through the camera (110). In one or more embodiments of the present disclosure, the electronic device (100) may capture a hand image by using the camera (110), recognize the user's hand from the hand image, and recognize a gesture made by the recognized user's hand. However, the present invention is not limited thereto, and the electronic device (100) may be implemented as a mobile device including a touchscreen, a smart phone, or a tablet PC. In this case, the electronic device (100) may recognize a gesture input from a user's touch input received through the touchscreen. In the embodiment illustrated in FIG. 1, the electronic device (100) may recognize a user's gesture input (30) drawing a desk shape.

[0054] In one or more embodiments of the present disclosure, the electronic device (100) includes a microphone (120) and can receive a voice input (40) from a user through the microphone (120). In the embodiment illustrated in FIG. 1, the electronic device (100) can receive a voice input (40) such as “Make me a desk that fits in the office.”

[0055] In one or more embodiments of the present disclosure, the electronic device (100) may obtain a two-dimensional (2D) guide image (50). In the present disclosure, the 'two-dimensional guide image (50)' means a two-dimensional image including guide information regarding at least one of the type, shape, or theme of an object for generating a three-dimensional virtual object (70). In the embodiment illustrated in FIG. 1, the two-dimensional guide image (50) may be a two-dimensional image representing a desk having iron legs and a wooden top. In one or more embodiments of the present disclosure, the electronic device (100) may receive the two-dimensional guide image (50) from an external server or storage device. However, the present disclosure is not limited thereto, and the electronic device (100) may obtain the two-dimensional guide image (50) input by a user input.

[0056] In step S230, the electronic device (100) obtains spatial information about a real space from the acquired image. In the present disclosure, 'spatial information' refers to information about a real space, and may include, for example, information about the type, category, or atmosphere of the real space. Referring also to FIG. 1, the electronic device (100) may obtain spatial information about a real space (10) from a spatial image (20). In one or more embodiments of the present disclosure, the electronic device (100) may perform vision recognition using an artificial intelligence model to recognize the type of the real space (10) from the spatial image, classify the category, and recognize objects within the real space (10). For example, the artificial intelligence model may be a deep neural network model trained to acquire spatial information such as the type, category, or atmosphere of a space from a spatial image (20) through a supervised learning method that applies multiple images of a space as input and applies label values ​​such as the type, category, or atmosphere of the space as ground truth. In the embodiment illustrated in FIG. 1, the electronic device (100) may acquire information about the type and category of a space called 'office' and information about the atmosphere or theme of a space called 'gray tone' from the spatial image (20).

[0057] In step S240, the electronic device (100) acquires object characteristic information from the acquired user input. In the present disclosure, 'object characteristic information' is information used to generate a three-dimensional virtual object (70) according to the user's intent, and may include, for example, information about the shape, location, size, type, category, color, or theme of the object. That is, the object characteristic information may indicate an object that will occupy a real-world space.

[0058] The electronic device (100) can obtain object characteristic information according to the user's intention from the user's gesture input (30) obtained from an image acquired through the camera (110). In one or more embodiments of the present disclosure, the electronic device (100) can recognize an area pointed by the user's hand and a trajectory of the area from the image, and recognize a gesture through the area. The electronic device (100) can extract object characteristic information including at least one of the position, size, and shape of the object from the recognized gesture input (30). In the embodiment illustrated in FIG. 1, the electronic device (100) can extract object characteristic information including at least one of the position, size, and shape of a desk from the gesture input (30).

[0059] The electronic device (100) can convert a voice input (40) received through a microphone (120) into a voice signal and perform Automatic Speech Recognition (ASR) to convert the voice signal into text. The electronic device (100) can analyze the text using a natural language understanding model and identify the user's intent and named entity based on the analysis result. The electronic device (100) can obtain object characteristic information including at least one of the type, shape, color, and theme of the object according to the user's intent and named entity based on the analysis result of the text. In the embodiment illustrated in FIG. 1, the electronic device (100) can obtain the intent of 'object creation' and information on named entities such as 'desk' and 'office' from the voice input (40).

[0060] The electronic device (100) can obtain object characteristic information from a two-dimensional guide image (50). In one or more embodiments of the present disclosure, the electronic device (100) can extract object characteristic information including at least one of the type, shape, color, and theme of the object from the two-dimensional guide image (50). In the embodiment illustrated in FIG. 1, the electronic device (100) can extract object characteristic information regarding the type of the object called a 'desk', the shape of the desk including a wooden top and iron legs, the color including brown and black, and the like from the two-dimensional guide image (50).

[0061] In step S250, the electronic device (100) obtains object creation information based on spatial information and object characteristic information. Referring to FIG. 1 together, the electronic device (100) may obtain object creation information (60) by mixing spatial information including at least one of the type, category, and theme of the real space (10) obtained from a spatial image (20), object characteristic information including at least one of the position, size, and shape of an object obtained from a gesture input (30), object characteristic information including at least one of the type, shape, color, and theme of an object obtained from a voice input (40), and object characteristic information including at least one of the type, shape, color, and theme of an object obtained from a two-dimensional guide image (50). In one or more embodiments of the present disclosure, the electronic device (100) may convert spatial information and object characteristic information into feature data by vector embedding, and may obtain feature data representing the object creation information (60) by mixing each of the converted feature data. 'Feature data' may be composed of an n-dimensional vector of one or more dimensions, a matrix, a tensor including a feature vector, etc. In one or more embodiments of the present disclosure, the electronic device (100) may convert spatial information and object characteristic information into feature vectors by vector embedding them, and may obtain a feature vector representing object generation information (60) by mixing each of the converted feature vectors in a multi-layer perceptron manner.However, the present invention is not limited thereto, and in one or more embodiments of the present disclosure, the electronic device (100) may obtain feature data representing object creation information (60) by inputting each feature data converted into a convolutional neural network (CNN) or a recurrent neural network (RNN).

[0062] In step S260, the electronic device (100) may input object generation information (60) into a generative artificial intelligence model to generate a three-dimensional virtual object based on the object generation information. In the present disclosure, the 'generative artificial intelligence model (146, see FIGS. 1 and 3)' is an artificial intelligence model trained to generate a three-dimensional virtual object based on spatial information and object information. In one or more embodiments of the present disclosure, the generative artificial intelligence model (146) may be implemented as a Generative Adversarial Network (GAN) or a diffusion model based on multi-modality. The diffusion model based on multi-modality may be implemented as, for example, Stable Diffusion or SDFusion. However, the present disclosure is not limited thereto, and the generative artificial intelligence model (146) may include any known artificial intelligence model trained to generate a three-dimensional virtual object based on spatial information and object information.

[0063] Referring to FIG. 1, the electronic device (100) can input object creation information (60) into a generative artificial intelligence model (146) and perform inferencing using the generative artificial intelligence model (146) to create a three-dimensional virtual object (70) for the object creation information (60). In the embodiment illustrated in FIG. 1, the electronic device (100) can create a three-dimensional virtual object (70), which is a virtual image representing the three-dimensional shape of a desk, through the generative artificial intelligence model (146).

[0064] In operation ⑤ of FIG. 1, the electronic device (100) can display the generated three-dimensional virtual object (70). In one or more embodiments of the present disclosure, the electronic device (100) may be configured with augmented reality glasses, and the augmented reality glasses may include a display unit (150, see FIG. 3) configured with a lens optical system and including a wave guide and an optical engine. In the embodiment illustrated in FIG. 1, the electronic device (100) may generate light of a virtual image constituting the three-dimensional virtual object (70) using the optical engine, and display the three-dimensional virtual object (70) on the real space (10) by projecting the light onto the wave guide. However, the present disclosure is not limited thereto, and when the electronic device (100) is implemented as a mobile device, for example, a smart phone or a tablet PC, in one or more embodiments of the present disclosure, the electronic device (100) may display a three-dimensional virtual object (70) through a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display, a 3D display, or an electrophoretic display.

[0065] Currently used and widely used AR technology can display virtual objects by placing them in real space. However, the virtual object placement is limited to the placement of pre-modeled 3D virtual objects. In other words, current AR technology cannot create 3D virtual objects with non-existent shapes, nor can users modify these created 3D virtual objects, providing only a limited AR experience. Consequently, this can hinder the expansion of AR services and reduce their usability.

[0066] The present disclosure aims to provide an electronic device (100) that generates a three-dimensional virtual object based on multi-modality information including at least one of a spatial image of a real space, a gesture input received from a user, a voice input, and a two-dimensional guide image, and provides the three-dimensional virtual object, and an operating method thereof.

[0067] The electronic device (100) of the present disclosure generates a three-dimensional virtual object (70) using various modalities, i.e., spatial information acquired from a spatial image (20), gesture input (30), voice input (40), and object characteristic information acquired from a two-dimensional guide image, thereby quickly generating and providing a three-dimensional model of a virtual object desired by a user. Accordingly, the electronic device (100) of the present disclosure provides a technical effect that enables an active augmented reality experience to the user. In addition, the electronic device (100) of the present disclosure generates a three-dimensional virtual object (70) using multi-modality input information in accordance with an augmented reality service environment that supports various input methods, thereby providing a convenient and highly usable augmented reality experience to the user.

[0068] An embodiment in which an electronic device (100) modifies a three-dimensional virtual object (70) based on user input will be described in detail in FIGS. 8 to 11.

[0069] FIG. 3 is a block diagram illustrating components of an electronic device (100) according to one or more embodiments of the present disclosure.

[0070] Referring to FIG. 3, the electronic device (100) may include a camera (110), a microphone (120), a processor (130), a memory (140), and a display unit (150). The camera (110), the microphone (120), the processor (130), the memory (140), and the display unit (150) may be electrically and / or physically connected to each other, respectively. Only components for explaining the operation of the electronic device (100) are illustrated in FIG. 3, and the components included in the electronic device (100) are not limited to those illustrated in FIG. 3. In one or more embodiments of the present disclosure, the electronic device (100) may further include a communication interface for performing data communication with an external device or server. In one or more embodiments of the present disclosure, the electronic device (100) is implemented as a portable device, in which case the electronic device (100) may further include a battery that supplies driving power to the camera (110), the microphone (120), the processor (130), and the display unit (150).

[0071] The camera (110) is configured to acquire images of a real space and objects within the real space. The camera (110) may include a lens module, an image sensor, and an image processing module. The camera (110) may acquire still images or video of an object by the image sensor (e.g., CMOS or CCD). The video may include a plurality of image frames continuously acquired by photographing the object through the camera (110). The image processing module may encode still images composed of a single image frame acquired through the image sensor or video data composed of a plurality of image frames and transmit them to the processor (130). In one or more embodiments of the present disclosure, the camera (110) may include a first camera (112) and a second camera (114).

[0072] The first camera (112) is configured to capture a real space and obtain a spatial image of the real space. In one or more embodiments of the present disclosure, the first camera (112) may be configured as a camera including an image sensor, such as an RGB camera or a grayscale camera. In one or more embodiments of the present disclosure, the first camera (112) may be implemented in a small form factor so as to be mounted on a portable electronic device (100), and may be implemented as a lightweight RGB camera that consumes low power.

[0073] The second camera (114) is configured to capture a hand image by photographing the user's hand. The second camera (114) may be configured as an RGB camera including an image sensor, but is not limited thereto. In one or more embodiments of the present disclosure, the second camera (114) may also be configured as a depth camera that acquires a depth value of an object (e.g., the user's hand). When the second camera (114) is configured as a depth camera, the second camera (114) may be implemented as at least one of a Time-Of-Flight (ToF) camera, a stereo vision camera, or a Light Detection And Ranging (LiDAR) sensor, for example. The second camera (114) may measure three-dimensional depth values ​​of each joint included in the user's hand to acquire a depth map.

[0074] The microphone (120) can receive a voice input (e.g., the user's speech) from a user and acquire a voice signal from the received voice input. The microphone (120) provides the acquired voice signal to the processor (130).

[0075] The processor (130) can execute one or more instructions of a program stored in the memory (140). The processor (130) may be composed of hardware components that perform arithmetic, logic, and input / output operations, as well as image processing. Although the processor (130) is illustrated as a single element in FIG. 3 , it is not limited thereto. In one or more embodiments of the present disclosure, the processor (130) may be composed of one or more elements.

[0076] The processor (130) may include various processing circuits and / or multiple processors. For example, the term "processor" as used in this disclosure, including the claims, may include various processing circuits, including at least one processor. One or more processors in at least one processor may be configured to perform various functions described in this disclosure individually and / or collectively in a distributed manner. As used herein, "processor," "at least one processor," and "one or more processors" may be configured to perform various functions. However, these terms encompass, without limitation, situations where one processor performs some of the functions and other processor(s) perform other parts of the functions, and situations where a single processor may perform all of the functions. Furthermore, at least one processor may include a combination of processors that perform various functions of the disclosed functions in a distributed manner. At least one processor may execute program instructions to achieve or perform various functions.

[0077] The processor (130) may be a general-purpose processor such as a CPU (Central Processing Unit), an AP (Application Processor), a DSP (Digital Signal Processor), a graphics-only processor such as a GPU (Graphics Processing Unit), a VPU (Vision Processing Unit), or an AI-only processor such as an NPU (Neural Processing Unit). The processor (130) may be controlled to process input data according to predefined operating rules or an AI model. Alternatively, if the processor (130) is an AI-only processor, the AI-only processor may be designed with a hardware structure specialized for processing a specific AI model.

[0078] The memory (140) may be configured as at least one type of storage medium, for example, a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), or an optical disk.

[0079] The memory (140) may store instructions related to functions and / or operations that enable the electronic device (100) to obtain spatial information from a spatial image, obtain object characteristic information from a gesture input, a voice input, and a two-dimensional guide image, and generate a three-dimensional virtual object using the spatial information and the object characteristic information to provide an augmented reality service. In one or more embodiments of the present disclosure, the memory (140) may store at least one of instructions, an algorithm, a data structure, a program code, and an application program that can be read by the processor (130). The instructions, algorithms, data structures, and program codes stored in the memory (140) may be implemented in a programming or scripting language such as, for example, C, C++, Java, or an assembler.

[0080] The memory (140) may store commands, algorithms, data structures, or program codes related to a spatial information extraction module (141), a gesture information extraction module (142), a voice information extraction module (143), a guide information extraction module (144), an information mixing module (145), a generative artificial intelligence model (146), and an object modification module (147). A 'module' included in the memory (140) refers to a unit that processes a function or operation performed by the processor (130), and this may be implemented as software such as commands, algorithms, data structures, or program codes.

[0081] The processor (130) can be implemented by executing instructions or program codes stored in the memory (140). Hereinafter, with reference to FIGS. 3 and 4 together, the functions and / or operations performed by the processor (130) by executing instructions or program codes of each of the plurality of modules stored in the memory (140) and the data input / output between the plurality of modules and components (e.g., the camera (110), the microphone (120), and the display unit (150)) will be described in detail.

[0082] FIG. 4 is a diagram illustrating data input / output between a software module stored in a memory (140) and a camera (110), a microphone (120), and a display unit (150) of an electronic device (100) according to one or more embodiments of the present disclosure.

[0083] Referring to FIGS. 3 and 4 together, the spatial information extraction module (141) is configured with commands or program codes related to a function and / or operation of extracting spatial information about a real space from a spatial image acquired through the first camera (112). In one or more embodiments of the present disclosure, 'spatial information' may include information about at least one of a type, category, color, theme, and atmosphere of a real space. The type or category of a real space refers to classification information of a space acquired as a result of inferring by an artificial intelligence model, and refers to, for example, an office, a cafe, a school, a library, a playground, etc. The color, theme, or atmosphere of a real space refers to characteristic information by the color or tone of the space, and refers to, for example, information about a gray tone, a brown tone, a green tone, etc. In one or more embodiments of the present disclosure, the spatial information extraction module (141) may include an artificial intelligence model trained to recognize the type of a real space from a spatial image, classify the category, and recognize the color, theme, etc. For example, an AI model may be a deep neural network model trained to acquire spatial information, such as the type, category, or atmosphere of a real space, from spatial images through a supervised learning method that inputs multiple images of a space and applies label values, such as the type or category of the space, as ground truth. The deep neural network model may be, for example, a convolutional neural network model.However, the deep neural network model is not limited thereto, and can be implemented with any known deep neural network model, such as a recurrent neural network, a restricted Boltzmann machine, a deep belief network, a bidirectional recurrent deep neural network, or a deep Q-network.

[0084] The processor (130) can extract information about at least one of the type, category, color, theme, and atmosphere of a real space from a spatial image acquired by the first camera (112) by executing commands or program codes of the spatial information extraction module (141). In one or more embodiments of the present disclosure, the processor (130) can obtain spatial information from the spatial image by inputting the spatial image into a deep neural network model and performing inferencing using the deep neural network model.

[0085] The gesture information extraction module (142) is configured with commands or program codes related to functions and / or operations for recognizing a user's gesture input from a hand image and extracting object characteristic information from the gesture input. The processor (130) can recognize an area pointed by the user's hand and a trajectory of the area from the hand image acquired through the second camera (114) by executing the commands or program codes of the gesture information extraction module (142), thereby recognizing the user's gesture input. However, the present invention is not limited thereto, and the electronic device (100) may further include a touch screen. The touch screen receives a user's touch input, and the processor (130) may recognize the user's gesture input from the touch input. The processor (130) may obtain object characteristic information of at least one of a position, a size, and a shape of an object for creating an object according to the user's intention from the recognized gesture input.

[0086] The voice information extraction module (143) is configured with commands or program codes related to functions and / or operations for recognizing voice input and extracting object characteristic information from the voice input. In one or more embodiments of the present disclosure, the voice information extraction module (143) may include an Automatic Speech Recognition (ASR) model and a Natural Language Understanding (NLU) model. The processor (130) may convert a voice input, i.e., sound, acquired through the microphone (120) into an acoustic signal and remove noise (e.g., non-speech components) from the acoustic signal to obtain a voice signal. The processor (130) may obtain object characteristic information from the acquired voice signal by executing commands or program codes of the voice information extraction module (143). In one or more embodiments of the present disclosure, the processor (130) may perform ASR to convert a voice signal into text and analyze the text using a Natural Language Understanding (NLU) model to recognize a user's intent and a named entity. The processor (130) can obtain object characteristic information including at least one of the type, shape, color, and theme of the object for creating the object according to the user's intention and object name.

[0087] The guide information extraction module (144) is configured with commands or program codes related to the function and / or operation of extracting object characteristic information from a two-dimensional (2D) guide image. In the present disclosure, a 'two-dimensional guide image' means an image that provides guide information regarding the type, shape, or theme of an object for the electronic device (100) to create a three-dimensional virtual object. In one or more embodiments of the present disclosure, the electronic device (100) further includes a communication interface and can receive a two-dimensional guide image from an external server or storage device (e.g., a web-based database) through the communication interface. However, the present disclosure is not limited thereto, and the electronic device (100) can obtain a two-dimensional guide image (50) input by a user input.

[0088] In one or more embodiments of the present disclosure, the guide information extraction module (144) may include an object recognition model composed of an artificial intelligence model. The object recognition model may be a deep neural network model trained to recognize objects through a supervised learning method that applies bounding box images that can be recognized as objects from a plurality of training images as input data, and applies label values ​​regarding the type, shape, color, and theme of the object within the bounding box as ground truth. The deep neural network model may be implemented as an object recognition model such as, but not limited to, a convolutional neural network model, a region-based convolutional neural network model (R-CNN), YOLO v4, CenterNet, or MobileNet. The processor (130) can extract object characteristic information including at least one of the type, shape, color, and theme of the object from a two-dimensional guide image by executing commands or program codes of the guide information extraction module (144).

[0089] In FIGS. 3 and 4, the guide information extraction module (144) is shown as being included in the memory (140), but the guide information extraction module (144) may be omitted depending on the embodiment.

[0090] The information mixing module (145) is configured with commands or program codes related to functions and / or operations for obtaining object creation information by mixing spatial information and object characteristic information. Referring to FIG. 4, the information mixing module (145) can receive spatial information including the type, theme, atmosphere, etc. of a space output by the spatial information extraction module (141). In addition, the information mixing module (145) can receive object characteristic information regarding the shape, location, size, etc. of an object from the gesture information extraction module (142), receive object characteristic information regarding the type, shape, theme, etc. of an object from the voice information extraction module (143), and receive object characteristic information regarding the type, shape, theme, etc. of an object from the guide information extraction module (144). The processor (130) can obtain object creation information for creating a virtual object by mixing spatial information received from the spatial information extraction module (141) and object characteristic information received from the gesture information extraction module (142), voice information extraction module (143), and guide information extraction module (144) by executing commands or program codes of the information mixing module (145).

[0091] In one or more embodiments of the present disclosure, the processor (130) may embed spatial information and object characteristic information, respectively, to convert them into feature data, and then mix the converted feature data to obtain feature data representing object creation information. A specific embodiment of how the processor (130) obtains feature data representing object creation information will be described in detail with reference to FIGS. 5 and 6 .

[0092] In one or more embodiments of the present disclosure, if a vector value of embedded feature data representing spatial information or object characteristic information is missing, the processor (130) may assign a randomly generated value to the missing vector value. A specific embodiment in which the processor (130) assigns a random value to the missing vector value will be described in detail in FIG. 7.

[0093] The generative artificial intelligence model (146) is an artificial intelligence model trained to generate a three-dimensional virtual object based on spatial information and object characteristic information. In one or more embodiments of the present disclosure, the generative artificial intelligence model (146) may be implemented as a Generative Adversarial Network (GAN) or a multi-modality-based diffusion model. The multi-modality-based diffusion model may be implemented as, for example, Stable Diffusion or SDFusion. However, the invention is not limited thereto, and the generative artificial intelligence model (146) may include any known artificial intelligence model trained to generate a three-dimensional virtual object based on spatial information and object characteristic information. Referring also to FIG. 4, the processor (130) inputs object generation information acquired by the information mixing module (145) into the generative artificial intelligence model (146), and performs inferencing using the generative artificial intelligence model (146) to generate a three-dimensional virtual object based on the object generation information. In one or more embodiments of the present disclosure, the processor (130) may generate a three-dimensional virtual object through inference by inputting a vector representing an object generation model into a generative artificial intelligence model (146). The processor (130) may display the three-dimensional virtual object generated by the generative artificial intelligence model (146) through a display unit (150).

[0094] Referring to FIG. 3, the object modification module (147) is configured with commands or program codes related to a function and / or operation of modifying object creation information based on a user input. In one or more embodiments of the present disclosure, the electronic device (100) may receive a gesture input for modifying object creation information. However, the present disclosure is not limited thereto, and when the electronic device (100) is implemented as a mobile device including a touch screen, for example, a smart phone or a tablet PC, the electronic device (100) may also receive a user's touch input for modifying object creation information. In one or more embodiments of the present disclosure, the electronic device (100) receives a user input for adjusting a weight value assigned to each of a multi-modality input, that is, spatial information acquired from a spatial image, object characteristic information acquired from a gesture input, object characteristic information acquired from a voice input, and object characteristic information acquired from a two-dimensional guide image, and the processor (130) may modify the object creation information based on the user input. In one or more embodiments of the present disclosure, the electronic device (100) receives a gesture input or touch input from a user that changes at least one of the shape, size, position, color, type, and theme of an object, and the processor (130) can modify object creation information based on the received gesture input or touch input. A specific embodiment in which the processor (130) modifies object creation information based on user input will be described in detail with reference to FIGS. 8 to 11.

[0095] The display unit (150) is configured to display a three-dimensional virtual object under the control of the processor (130). If the electronic device (100) is an augmented reality device such as augmented reality glasses, the display unit (150) may be configured with a lens optical system and may include a waveguide and an optical engine. The optical engine may be configured with a projector that generates light of a three-dimensional virtual object composed of a virtual image and projects the light onto a waveguide. The optical engine may include, for example, an image panel, an illumination optical system, a projection optical system, etc. In one or more embodiments of the present disclosure, the optical engine may be disposed on the frame or temples of the augmented reality glasses. In one or more embodiments of the present disclosure, the optical engine may display a virtual object by projecting the three-dimensional virtual object onto a waveguide under the control of the processor (130).

[0096] However, the present invention is not limited thereto, and the display unit (150) may be configured as at least one of, for example, a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display, a 3D display, and an electrophoretic display. In this case, the display unit (150) may be configured as a touch screen including a touch pad.

[0097] FIG. 5 is a flowchart illustrating a method by which an electronic device (100) according to one or more embodiments of the present disclosure obtains feature data representing object creation information.

[0098] Steps S510 to S550 of FIG. 5 are steps that specify the operation of step S250 illustrated in FIG. 2. After the operation of step S550 of FIG. 5 is performed, step S260 illustrated in FIG. 2 may be performed.

[0099] FIG. 6 is a diagram illustrating an operation of an electronic device (100) according to one or more embodiments of the present disclosure to obtain feature data representing object creation information by embedding multi-modality information.

[0100] Hereinafter, with reference to FIGS. 5 and 6, an operation of an electronic device (100) acquiring feature data representing object creation information by embedding multi-modality information will be described.

[0101] In step S510 of FIG. 5, the electronic device (100) converts spatial information into first feature data by vector embedding it. Here, 'feature data' means data including an n-dimensional vector of one or more dimensions, a matrix, or a feature vector. Referring also to FIG. 6, the electronic device (100) may obtain spatial information about a real space from a spatial image obtained through a camera (110, see FIGS. 1 and 3), and convert the obtained spatial information into first feature data (611) by vector embedding it. The processor (130, see FIG. 3) of the electronic device (100) may execute commands or program codes of the spatial information extraction module (141) to extract spatial information including at least one of a type and a theme of a real space from the spatial image, and may vector embed the extracted spatial information. The vector embedding model is a model that quantifies text-type information into vector values, and can be implemented using known embedding models such as one-hot encoding or label encoding.

[0102] However, it is not limited thereto, and the processor (130) can convert spatial information into an n-dimensional vector of one or more dimensions, a matrix, or a tensor such as a feature vector. The functions and / or operations of the vector embedding model included in the gesture information extraction module (142), the voice information extraction module (143), and the guide information extraction module (144) are the same as the vector embedding model included in the spatial information extraction module (141), and therefore, redundant descriptions are omitted below. In the embodiment illustrated in FIG. 6, the processor (130) can obtain first feature data (610) by executing the vector embedding module to embed the type of space and the theme of space included in the spatial information, respectively. The first feature data (610) can include first spatial data (611) in which information about the type of space is converted into a vector value, and second spatial data (612) in which information about the theme of space is converted into a vector value.

[0103] In step S520 of FIG. 5, the electronic device (100) converts object characteristic information obtained from a gesture input into second characteristic data by vector embedding. Referring also to FIG. 6, the processor (130) of the electronic device (100) may execute commands or program codes of the gesture information extraction module (142) to recognize a user's gesture input from a hand image, extract object characteristic information including at least one of a position, a size, and a shape of an object according to the user's intention from the gesture input, and vector embed the extracted object characteristic information. In the embodiment illustrated in FIG. 6, the processor (130) may execute a vector embedding module to embed information on the position, the size, and the shape of an object included in the object characteristic information, thereby obtaining second characteristic data (620). The second feature data (620) may include first object feature data (621) in which information about the location of an object is converted into a vector value, second object feature data (622) in which information about the size of an object is converted into a vector value, and third object feature information data (623) in which information about the shape of an object is converted into a vector value.

[0104] In step S530 of FIG. 5, the electronic device (100) converts object characteristic information acquired from a voice input into third characteristic data by vector embedding. Referring also to FIG. 6, the processor (130) of the electronic device (100) may execute commands or program codes of the voice information extraction module (143) to extract object characteristic information including at least one of the type, shape, and theme of an object according to the user's intention from a voice signal acquired by a microphone (120, see FIGS. 1 and 3), and may vector embed the extracted object characteristic information. In the embodiment illustrated in FIG. 6, the processor (130) may execute a vector embedding module to embed information on the type, shape, and theme of an object included in the object characteristic information, thereby acquiring third characteristic data (630). The third feature data (630) may include first object feature data (631) in which information about the type of object is converted into a vector value, second object feature data (632) in which information about the shape of the object is converted into a vector value, and third object feature data (633) in which information about the theme of the object is converted into a vector value.

[0105] In step S540 of FIG. 5, the electronic device (100) converts the object characteristic information obtained from the two-dimensional guide image into fourth characteristic data by vector embedding. Referring also to FIG. 6, the processor (130) of the electronic device (100) may execute commands or program codes of the guide information extraction module (144) to extract object characteristic information including at least one of the type, shape, and theme of the object from the two-dimensional guide image, and may vector embed the extracted object characteristic information. In the embodiment illustrated in FIG. 6, the processor (130) may execute the vector embedding module to embed information on the type, shape, and theme of the object included in the object characteristic information, thereby obtaining fourth characteristic data (640). The fourth feature data (640) may include first object feature data (641) in which information about the type of object is converted into a vector value, second object feature data (642) in which information about the shape of the object is converted into a vector value, and third object feature data (643) in which information about the theme of the object is converted into a vector value.

[0106] Steps S510 to S540 illustrated in FIG. 5 may be performed at the same time, but are not limited thereto. In one or more embodiments of the present disclosure, steps S510 to S540 may be performed sequentially in any order with a time difference.

[0107] The first feature data (611) illustrated in FIG. 6 is illustrated as an 8-dimensional vector, and the second feature data (620), the third feature data (630), and the fourth feature data (640) are illustrated as 12-dimensional vectors. However, this is for convenience of explanation, and the embodiments of the present disclosure are not limited to those illustrated in FIG. 6. In one or more embodiments of the present disclosure, the first feature data (611) to the fourth feature data (614) may each be implemented as an n-dimensional vector having 1 or more dimensions.

[0108] In step S550 of FIG. 5, the electronic device (100) obtains a vector representing object creation information based on the first to fourth feature data. Referring also to FIG. 6, the processor (130) of the electronic device (100) inputs the first feature data (610), the second feature data (620), the third feature data (630), and the fourth feature data (640) into the information mixing module (145), and executes commands or program codes of the information mixing module (145) to concatenate the first feature data (610), the second feature data (620), the third feature data (630), and the fourth feature data (640), thereby obtaining mixed feature data (650). In one or more embodiments of the present disclosure, the processor (130) may obtain mixed feature data (650) by mixing the first feature data (610) to the fourth feature data (640) in a multi-layer perceptron manner. However, the present invention is not limited thereto, and the processor (130) may obtain mixed feature data (650) representing object generation information by inputting the first feature data (610) to the fourth feature data (640) into a convolutional neural network (CNN) or a recurrent neural network (RNN).

[0109] In the embodiment illustrated in FIG. 6, the mixed feature data (650) may be composed of an n-dimensional vector, matrix, or feature vector. In one or more embodiments of the present disclosure, the mixed feature data (650) may be a vector in which the vector values ​​of the first feature data (610), the second feature data (620), the third feature data (630), and the fourth feature data (640) are lined up in a row according to the result of the multi-layer perceptron.

[0110] An electronic device (100) inputs mixed feature data (650) including data values ​​(e.g., vector values) regarding object creation information into a generative artificial intelligence model (146, see FIGS. 1 and 3), and performs inferencing using the generative artificial intelligence model (146) to create a three-dimensional virtual object based on the object creation information.

[0111] FIG. 7 is a diagram illustrating an operation of an electronic device (100) according to one or more embodiments of the present disclosure to assign a missing vector value to a vector (700) representing object creation information.

[0112] Referring to FIG. 7, a vector (700) may include vector values ​​converted by vector-embedding object creation information including spatial information and object characteristic information. In one or more embodiments of the present disclosure, the vector (700) may be a vector obtained by connecting first feature data (610, see FIG. 6) obtained by vector-embedding spatial information extracted from a spatial image, second feature data (620, see FIG. 6) obtained by vector-embedding object characteristic information extracted from a gesture input, third feature data (630, see FIG. 6) obtained by vector-embedding object characteristic information extracted from a voice input, and fourth feature data (640, see FIG. 6) obtained by vector-embedding object characteristic information extracted from a two-dimensional guide image through a multi-layer perceptron.

[0113] The vector (700) includes vector values, but may have some missing vector values ​​(701, 702). In the embodiment illustrated in FIG. 7, the first vector (701) and the second vector (702) may not have vector values. In one or more embodiments of the present disclosure, the processor (130, see FIG. 3) of the electronic device (100) may randomly generate random values ​​(711, 712) and assign the generated random values ​​(711, 712) to the first vector (701) and the second vector (702) having missing vector values. The processor (130) may, for example, assign the randomly generated first random value (711) to the first vector (701) and the second random value (712) to the second vector (702). Through this, the processor (130) can obtain a reconstructed vector (710).

[0114] In the embodiment illustrated in FIG. 7, the electronic device (100) may fill in the missing vectors (701, 702) by assigning randomly generated random values ​​(711, 712) to vectors (701, 702) with missing vector values ​​in order to maintain the dimension of the vector (700) input to the generative artificial intelligence model (146, see FIGS. 1 and 3). Since the reconstructed vector (710) includes random random values ​​(711, 712), when the reconstructed vector (710) is input to the generative artificial intelligence model (146), a three-dimensional virtual object having a shape, size, location, type, theme, or atmosphere that does not match the user's intention may be generated. In this case, the user may modify the object generation information.

[0115] FIG. 8 is a flowchart illustrating a method for an electronic device (100) to create a three-dimensional virtual object by modifying object creation information based on user input according to one or more embodiments of the present disclosure.

[0116] Steps S810 to S830 of FIG. 8 may be performed after the operation of step S230 illustrated in FIG. 2 is performed. After step S830 of FIG. 8 is performed, the operation of S250 illustrated in FIG. 2 may be performed.

[0117] In step S810, the electronic device (100) receives a user input for modifying object creation information. In one or more embodiments of the present disclosure, the electronic device (100) may receive a user input for adjusting a weight applied to each piece of multi-modality information. For example, the electronic device (100) may receive a user input for adjusting a value of a weight assigned to each piece of spatial information acquired from a spatial image, object characteristic information acquired from a gesture input, object characteristic information acquired from a voice input, and object characteristic information acquired from a two-dimensional guide image.

[0118] In one or more embodiments of the present disclosure, the electronic device (100) may receive a user input that changes at least one of the shape, size, position, color, type, and theme of an object. For example, the electronic device (100) may receive a gesture input from a user that modifies the shape, size, position, etc. of an object.

[0119] In step S820, the electronic device (100) modifies object creation information based on the received user input. In one or more embodiments of the present disclosure, the electronic device (100) may change the weights assigned to each of spatial information obtained from a spatial image, object characteristic information obtained from a gesture input, object characteristic information obtained from a voice input, and object characteristic information obtained from a two-dimensional guide image based on user input that adjusts the weights applied to each of the multi-modality information. A specific embodiment in which the electronic device (100) modifies the weights applied to each of the multi-modality information based on the user input will be described in detail with reference to FIG. 10.

[0120] In one or more embodiments of the present disclosure, the electronic device (100) may modify at least one of the shape, size, location, color, type, and theme of an object included in object creation information based on user input. A specific embodiment in which the electronic device (100) modifies object creation information based on user input will be described in detail in FIG. 11.

[0121] In step S830, the electronic device (100) inputs the modified object creation information into a generative artificial intelligence model and performs inferencing using the generative artificial intelligence model to create a three-dimensional virtual object. Since the object creation information has been modified, the shape, size, location, color, type, or theme of the three-dimensional virtual object acquired by the generative artificial intelligence model may be changed.

[0122] FIG. 9 is a diagram illustrating components of an electronic device (100) that creates and displays a three-dimensional virtual object by modifying object creation information according to one or more embodiments of the present disclosure.

[0123] Referring to FIG. 9, multi-modality information is input into the information mixing module (145). In one or more embodiments of the present disclosure, the 'multi-modality information' may include spatial information obtained from a spatial image, object characteristic information obtained from a gesture input, object characteristic information obtained from a voice input, and object characteristic information obtained from a two-dimensional guide image.

[0124] The object modification module (147) receives multi-modality information and a user's modification input. In one or more embodiments of the present disclosure, the user's modification input may be a user input for adjusting a weight applied to each of the multi-modality information, or a user input for changing at least one of the shape, size, position, color, type, and theme of the object. The processor (130, see FIG. 3) of the electronic device (100) may execute commands or program codes of the object modification module (147) to modify object creation information according to the user input. The object modification module (147) may output object modification information.

[0125] The processor (130) can modify object creation information by multi-modality information using object modification information by executing commands or program codes of the information mixing module (145), and obtain modified object creation information. In one or more embodiments of the present disclosure, the processor (130) can modify object creation information by concatenating vectors obtained by vector embedding each of the multi-modality information through a multi-layer perceptron, and modifying vector values ​​of the vectors obtained through this.

[0126] The processor (130) can input a vector representing modified object creation information obtained as a result of the modification into a generative artificial intelligence model (146) to create a three-dimensional virtual object for the modified object creation information.

[0127] FIG. 10 is a diagram illustrating an operation of an electronic device (100) according to one or more embodiments of the present disclosure to modify object creation information by receiving a user input for adjusting a weight of multi-modality information (1010, 1020, 1030, 1040).

[0128] Referring to FIG. 10, the electronic device (100) receives a user input for adjusting weights for each of spatial information and object characteristic information acquired through multi-modality information (Operation ①). The electronic device (100) may receive a user input for changing weights applied to each of the multi-modality information, i.e., object characteristic information (1010) acquired from a gesture input, spatial information (1020) acquired from a spatial image, object characteristic information (1030) acquired from a voice input, and object characteristic information (1040) acquired from a two-dimensional guide image. In the embodiment illustrated in FIG. 10, the electronic device (100) is implemented as a mobile device including a touch screen, for example, a smart phone or a tablet PC, and may receive a user's touch input for changing weights. However, the present invention is not limited thereto, and when the electronic device (100) is implemented as an augmented reality device such as augmented reality glasses or a head mounted display, the electronic device (100) may receive a user's hand pointing input for changing the weight. In one or more embodiments of the present disclosure, the electronic device (100) may include a gaze tracking sensor and may receive a user input for modifying a weight applied to each piece of multi-modality information based on a gaze point according to a gaze direction of the user acquired using the gaze tracking sensor.

[0129] The electronic device (100) adjusts the weights applied to each of the multi-modality information (operation ②). In one or more embodiments of the present disclosure, the processor (130, see FIG. 3) of the electronic device (100) may change the weights applied to each of object characteristic information (1010) obtained from a gesture input, spatial information (1020) obtained from a spatial image, object characteristic information (1030) obtained from a voice input, and object characteristic information (1040) obtained from a two-dimensional guide image based on a user input. In the embodiment illustrated in FIG. 10, the object characteristic information (1010) obtained from the gesture input includes information about the size, position, and shape of the object, is composed of vector values ​​converted through vector embedding, and a first weight (α) may be applied by the user input. Spatial information (1020) acquired from a spatial image includes information on the type and theme of a real space, is composed of vector values ​​converted through vector embedding, and a second weight (β) can be applied by a user input. Object characteristic information (1030) acquired from a voice input includes information on the shape, type, and theme of an object, is composed of vector values ​​converted through vector embedding, and a third weight (γ) can be applied by a user input. Object characteristic information (1040) acquired from a two-dimensional guide image includes information on the shape, type, and theme of an object, is composed of vector values ​​converted through vector embedding, and a fourth weight (δ) can be applied by a user input.

[0130] The electronic device (100) modifies object creation information by adjusting the weights applied to each piece of multi-modality information (Operation ③). In the embodiment illustrated in FIG. 10, the processor (130) of the electronic device (100) may modify object creation information by applying a first weight (α) to object characteristic information (1010) acquired from a gesture input, a second weight (β) to spatial information (1020) acquired from a spatial image, a third weight (γ) to object characteristic information (1030) acquired from a voice input, and a fourth weight (δ) to object characteristic information (1040) acquired from a two-dimensional guide image.

[0131] The electronic device (100) obtains modified object creation information (1050) by mixing the weight-adjusted spatial information and object characteristic information (Operation ④). In one or more embodiments of the present disclosure, the processor (130) of the electronic device (100) may obtain modified object creation information (1050) by concatenating vector values ​​of object characteristic information (1010) to which a first weight (α) is applied, spatial information (1020) to which a second weight (β) is applied, object characteristic information (1030) to which a third weight (γ) is applied, and object characteristic information to which a fourth weight (δ) is applied, through a multi-layer perceptron. However, it is not limited thereto, and the processor (130) can obtain modified object creation information (1050) by inputting vector values ​​of object characteristic information (1010) to which a first weight (α) is applied, spatial information (1020) to which a second weight (β) is applied, object characteristic information (1030) to which a third weight (γ) is applied, and object characteristic information to which a fourth weight (δ) is applied into a convolutional neural network (CNN) or a recurrent neural network (RNN).

[0132] The processor (130) of the electronic device (100) inputs modified object creation information (1050) into a generative artificial intelligence model (146, see FIGS. 1 and 3), and can create a modified three-dimensional virtual object according to the user's intention through inference using the generative artificial intelligence model (146).

[0133] FIG. 11 is a diagram illustrating an operation of an electronic device (100) according to one or more embodiments of the present disclosure to receive a user input for modifying the shape, size, position, etc. of an object and modify object creation information.

[0134] Referring to FIG. 11, the electronic device (100) receives a user input for changing at least one of the shape, size, position, color, type, and theme of an object (Operation ①). In one or more embodiments of the present disclosure, the electronic device (100) may display a three-dimensional virtual object (1100) and receive a user's gesture input (1110) for changing at least one of the shape, size, position, color, type, and theme of the displayed three-dimensional virtual object. In the embodiment illustrated in FIG. 11, the electronic device (100) may receive a user's gesture input (1110) for drawing a tilted '∩' shape. However, the present disclosure is not limited thereto, and in one or more embodiments of the present disclosure, the electronic device (100) is implemented as a mobile device including a touchscreen, for example, a smart phone or a tablet PC, in which case the electronic device (100) can receive a user's touch input for changing at least one of the shape, size, position, color, type, and theme of a three-dimensional virtual object (1100) displayed on the touchscreen. In another embodiment of the present disclosure, the electronic device (100) may input a space image acquired by photographing a space other than the space applied to generate the three-dimensional virtual object (1100).

[0135] The electronic device (100) modifies at least one of spatial information and object characteristic information based on a user input (Operation ②). In the embodiment illustrated in FIG. 11, the object creation information (1120) may include object characteristic information (1121) obtained from a gesture input, spatial information (1122) obtained from a spatial image, object characteristic information (1123) obtained from a voice input, and object characteristic information (1124) obtained from a two-dimensional guide image. In one or more embodiments of the present disclosure, the processor (130, see FIG. 3) of the electronic device (100) may execute instructions or program codes of the gesture information extraction module (142) to vector embed the gesture input (1110) received from the user to obtain object modification information (1130). The processor (130) may replace the object characteristic information (1121) included in the object creation information (1120) using the obtained object modification information (1130).

[0136] The electronic device (100) obtains modified object creation information (1140) by mixing modified spatial information and object characteristic information (Operation ③). In one or more embodiments of the present disclosure, the processor (130) of the electronic device (100) may obtain modified object creation information (1140) by connecting object modification information (1130), spatial information (1122), and object characteristic information (1123, 1124) through a multi-layer perceptron. The modified object creation information (1140) may be a vector including all of the vector values ​​of the object modification information (1130), the vector values ​​of the spatial information (1122), and the vector values ​​of the object characteristic information (1123, 1124). However, it is not limited thereto, and the processor (130) can obtain modified object creation information (1140) by inputting vector values ​​representing object modification information (1130), spatial information (1122), and object characteristic information (1123, 1124) into a convolutional neural network (CNN) or a recurrent neural network (RNN).

[0137] The electronic device (100) generates a three-dimensional virtual object (1150) using the modified object creation information (1140) (Operation ④). In one or more embodiments of the present disclosure, the processor (130) of the electronic device (100) inputs the modified object creation information (1140) into a generative artificial intelligence model (146) and performs inferencing using the generative artificial intelligence model (146) to generate the three-dimensional virtual object (1150).

[0138] In the embodiment illustrated in FIG. 11, the previously generated three-dimensional virtual object (1100) is a desk including a circular top and a circular bottom, and having one leg connecting the top and bottom, and can be modified into a three-dimensional virtual object (1150) that is a desk in the shape of a '∩' having a top and two legs based on a gesture input (1110) received from a user.

[0139] In the embodiments illustrated in FIGS. 8 to 11, the electronic device (100) may receive a user input for adjusting a weight applied to each piece of multi-modality information, or may receive a user input for changing at least one of the shape, size, position, color, type, and theme of an object, and may modify a three-dimensional virtual object based on the received user input. The electronic device (100) according to one or more embodiments of the present disclosure provides a technical effect of enabling a user to interact with an augmented reality service as well as allowing a plurality of multi-users to share an experience of the augmented reality service by allowing the user to further modify a primarily generated three-dimensional virtual object.

[0140] FIG. 12a is a diagram illustrating an operation of an electronic device (100) according to one or more embodiments of the present disclosure to generate and display a three-dimensional virtual object (1250a) based on multi-modality information (1210, 1220, 1230, 1240).

[0141] In the embodiment illustrated in FIG. 12a, the electronic device (100) may obtain spatial information such as 'brown tone of a cafe' from a spatial image (1210), obtain object characteristic information from a voice input (1220) such as "Make me a desk that matches the cafe," obtain object characteristic information about the shape, size, location, etc. of an object from a gesture input (1230) of drawing a desk, and obtain object characteristic information about the shape, size, type, color, theme, etc. of an object from a two-dimensional guide image (1240) including a drawer and a white-colored desk. The processor (130, see FIG. 3) of the electronic device (100) may embed the obtained information into vector values ​​and input the embedded vectors into a generative artificial intelligence model (146, see FIGS. 1 and 3) to generate a three-dimensional virtual object (1250a). In the embodiment illustrated in FIG. 12a, the three-dimensional virtual object (1250a) may be a three-dimensional graphic of a desk having a shape (rounded wooden corners) and color (e.g., brown) that matches the tone, theme, or atmosphere of a real space (e.g., a cafe) (1200a).

[0142] The electronic device (100) can display the generated three-dimensional virtual object (1250a) by overlaying it on the real space (1200a). In FIG. 12A, the electronic device (100) implemented as a mobile device is illustrated as displaying the real space (1200a) and the three-dimensional virtual object (1250a) on the display unit, but the present disclosure is not limited thereto. In one or more embodiments of the present disclosure, the electronic device (100) is implemented as an augmented reality device such as augmented reality glasses, and can display the real space (1200a) transmitted through an optical lens and the three-dimensional virtual object (1250a) projected through a waveguide in the form of augmented reality or mixed reality.

[0143] FIG. 12b is a diagram illustrating an operation of an electronic device (100) according to one or more embodiments of the present disclosure to generate and display a three-dimensional virtual object (1250b) based on multi-modality information (1210, 1220, 1230, 1240).

[0144] The embodiment illustrated in FIG. 12b is identical to the embodiment illustrated in FIG. 12a except that the real space (1200b) is a gray-toned office rather than a cafe, and the electronic device (100) receives a voice input (1220) saying “Make me a desk that fits this office,” and therefore, a duplicate description is omitted.

[0145] Referring to FIG. 12b, the electronic device (100) can generate a three-dimensional virtual object (1250b) by performing inference by mixing multi-modality information including a spatial image (1210), a voice input (1220), a gesture input (1230), and a two-dimensional guide image (1240) and inputting the mixed information into a generative artificial intelligence model (146, see FIGS. 1 and 3). In the embodiment illustrated in FIG. 12b, the three-dimensional virtual object (1250b) can be a three-dimensional graphic of a desk having a shape (e.g., a shape of sharp-edged iron) and a color (e.g., gray) that matches the gray tone, theme, or atmosphere of a real space (e.g., an office) (1200b). The electronic device (100) can display the generated three-dimensional virtual object (1250b) by overlaying it on the real space (1200b).

[0146] FIG. 12c is a diagram illustrating an operation of an electronic device (100) according to one or more embodiments of the present disclosure to generate and display a three-dimensional virtual object (1250c) based on multi-modality information (1210, 1220, 1230, 1240).

[0147] The embodiment illustrated in FIG. 12c is identical to the embodiment illustrated in FIG. 12b except that the two-dimensional guide image (1240) is an image including a desk made of wood with a brown tone, and therefore, a duplicate description is omitted.

[0148] Referring to FIG. 12c, the electronic device (100) can generate a three-dimensional virtual object (1250c) by performing inference by mixing multi-modality information including a spatial image (1210), a voice input (1220), a gesture input (1230), and a two-dimensional guide image (1240) and inputting the mixed information into a generative artificial intelligence model (146, see FIGS. 1 and 3). In the embodiment illustrated in FIG. 12c, the three-dimensional virtual object (1250c) can be a three-dimensional graphic of a desk having a shape (e.g., a shape with sharp edges made of wood) and a color (e.g., brown) that matches the tone, theme, or atmosphere of a real space (e.g., an office) (1200c). The electronic device (100) can display the generated three-dimensional virtual object (1250c) by overlaying it on the real space (1200c).

[0149] FIG. 12d is a diagram illustrating an operation of an electronic device (100) according to one or more embodiments of the present disclosure to generate and display a three-dimensional virtual object (1250d) based on multi-modality information (1210, 1220, 1230, 1240).

[0150] The embodiment illustrated in FIG. 12d is identical to the embodiment illustrated in FIG. 12c except that the electronic device (100) receives a voice input (1220) saying “Make a bookshelf that fits this office”, and therefore, a duplicate description is omitted.

[0151] Referring to FIG. 12d, the electronic device (100) can generate a three-dimensional virtual object (1250d) by performing inference by mixing multi-modality information including a spatial image (1210), a voice input (1220), a gesture input (1230), and a two-dimensional guide image (1240) and inputting the mixed information into a generative artificial intelligence model (146, see FIGS. 1 and 3). In the embodiment illustrated in FIG. 12d, the three-dimensional virtual object (1250d) can be a three-dimensional graphic of a bookshelf having a shape (e.g., a shape with sharp edges made of wood) and a color (e.g., brown) that matches the tone, theme, or atmosphere of a real space (e.g., an office) (1200d). The electronic device (100) can display the generated three-dimensional virtual object (1250d) by overlaying it on the real space (1200d).

[0152] FIG. 12e is a diagram illustrating an operation of an electronic device (100) according to one or more embodiments of the present disclosure to generate and display a three-dimensional virtual object (1250e) based on multi-modality information (1210, 1220, 1230, 1240).

[0153] The embodiment illustrated in FIG. 12e is identical to the embodiment illustrated in FIG. 12d except for the user's gesture input (1230) of drawing a square top plate and four legs, so a duplicate description is omitted.

[0154] Referring to FIG. 12e, the electronic device (100) can generate a three-dimensional virtual object (1250e) by performing inference by mixing multi-modality information including a spatial image (1210), a voice input (1220), a gesture input (1230), and a two-dimensional guide image (1240) and inputting the mixed information into a generative artificial intelligence model (146, see FIGS. 1 and 3). In the embodiment illustrated in FIG. 12e, the three-dimensional virtual object (1250e) can be a three-dimensional graphic of a desk having a shape (e.g., a shape with a square top with sharp corners and four legs) and a color (e.g., brown) that matches the tone, theme, or atmosphere of a real space (e.g., an office) (1200e). The electronic device (100) can display the generated three-dimensional virtual object (1250e) by overlaying it on the real space (1200e).

[0155] In the embodiments illustrated in FIGS. 12a to 12e, the electronic device (100) can adjust weights applied to each of the multi-modality information (1210, 1220, 1230, 1240) or modify the three-dimensional virtual object (1250a, 1250b, 1250c, 1250d, 1250e) upon receiving user input that changes at least one of the shape, size, position, color, type, and theme of the object.

[0156] One aspect of the present disclosure provides an electronic device (100) that generates a virtual object based on multi-modality information. The electronic device (100) according to one or more embodiments of the present disclosure may include a camera (110) that acquires an image, a memory (140) that stores at least one instruction, and at least one processor (130) that executes the at least one instruction. By the at least one processor (130) executing the at least one instruction, the electronic device may acquire spatial information about a real space based on an image acquired through the camera (110). By the at least one processor (130) executing the at least one instruction, the electronic device may acquire a user input based on an image acquired through the camera (110). By the at least one processor (130) executing the at least one instruction, the electronic device may acquire object characteristic information from a user input. The electronic device can obtain object generation information for generating a virtual object based on the acquired spatial information and object characteristic information by executing the at least one processor (130) of the at least one command. The electronic device can input the acquired object generation information into a generative artificial intelligence model trained to generate a three-dimensional virtual object based on information about space and objects by executing the at least one processor (130) of the at least one command, thereby generating a virtual object based on the object generation information.

[0157] In one or more embodiments of the present disclosure, the camera (110) may include a first camera (112) that photographs a real space to obtain a spatial image of the real space. By having the at least one processor (130) execute the at least one command, the electronic device may obtain spatial information regarding at least one of a type, category, color, theme, and atmosphere of the real space from the spatial image obtained through the first camera (112).

[0158] In one or more embodiments of the present disclosure, the camera (110) may include a second camera (114) that captures a hand image by photographing a user's hand. When the at least one processor (130) executes the at least one command, the electronic device may recognize a user's gesture input from the hand image acquired through the second camera (114), and extract object characteristic information regarding at least one of a shape, a position, and a size of the object from the recognized gesture input.

[0159] In one or more embodiments of the present disclosure, the second camera (114) is configured as a depth camera including at least one of a Time of Flight (ToF) camera, a stereo vision camera, and a Light Detection and Ranging (LiDAR) sensor, and can capture a depth image by taking a picture of a user's hand. When the at least one processor (130) executes the at least one command, the electronic device can recognize a user's gesture input from the depth image acquired through the second camera (114).

[0160] In one or more embodiments of the present disclosure, the electronic device (100) may further include a touchscreen for receiving a user's touch input. By having the at least one processor (130) execute the at least one command, the electronic device may recognize a user's gesture input from a touch input received through the touchscreen, and extract object characteristic information regarding at least one of a shape, a position, and a size of an object from the recognized gesture input.

[0161] In one or more embodiments of the present disclosure, the electronic device (100) may further include a microphone (120) for receiving a voice input from a user. The electronic device may obtain a voice signal from a voice input received through the microphone (120) by having the at least one processor (130) execute the at least one command. The electronic device may convert the voice signal into text by having the at least one processor (130) execute the at least one command, and may obtain object characteristic information including at least one of a type, shape, color, and theme of the object by analyzing the text using a natural language understanding model.

[0162] In one or more embodiments of the present disclosure, the electronic device may obtain a two-dimensional (2D) guide image by executing the at least one command by the at least one processor (130), and extract object characteristic information including at least one of the type, shape, color, and theme of the object from the two-dimensional (2D) guide image.

[0163] In one or more embodiments of the present disclosure, the electronic device may convert spatial information into first feature data by vector embedding it when the at least one processor (130) executes the at least one command. The electronic device may convert object characteristic information obtained from a gesture input into second feature data by vector embedding it when the at least one processor (130) executes the at least one command. The electronic device may convert object characteristic information obtained from a voice input into third feature data by vector embedding it when the at least one processor (130) executes the at least one command. The electronic device may convert object characteristic information obtained from a two-dimensional guide image into fourth feature data by vector embedding it when the at least one processor (130) executes the at least one command. The electronic device may obtain feature data representing object creation information based on the first feature data to the fourth feature data by executing the at least one command.

[0164] In one or more embodiments of the present disclosure, when the at least one processor (130) executes the at least one command, the electronic device can assign a randomly generated random number value as the missing vector value when a vector value representing spatial information or object characteristic information among the first to fourth feature data is missing.

[0165] In one or more embodiments of the present disclosure, the electronic device can modify object creation information based on user input by causing the at least one processor (130) to execute the at least one command.

[0166] In one or more embodiments of the present disclosure, the electronic device can modify object creation information by adjusting weight values ​​assigned to each of spatial information obtained from an image, object characteristic information obtained from a gesture input, object characteristic information obtained from a voice input, and object characteristic information obtained from a two-dimensional guide image based on a user input, by having the at least one processor (130) execute the at least one command.

[0167] In one or more embodiments of the present disclosure, the electronic device can modify object creation information by changing at least one of the shape, size, position, color, type, and theme of the object based on a user input by having the at least one processor (130) execute the at least one command.

[0168] Another aspect of the present disclosure provides a method for an electronic device (100) to generate a virtual object based on multi-modality information. In one or more embodiments of the present disclosure, a method for operating an electronic device (100) may include a step (S210) of acquiring an image through a camera (110). The method for operating the electronic device (100) may include a step (S220) of acquiring a user input based on the image acquired through the camera (110). The method for operating the electronic device (100) may include a step (S230) of acquiring spatial information about a real space from the acquired image. The method for operating the electronic device (100) may include a step (S240) of acquiring object characteristic information from the acquired user input. The method for operating the electronic device (100) may include a step (S250) of acquiring object generation information for generating a virtual object based on the acquired spatial information and object characteristic information. The operating method of the electronic device (100) may include a step (S260) of inputting acquired object creation information into a generative artificial intelligence model trained to create a three-dimensional virtual object based on information about space and objects, thereby creating a virtual object for the object creation information.

[0169] In one or more embodiments of the present disclosure, the step of obtaining spatial information (S230) may include a step of obtaining a spatial image using a first camera (112) configured to photograph a real space and obtain a spatial image of the real space, and a step of obtaining spatial information about at least one of a type, a category, a color, a theme, and an atmosphere of the real space from the obtained spatial image.

[0170] In one or more embodiments of the present disclosure, the step (S240) of obtaining the object characteristic information may include the step of obtaining a hand image through a second camera (114) configured to photograph a user's hand to obtain a hand image, and the step of recognizing a user's gesture input from the obtained hand image. The step (S240) of obtaining the object characteristic information may include the step of extracting object characteristic information regarding at least one of a shape, a position, and a size of the object from the recognized gesture input.

[0171] In one or more embodiments of the present disclosure, the second camera (114) may be configured as a depth camera including at least one of a Time of Flight (ToF) camera, a stereo vision camera, and a Light Detection and Ranging (LiDAR) sensor. In the step of acquiring the hand image, the electronic device (100) may capture a photograph of the user's hand to acquire a depth image. In the step of recognizing the user's gesture input, the electronic device (100) may recognize the user's gesture input from the depth image acquired through the second camera (114).

[0172] In one or more embodiments of the present disclosure, the step (S240) of obtaining object characteristic information may include the step of receiving a user's touch input via a touch screen, and the step of recognizing a user's gesture input from the touch input received via the touch screen. The step (S240) of obtaining object characteristic information may include the step of extracting object characteristic information regarding at least one of a shape, a position, and a size of the object from the recognized gesture input.

[0173] In one or more embodiments of the present disclosure, the step (S240) of obtaining the object characteristic information may include: receiving a user's voice input via a microphone (120); obtaining a voice signal from the received voice input; and converting the voice signal into text. The step (S240) of obtaining the object characteristic information may include a step of obtaining object characteristic information including at least one of a type, shape, color, and theme of the object by analyzing text using a natural language understanding model.

[0174] In one or more embodiments of the present disclosure, the method of operating the electronic device (100) may further include a step of acquiring a two-dimensional (2D) guide image. The step (S240) of acquiring object characteristic information may include a step of extracting object characteristic information including at least one of the type, shape, color, and theme of the object from the two-dimensional guide image.

[0175] In one or more embodiments of the present disclosure, the step of obtaining object creation information (S240) may include a step of converting spatial information into first feature data by vector embedding. The step of obtaining object creation information (S240) may include a step of converting object characteristic information obtained from a gesture input into second feature data by vector embedding. The step of obtaining object creation information (S240) may include a step of converting object characteristic information obtained from a voice input into third feature data by vector embedding. The step of obtaining object creation information (S240) may include a step of converting object characteristic information obtained from a two-dimensional guide image into fourth feature data by vector embedding. The step of obtaining object creation information (S240) may include a step of obtaining feature data representing object creation information based on the first feature data to the fourth feature data.

[0176] In one or more embodiments of the present disclosure, the method of operating the electronic device (100) may further include a step (S810) of receiving a user input for modifying object creation information; and a step (S820) of modifying the object creation information based on the received user input.

[0177] In one or more embodiments of the present disclosure, in the step (S820) of modifying the object creation information, the electronic device (100) may modify the object creation information by adjusting a weight value assigned to each of spatial information obtained from an image, object characteristic information obtained from a gesture input, object characteristic information obtained from a voice input, and object characteristic information obtained from a two-dimensional guide image based on a user input.

[0178] In one or more embodiments of the present disclosure, in the step of modifying the object creation information (S820), the electronic device (100) may modify the object creation information by changing at least one of the shape, size, position, color, type, and theme of the object based on a user input.

[0179] Another aspect of the present disclosure provides a computer program product including a computer-readable storage medium. The storage medium may include instructions readable by the electronic device (100) so that the electronic device (100) performs the following operations: acquiring an image through a camera (110); acquiring a user input based on the image acquired through the camera (110); acquiring spatial information about a real space from the acquired image; acquiring object characteristic information from the acquired user input; acquiring object generation information for generating a virtual object based on the acquired spatial information and object characteristic information; and inputting the acquired object generation information into a generative artificial intelligence model trained to generate a three-dimensional virtual object based on information about space and objects, thereby generating a virtual object based on the object generation information.

[0180] The program executed by the electronic device (100) described in the present disclosure may be implemented as hardware components, software components, and / or a combination of hardware components and software components. The program may be executed by any system capable of executing computer-readable instructions.

[0181] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to do a desired thing or may independently or collectively command a processing device to do a desired thing.

[0182] Software may be implemented as a computer program containing instructions stored on a computer-readable storage medium. Examples of computer-readable storage media include magnetic storage media (e.g., read-only memory (ROM), random-access memory (RAM), floppy disks, hard disks, etc.) and optical readable media (e.g., CD-ROMs, DVDs (Digital Versatile Discs)). The computer-readable storage media may be distributed across network-connected computer systems, so that computer-readable code may be stored and executed in a distributed manner. The media may be readable by a computer, stored in a memory, and executed by a processor.

[0183] A computer-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" simply means that the storage medium does not contain signals and is tangible, but does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0184] Additionally, programs according to the embodiments disclosed herein may be provided as part of a computer program product. The computer program product may be traded as a commodity between sellers and buyers.

[0185] A computer program product may include a software program, a computer-readable storage medium having the software program stored thereon. For example, the computer program product may be available from a manufacturer of an electronic device (100) or an electronic market (e.g., Samsung Galaxy Store). TM ) may include a product in the form of a software program (e.g., a downloadable application) that is distributed electronically. For electronic distribution, at least a portion of the software program may be stored in a storage medium or temporarily created. In this case, the storage medium may be a storage medium of a server of a manufacturer of the electronic device (100), a server of an electronic market, or a relay server that temporarily stores the software program.

[0186] In a system comprising an electronic device (100) and / or a server, the computer program product may include a storage medium of the server or the storage medium of the electronic device (100). Alternatively, if there is a third device (e.g., a wearable device) that is communicatively connected to the electronic device (100), the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include a software program itself that is transmitted from the electronic device (100) to the third device, or from the third device to the electronic device.

[0187] In this case, either the electronic device (100) or the third device may execute the computer program product to perform the method according to the disclosed embodiments. Alternatively, at least one of the electronic device (100) and the third device may execute the computer program product to perform the method according to the disclosed embodiments in a distributed manner.

[0188] For example, the electronic device (100) may execute a computer program product stored in a memory (140, see FIG. 3) to control another electronic device (e.g., a wearable device) that is communicatively connected to the electronic device (100) to perform a method according to the disclosed embodiments.

[0189] As another example, a third device may execute a computer program product to control an electronic device in communication with the third device to perform a method according to the disclosed embodiment.

[0190] When a third device executes a computer program product, the third device may download the computer program product from the electronic device (100) and execute the downloaded computer program product. Alternatively, the third device may execute a computer program product provided in a pre-loaded state to perform the method according to the disclosed embodiments.

[0191] Although the embodiments described above have been described with limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above description. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components such as the described computer system or modules are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

Claims

1. In electronic devices, display; A camera that acquires images; a memory storing at least one instruction; and At least one processor executing at least one of the above instructions; Including, The electronic device, wherein said at least one processor executes said at least one instruction: Obtain spatial information about the real space based on the image acquired through the above camera, Obtaining user input based on the image acquired through the above camera, Obtain object characteristic information from the above user input, Obtaining object creation information for creating a virtual object based on the above spatial information and the above object characteristic information, By inputting the object creation information into a generative artificial intelligence model trained to create a three-dimensional virtual object based on information about space and objects, a virtual object is created for the object creation information, An electronic device that controls the display to display the virtual object.

2. In paragraph 1, The above camera includes a first camera that photographs a real space to obtain a spatial image of the real space, The electronic device, wherein said at least one processor executes said at least one instruction: An electronic device that obtains spatial information about at least one of the type, category, color, theme, and atmosphere of a real space from the spatial image obtained through the first camera.

3. In paragraph 1, The above camera includes a second camera that photographs the user's hand to obtain an image of the hand, The electronic device, wherein said at least one processor executes said at least one instruction: Recognize the user's gesture input from the hand image acquired through the second camera, An electronic device that extracts object characteristic information regarding at least one of shape, position, and size of an object from the gesture input.

4. In paragraph 3, The second camera is configured as a depth camera including at least one of a ToF (Time of Flight) camera, a stereo vision camera, and a Light Detection and Ranging (LiDAR) sensor, and captures an image of the user's hand to obtain a depth image, The electronic device, wherein said at least one processor executes said at least one instruction: An electronic device that recognizes a user's gesture input from the depth image acquired through the second camera.

5. In paragraph 1, A touchscreen that receives user touch input; Including more, The electronic device, wherein said at least one processor executes said at least one instruction: Recognizes gesture input from touch input received through the above touch screen, An electronic device that extracts object characteristic information regarding at least one of shape, position, and size of an object from the gesture input.

6. In paragraph 1, A microphone for receiving voice input from the user; Including more, The electronic device, wherein said at least one processor executes said at least one instruction: Obtaining a voice signal from a voice input received through the above microphone, Convert the above speech signal into text, An electronic device that extracts object characteristic information including at least one of the type, shape, color, and theme of the object by analyzing the text using a natural language understanding model.

7. In paragraph 1, The electronic device, wherein said at least one processor executes said at least one instruction: Obtain a 2D guide image, An electronic device that extracts object characteristic information including at least one of the type, shape, color, and theme of the object from the two-dimensional guide image.

8. In paragraph 1, The above object characteristic information includes first object characteristic information obtained from spatial information, second object characteristic information obtained from gesture input, third characteristic information obtained from voice input, and fourth characteristic information obtained from a two-dimensional guide image. The electronic device, wherein said at least one processor executes said at least one instruction: Converting the first object characteristic information into first feature data by vector embedding the above spatial information, Convert the second object characteristic information obtained from the above gesture input into second feature data by vector embedding, Convert the third object characteristic information obtained from the above voice input into third feature data by vector embedding, The fourth object characteristic information obtained from the above two-dimensional guide image is converted into fourth feature data by vector embedding, An electronic device that obtains feature data representing object creation information based on the first feature data to the fourth feature data.

9. In paragraph 8, The electronic device, wherein said at least one processor executes said at least one instruction: An electronic device that modifies object creation information based on user inputs.

10. In paragraph 9, The electronic device, wherein said at least one processor executes said at least one instruction: An electronic device that modifies the object creation information by adjusting a weight value assigned to each of the first object characteristic information, the second object characteristic information obtained from the gesture input, the third object characteristic information obtained from the voice input, and the fourth object characteristic information obtained from the two-dimensional guide image based on the user inputs.

11. A method for an electronic device to create a virtual object, A step of acquiring an image through a camera; A step of obtaining user input based on an image obtained through the above camera; A step of obtaining spatial information about a real space from the acquired image; A step of obtaining object characteristic information from the acquired user input; A step of obtaining object creation information for creating a virtual object based on the above spatial information and the above object characteristic information; A step of inputting the object creation information into a generative artificial intelligence model trained to create a three-dimensional virtual object based on information about space and objects, thereby creating a virtual object for the object creation information; and A step of displaying the above virtual object; A method comprising:

12. In paragraph 11, The above object characteristic information includes first object characteristic information obtained from spatial information, second object characteristic information obtained from gesture input, third characteristic information obtained from voice input, and fourth characteristic information obtained from a two-dimensional guide image. The step of obtaining the above object creation information is: A step of converting the first object characteristic information into first feature data by vector embedding the above spatial information; A step of converting the second object characteristic information obtained from the gesture input into second characteristic data by vector embedding; A step of converting the third object characteristic information obtained from the voice input into third feature data by vector embedding; A step of converting the fourth object characteristic information obtained from the two-dimensional guide image into fourth characteristic data by vector embedding; and A step of obtaining feature data representing the object creation information based on the first to fourth feature data; A method comprising:

13. In paragraph 12, A step of receiving user inputs for modifying the above object creation information; and A step of modifying the object creation information based on the received user inputs; A method further comprising:

14. In paragraph 13, The steps for modifying the above object creation information are: A method for modifying the object creation information by adjusting the weight values ​​assigned to each of the first object characteristic information obtained from the image based on the user input, the second object characteristic information obtained from the gesture input, the third object characteristic information obtained from the voice input, and the fourth object characteristic information obtained from the two-dimensional guide image.

15. In a computer program product including a computer-readable storage medium, A step of acquiring an image through a camera; A step of obtaining user input based on an image obtained through the above camera; A step of obtaining spatial information about a real space from the above image; A step of obtaining object characteristic information from the above user input; A step of obtaining object creation information for creating a virtual object based on the above spatial information and the above object characteristic information; A step of inputting the object creation information into a generative artificial intelligence model trained to create a three-dimensional virtual object based on information about space and objects, thereby creating a virtual object for the object creation information; and A step of displaying the above virtual object on a display; A computer program product comprising:

Images