Electronic device for generating virtual object and operating method thereof

Abstract

An electronic device according to one or more embodiments includes: a display; a camera for acquiring an image; a memory to store at least one instruction; and at least one processor to execute the at least one instruction, where the at least one processor executes the at least one instruction to: acquire spatial information about a real space based on an image acquired through the camera; acquiring a user input based on the image acquired through the camera; obtaining object characteristic information from the user input; obtaining object generation information for generating a virtual object based on the spatial information and the object characteristic information; generating a virtual object related to the object generation information by inputting the object generation information to a generative artificial intelligence model trained to generate a three-dimensional virtual object based on information about a space and an object; and controlling the display to display the virtual object.

Description

TECHNICAL FIELD

[0001] The disclosure relates to an electronic device for generating a three-dimensional (3D) virtual object in a real-world space and an operating method thereof. More specifically, the disclosure provides an electronic device for generating a 3D virtual object by using multi-modal information and providing the generated 3D virtual object and an operating method thereof. BACKGROUND

[0002] Augmented reality (AR) is a technology that superimposes virtual objects on a physical environment space or on real-world objects in a real world and displays them together, and has an advantage of combining virtual objects and virtual information into a real-world space. AR devices (e.g., smart glasses) using AR technology have become useful in daily life, such as information search, route guidance, and camera shooting. In particular, smart glasses are also worn as fashion items and are mainly used for outdoor activities.

[0003] AR technology that is currently being used and popularized is capable of placing and displaying virtual objects in a real-world space, but the arrangement of virtual objects is limited to the arrangement of a pre-modeled three-dimensional (3D) virtual object. For example, a 3D object modeling technology is a technology for generating a 3D virtual object by using depth information, pose information, and texture information, but has a technical limitation of being unable to generate an object that does not exist in reality. As another example, a modeling method using a deep neural network model (deep learning) is a method of performing object modeling to generate a 3D virtual object by optimizing two-dimensional (2D) images and position data of an object captured at various positions using a trained deep neural network model, and also has a limitation of being unable to use the modeling method when generating an object that does not exist in reality. In addition, a currently available distributed virtual object generation technology cannot modify a generated 3D virtual object.

[0004] As described above, the current AR technology cannot generate a virtual object having a shape that does not exist in reality, and does not allow a user to modify a generated virtual object, thereby providing only a limited AR experience, which can hinder the expansion of AR services. SUMMARY

[0005] TECHNICAL SOLUTION According to one or more example embodiments, an electronic device may include: a display; a camera configured to acquire images; a memory storing at least one instruction; and at least one processor configured to execute the at least one instruction to perform the following operations: obtaining spatial information about a real-world space based on the images acquired by the camera; obtaining user input based on the images acquired by the camera; obtaining object characteristic information from the user input; obtaining object generation information for generating virtual objects based on the spatial information and the object characteristic information; generating the virtual object for the object generation information by inputting the object generation information into a generative artificial intelligence (AI) model, wherein the generative AI model is trained to generate three-dimensional (3D) virtual objects based on information about space and objects; and controlling the display to display the virtual objects.

[0006] According to one or more example embodiments, a method for generating virtual objects performed by an electronic device may include: acquiring an image via a camera; obtaining user input based on the image acquired via the camera; obtaining spatial information about a real-world space based on the image; obtaining object characteristic information from the user input; obtaining object generation information for generating the virtual object based on the spatial information and the object characteristic information; generating the virtual object for the object generation information by inputting the object generation information into a generative artificial intelligence (AI) model, wherein the generative AI model is trained to generate three-dimensional (3D) virtual objects based on information about space and objects; and displaying the virtual object using a display.

[0007] According to one or more example embodiments, a computer program product may include a computer-readable storage medium, wherein the computer-readable storage medium includes instructions readable by an electronic device for performing the following operations: acquiring an image via a camera; obtaining user input based on the image acquired via the camera; obtaining spatial information about a real-world space based on the image; obtaining object characteristic information from the user input; obtaining object generation information for generating a virtual object based on the spatial information and the object characteristic information; generating the virtual object for the object generation information by inputting the object generation information into a generative artificial intelligence (AI) model, wherein the generative AI model is trained to generate a three-dimensional (3D) virtual object based on information about space and the object; and displaying the virtual object using a display.

[0008] According to one or more example embodiments, a head-mounted display (HMD) may include: a display; a camera configured to acquire images; a memory storing at least one instruction; and at least one processor configured to execute the at least one instruction to perform the following operations: obtaining spatial information about real-world space based on the images acquired through the camera; obtaining user input based on the images acquired through the camera; obtaining object characteristic information based on the user input, the object characteristic information describing an object that will occupy the real-world space; obtaining object generation information based on the spatial information and the object characteristic information for generating a virtual object corresponding to the object that will occupy the real-world space; generating the virtual object for the object generation information by inputting the object generation information into a generative artificial intelligence (AI) model; and controlling the display to display the virtual object in real-world space. Attached Figure Description

[0009] The embodiments will be more clearly understood through the following detailed description taken in conjunction with the accompanying drawings, in which: Figure 1 This is a conceptual diagram illustrating the operation of an electronic device according to one or more embodiments of the present disclosure to generate a three-dimensional (3D) virtual object based on multimodal information and display the generated 3D virtual object.

[0010] Figure 2 This is a flowchart of an operation method of an electronic device according to one or more embodiments of the present disclosure.

[0011] Figure 3 This is a block diagram illustrating components of an electronic device according to one or more embodiments of the present disclosure.

[0012] Figure 4 This is a diagram illustrating data input and output performed in an electronic device between a software module stored in memory and each of a camera, microphone, and display, according to one or more embodiments of the present disclosure.

[0013] Figure 5 This is a flowchart of a method performed by an electronic device to obtain a vector representing object generation information, according to one or more embodiments of the present disclosure.

[0014] Figure 6 This is a diagram illustrating the operation of an electronic device according to one or more embodiments of the present disclosure to obtain feature data representing object generation information by embedding multimodal information.

[0015] Figure 7 This is a diagram illustrating the operation of an electronic device according to one or more embodiments of the present disclosure to assign random values ​​to missing vector values ​​in feature data representing object generation information.

[0016] Figure 8 This is a flowchart illustrating a method performed by an electronic device to generate a 3D virtual object by modifying object generation information based on user input, according to one or more embodiments of the present disclosure.

[0017] Figure 9 This is a diagram illustrating components of an electronic device for generating 3D virtual objects by modifying object generation information, according to one or more embodiments of the present disclosure.

[0018] Figure 10 This is a diagram illustrating the operation of an electronic device according to one or more embodiments of the present disclosure to modify object generation information by receiving user input for adjusting the weights of multimodal information.

[0019] Figure 11 This is a diagram illustrating the operation of an electronic device according to one or more embodiments of the present disclosure to modify object generation information by receiving user input for modifying the shape, size, and position of an object.

[0020] Figure 12a This is a diagram illustrating the operation of an electronic device according to one or more embodiments of the present disclosure to generate and display a 3D virtual object based on multimodal information.

[0021] Figure 12b This is a diagram illustrating the operation of an electronic device according to one or more embodiments of the present disclosure to generate and display a 3D virtual object based on multimodal information.

[0022] Figure 12c This is a diagram illustrating the operation of an electronic device according to one or more embodiments of the present disclosure to generate and display a 3D virtual object based on multimodal information.

[0023] Figure 12d This is a diagram illustrating the operation of an electronic device according to one or more embodiments of the present disclosure to generate and display a 3D virtual object based on multimodal information.

[0024] Figure 12e This is a diagram illustrating the operation of an electronic device according to one or more embodiments of the present disclosure to generate and display a 3D virtual object based on multimodal information. Detailed Implementation

[0025] The terms currently widely used are selected for use herein by taking into account the functionality of this disclosure; however, these terms may be changed based on the intent of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, the applicant may arbitrarily choose specific terms, and in such cases, the meaning of the selected terms will be described in detail in the detailed description of the corresponding embodiments of this disclosure. Therefore, the terms used herein should not be defined by their simple appellations, but rather based on their meanings and the overall description of this disclosure.

[0026] Unless the context clearly indicates otherwise, the singular expressions used herein are intended to include the plural expressions as well. All terms used herein (including technical or scientific terms) may have the same meaning as commonly understood by one of ordinary skill in the art.

[0027] Throughout this disclosure, when a component "comprises" or "includes" an element, the component may also include other elements, without excluding them, unless there is a specific description to the contrary. Furthermore, terms such as "part" and "module" as used herein indicate a unit for performing at least one function or operation, and may be implemented as hardware or software, or a combination of hardware and software.

[0028] Depending on the context, the expression “configured as (or set to)” as used herein may be used interchangeably with expressions such as “suitable for,” “capable of,” “designed for,” “suitable for,” “manufactured as,” or “capable of.” The term “configured as (or set to)” may not necessarily mean only “specifically designed for” in terms of hardware. Rather, in some contexts, the expression “system configured as…” may mean that the system, together with other devices or components, is “capable of….” For example, the expression “processor configured as (or set to) perform A, B, and C” may mean a dedicated processor (e.g., an embedded processor) for performing the corresponding operations, or a general-purpose processor (e.g., a central processing unit (CPU) or application processor (AP)) capable of performing the corresponding operations by executing one or more software programs stored in memory.

[0029] Furthermore, it should be understood that when a component is referred to herein as “connected” or “coupled” to another component, the component may be directly connected or coupled to the other component, but may also be connected or coupled to the other component via another intermediate component, unless there is a specific description to the contrary.

[0030] In this disclosure, "electronic device" can be a mobile device, a smartphone, or a tablet PC. However, electronic device is not limited to these and can be implemented as a variety of electronic devices, such as laptop computers, desktop computers, e-book terminals, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation devices, MP3 players, cameras, Internet Protocol Television (IPTV), digital television (DTV), wearable devices, etc.

[0031] In one or more embodiments of this disclosure, the electronic device may be an augmented reality (AR) device. An "AR device" is a device capable of implementing AR, and for example, may be implemented as AR glasses worn on a user's face. However, the AR device is not limited to this and may be implemented as a head-mounted display (HMD) device worn on a user's head, an AR helmet, etc.

[0032] As used in this article, “AR” refers to the technology used to display virtual images or to display real-world objects and virtual images together in a real-world physical environment.

[0033] As used herein, "real-world space" refers to a space in the real world as seen by a user through an AR device. In one or more embodiments of this disclosure, real-world space may mean an indoor space. Real-world objects may be placed within real-world space.

[0034] Furthermore, a "virtual object" is an image generated via an optical engine and may include both static and dynamic images. Such a virtual object is observed alongside a real-world scene and can be an image representing information about real-world objects in the scene, information about the operation of the AR device, control menus, etc. In one or more embodiments of this disclosure, a "virtual object" may include a user interface (UI) provided by an application or program executed by the AR device.

[0035] As used in this article, "3D virtual object" refers to a virtual object that is implemented as a 3D stereoscopic object.

[0036] The functions related to artificial intelligence (AI) according to this disclosure are executed via a processor and memory. The processor can be configured as one or more processors. In this case, the one or more processors can be general-purpose processors such as CPUs, APs, digital signal processors (DSPs), dedicated graphics processors such as graphics processing units (GPUs), vision processing units (VPUs), or dedicated AI processors such as neural processing units (NPUs). The one or more processors control the input data to be processed according to predefined operating rules or AI models stored in memory. Optionally, when the one or more processors are dedicated AI processors, dedicated AI processors can be designed using hardware architectures specifically designed to process particular AI models.

[0037] Predefined operating rules or AI models are created via a training process. In this context, "created via a training process" means creating predefined operating rules or AI models configured to perform desired characteristics (or purposes) by training a basic AI model on a large amount of training data using a learning algorithm. This training process can be performed by the device itself performing the AI ​​according to this disclosure, or via a separate server and / or system. Examples of learning algorithms may include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning.

[0038] In this disclosure, an "AI model" can be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values, and neural network computation can be performed by calculating the results of computation in the previous layer and the multiple weight values. The multiple weight values ​​assigned to each of the multiple neural network layers can be optimized by the results of training the AI ​​model. For example, the multiple weight values ​​can be updated to reduce or minimize the loss or cost gained in the AI ​​model during the training process. The artificial neural network can include deep neural networks (DNNs), and can be, for example, convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent DNNs (BRDNNs), or deep Q-networks (DQNs), but is not limited thereto.

[0039] As used herein, "visual recognition" refers to image signal processing involving inputting images into an AI model and using the AI ​​model to perform inference to detect objects in the input images, classify objects into specific categories, or segment objects. In one or more embodiments of this disclosure, visual recognition may refer to image processing involving identifying objects from images captured by a camera using an AI model to obtain classification or type information of the objects.

[0040] One or more embodiments of the present disclosure will now be described more fully with reference to the accompanying drawings to enable those skilled in the art to readily implement embodiments of the present disclosure. However, the present disclosure may be implemented in various forms, and it should not be construed as being limited to the one or more embodiments set forth herein.

[0041] The present disclosure is described in detail below with reference to the accompanying drawings.

[0042] Figure 1 This is a conceptual diagram illustrating the operation of an electronic device 100 according to one or more embodiments of the present disclosure to generate a 3D virtual object 70 based on multimodal input and display the generated 3D virtual object.

[0043] Reference Figure 1 The electronic device 100 may be glasses-shaped AR glasses worn on the user's face. However, the electronic device 100 is not limited to this, and in one or more embodiments of this disclosure, the electronic device 100 may be implemented as various electronic devices, such as mobile devices, smartphones, laptops, desktops, tablet PCs, e-book terminals, digital broadcasting terminals, PDAs, PMPs, navigation devices, MP3 players, cameras, IPTV, DTV, and wearable devices, etc.

[0044] Electronic device 100 may include camera 110. (See reference...) Figure 1 The electronic device 100 can obtain a spatial image 20 by using a camera 110 to capture images of the real-world space 10, receive gesture input 30 and voice input 40 from the user, and obtain a two-dimensional (2D) guide image 50 (operation ①).

[0045] The electronic device 100 can obtain spatial information from the spatial image 20, and obtain multiple object characteristic information from the gesture input 30, voice input 40 and 2D guide image 50 (operation ②).

[0046] The electronic device 100 can obtain object generation information 60 (operation ③) by mixing the acquired spatial information with multiple object characteristic information.

[0047] The electronic device 100 can generate a 3D virtual object 70 by inputting object generation information 60 into the generative AI model 146 and performing inference on it (operation ④).

[0048] The electronic device 100 can display the generated 3D virtual object 70 (operation ⑤).

[0049] See below for reference. Figure 1 and Figure 2 Describe in detail the function and / or operation of the electronic device 100.

[0050] Figure 2 This is a flowchart of an operation method of an electronic device 100 according to one or more embodiments of the present disclosure.

[0051] In operation S210, electronic device 100 acquires an image via a camera. Combined Figure 2 Reference Figure 1 The electronic device 100 can obtain a spatial image 20 by using a camera 110 to capture images of the real-world space 10.

[0052] In operation S220, electronic device 100 obtains user input based on an image acquired by camera 110. In one or more embodiments of this disclosure, electronic device 100 can obtain a hand image by capturing an image of the user's hand using camera 110, identify the user's hand from the hand image, and identify the gesture performed by the identified user's hand. However, this disclosure is not limited thereto, and electronic device 100 can be implemented as a mobile device, smartphone, or tablet PC including a touchscreen. In this case, electronic device 100 can identify gesture input from user touch input received via the touchscreen. Figure 1 In the embodiments of this disclosure shown, the electronic device 100 can recognize gesture input 30 from a user drawing a table shape.

[0053] In one or more embodiments of this disclosure, the electronic device 100 may include a microphone 120 and receive voice input 40 from a user via the microphone 120. Figure 1 In the embodiment of this disclosure shown, electronic device 100 can receive voice input 40 "Make a desk suitable for the office".

[0054] In one or more embodiments of this disclosure, electronic device 100 may acquire a 2D guide image 50. As used herein, "2D guide image 50" refers to a 2D image including guide information associated with at least one of the type, shape, or theme of an object used to generate the 3D virtual object 70. Figure 1 In the embodiments of this disclosure shown, the 2D guide image 50 may be a 2D image showing a table with iron legs and a wooden top. In one or more embodiments of this disclosure, the electronic device 100 may receive the 2D guide image 50 from an external server or storage device. However, this disclosure is not limited thereto, and the electronic device 100 may acquire the 2D guide image 50 input via user input.

[0055] In operation S230, the electronic device 100 obtains spatial information about real-world space from the acquired image. In this disclosure, "spatial information" is information about real-world space and may include, for example, information about the type, category, or atmosphere of the real-world space. Figure 2Reference Figure 1 The electronic device 100 can obtain spatial information about the real-world space 10 from the spatial image 20. In one or more embodiments of this disclosure, the electronic device 100 can use an AI model to perform visual recognition to identify the type of the real-world space 10 from the spatial image 20, classify the real-world space 10 into categories, and identify objects within the real-world space 10. For example, the AI ​​model can be a DNN model trained via a supervised learning method to obtain spatial information (such as the type, category, or atmosphere of the space) from the spatial image 20, which applies multiple images of the space as input and applies label values ​​associated with the type, category, or atmosphere of the space as ground truth values. Figure 1 In the embodiments of this disclosure shown, electronic device 100 can obtain information about the type or category of the space (such as “office”) and information about the atmosphere or theme of the space (such as “grayscale”) from spatial image 20.

[0056] In operation S240, the electronic device 100 obtains multiple pieces of object characteristic information from the received user input. As used herein, "object characteristic information" refers to information used to generate the 3D virtual object 70 according to the user's intent, and may include, for example, information about the object's shape, position, size, type, category, color, or theme. In other words, object characteristic information can describe the object that will occupy real-world space.

[0057] Electronic device 100 can recognize object characteristic information based on user intent from a user's gesture input 30 obtained from an image acquired via camera 110. In one or more embodiments of this disclosure, electronic device 100 can recognize a user's gesture by identifying the area the user is pointing at and the trajectory of that area from an image. Electronic device 100 can extract object characteristic information, including at least one of the object's position, size, or shape, from the recognized gesture input 30. Figure 1 In the embodiments of this disclosure shown, the electronic device 100 can extract object characteristic information, including at least one of the position, size, or shape of the table, from the gesture input 30.

[0058] Electronic device 100 can convert voice input 40 received through microphone 120 into a speech signal and perform automatic speech recognition (ASR) to convert the speech signal into text. Electronic device 100 can analyze the text using a natural language understanding (NLU) model and identify the user's intent and named entities based on the analysis results. Electronic device 100 can obtain object characteristic information, including at least one of the following: object type, shape, color, or theme, based on the analysis results of the text. Figure 1In the embodiments of this disclosure shown, the electronic device 100 can obtain the intent of “object generation” and information about named entities such as “table” and “office” from the voice input 40.

[0059] Electronic device 100 can obtain object characteristic information from 2D guide image 50. In one or more embodiments of this disclosure, electronic device 100 can extract object characteristic information from 2D guide image 50, including at least one of object type, shape, color, or theme. Figure 1 In the embodiments of this disclosure shown, the electronic device 100 can extract object characteristic information from the 2D guide image 50, such as the type of the object (e.g., "table"), the shape of the table including a wooden top and iron legs, and colors including brown and black.

[0060] In operation S250, electronic device 100 obtains object generation information based on spatial information and multiple object characteristic information. Combined with... Figure 2 Reference Figure 1 The electronic device 100 can obtain object generation information 60 by mixing spatial information obtained from spatial image 20, including at least one of the type, category, or theme of real-world space 10; object characteristic information obtained from gesture input 30, including at least one of the position, size, or shape of an object; object characteristic information obtained from voice input 40, including at least one of the type, shape, color, or theme of an object; and object characteristic information obtained from 2D guide image 50, including at least one of the type, shape, color, or theme of an object. In one or more embodiments of this disclosure, the electronic device 100 can convert spatial information and multiple object characteristic information into multiple feature data by performing vector embedding on the spatial information and multiple object characteristic information respectively, and obtain feature data representing object generation information 60 by mixing the multiple feature data together. The "feature data" can consist of tensors including n-dimensional vectors, matrices, or eigenvectors having one or more dimensions. In one or more embodiments of this disclosure, the electronic device 100 can transform the spatial information and multiple object characteristic information by performing vector embedding on the spatial information and multiple object characteristic information respectively to convert the spatial information and multiple object characteristic information into feature vectors, and obtain feature vectors representing object generation information 60 by using multilayer perceptron (MLP) technology to mix the feature vectors together. However, this disclosure is not limited thereto, and in one or more embodiments of this disclosure, the electronic device 100 can obtain feature data representing object generation information 60 by inputting multiple feature data into a CNN or RNN.

[0061] In operation S260, electronic device 100 can generate 3D virtual objects based on object generation information by inputting object generation information into a generative AI model. As used herein, a "generative AI model" (… Figure 1 and Figure 3 Generative AI model 146 is an AI model trained to generate 3D virtual objects based on spatial and object information. In one or more embodiments of this disclosure, generative AI model 146 may be implemented as a generative adversarial network (GAN) or a multimodal diffusion-based model. A multimodal diffusion-based model may be implemented as, for example, stable diffusion or SDFusion. However, generative AI model 146 is not limited thereto and may include any known AI model in the art trained to generate 3D virtual objects based on spatial and object information.

[0062] Combination Figure 2 Reference Figure 1 The electronic device 100 can input object generation information 60 into a generative AI model 146, and generate a 3D virtual object 70 for the object generation information 60 by performing inference using the generative AI model 146. Figure 1 In the embodiments of this disclosure shown, electronic device 100 can generate 3D virtual object 70 via generative AI model 146, which is a virtual image representing the 3D shape of a table.

[0063] exist Figure 1 In operation ⑤, the electronic device 100 can display the generated 3D virtual object 70. In one or more embodiments of this disclosure, the electronic device 100 may consist of AR glasses, and the AR glasses may consist of a lens optical system and include a display comprising a waveguide and an optical engine. Figure 3 (150). Figure 1 In the embodiments of this disclosure shown, the electronic device 100 can generate light constituting a virtual image of the 3D virtual object 70 using an optical engine and project the light onto a waveguide to display the 3D virtual object 70 in real-world space 10. However, this disclosure is not limited thereto, and in one or more embodiments of this disclosure, when the electronic device 100 is implemented as a mobile device such as a smartphone or tablet PC, the electronic device 100 can display the 3D virtual object 70 on a liquid crystal display (LCD), a thin-film transistor LCD (TFT-LCD), an organic light-emitting diode (OLED) display, a flexible display, a 3D display, an electrophoretic display, etc.

[0064] Current AR technologies, while capable of placing and displaying virtual objects in real-world space, are limited to arranging pre-modeled 3D virtual objects. In other words, current AR technology cannot generate 3D virtual objects with shapes that don't exist in reality, and it doesn't allow users to modify the generated 3D virtual objects, thus providing only a limited AR experience. Therefore, the expansion of AR services may be hindered, and its utilization may be reduced.

[0065] This disclosure provides an electronic device 100 and a method thereof for performing the following operations: generating a 3D virtual object based on multimodal information including at least one of a spatial image related to real-world space, a gesture input received from a user, a voice input, or a 2D guide image, and providing the generated 3D virtual object.

[0066] The electronic device 100 of this disclosure can generate a 3D virtual object 70 by using information from multiple modalities (i.e., spatial information obtained from spatial image 20 and multiple object characteristic information obtained from gesture input 30, voice input 40, and 2D guide image 50), thereby quickly generating and providing a 3D model of the virtual object desired by the user. In doing so, the electronic device 100 of this disclosure provides the technical effect of enabling users to have an active AR experience. Furthermore, the electronic device 100 of this disclosure, based on an AR service environment supporting multiple input modalities, generates the 3D virtual object 70 by using multimodal input information, thereby providing users with a convenient and highly usable AR experience.

[0067] The following reference Figures 8 to 11 This disclosure describes in detail one or more embodiments of the electronic device 100 that modifies the 3D virtual object 70 based on user input.

[0068] Figure 3 This is a block diagram illustrating the components of an electronic device 100 according to one or more embodiments of the present disclosure.

[0069] Reference Figure 3 The electronic device 100 may include a camera 110, a microphone 120, a processor 130, a memory 140, and a display 150. The camera 110, microphone 120, processor 130, memory 140, and display 150 may be electrically and / or physically connected to each other. Figure 3 Only the necessary components for describing the operation of electronic device 100 are shown, and the components included in electronic device 100 are not limited to... Figure 3The components are shown. In one or more embodiments of this disclosure, the electronic device 100 may further include a communication interface for performing data communication with an external device or server. In one or more embodiments of this disclosure, the electronic device 100 may be implemented as a portable device, and in this case, the electronic device 100 may further include a battery for supplying driving power to the camera 110, microphone 120, processor 130, and display 150.

[0070] Camera 110 is configured to acquire images of real-world space and objects within that space. Camera 110 may include a lens module, an image sensor, and an image processing module. Camera 110 may acquire still images or videos of objects using an image sensor (e.g., a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) sensor). Video may include multiple image frames acquired sequentially by camera 110 capturing images of objects. The image processing module may encode still image data consisting of a single image frame acquired by the image sensor or video data consisting of multiple image frames acquired by the image sensor and send the encoded data to processor 130. In one or more embodiments of this disclosure, camera 110 may include a first camera 112 and a second camera 114.

[0071] The first camera 112 is configured to acquire a spatial image of real-world space by capturing images of real-world space. In one or more embodiments of this disclosure, the first camera 112 may be configured as a camera including an image sensor, such as a red-green-blue (RGB) camera or a grayscale camera. In one or more embodiments of this disclosure, the first camera 112 may be implemented in a miniaturized design so that it can be mounted on the electronic device 100 in a portable form, and may be implemented as a lightweight RGB camera that consumes low power.

[0072] The second camera 114 is configured to obtain a hand image by capturing an image of the user's hand. The second camera 114 may be configured as an RGB camera including an image sensor, but is not limited thereto. In one or more embodiments of this disclosure, the second camera 114 may be configured as a depth camera to obtain depth values ​​of an object (e.g., the user's hand). When the second camera 114 is configured as a depth camera, it may be implemented as at least one of, for example, a time-of-flight (ToF) camera, a stereo vision camera, or a light detection and ranging (LiDAR) sensor, and obtain a depth map by measuring 3D depth values ​​of joints included in the user's hand.

[0073] Microphone 120 can receive voice input (e.g., the user's speech) from the user and obtain a voice signal from the received voice input. Microphone 120 provides the obtained voice signal to processor 130.

[0074] Processor 130 can execute one or more instructions of a program stored in memory 140. Processor 130 may consist of hardware components that perform arithmetic, logic, and input / output (I / O) operations, as well as image processing. Figure 3 In this disclosure, processor 130 is shown as a single component, but is not limited thereto. In one or more embodiments of this disclosure, processor 130 may include one or more elements.

[0075] The processor 130 according to embodiments of this disclosure may include various processing circuitry and / or multiple processors. For example, as used herein (including the claims), the term "processor" may include various processing circuitry comprising at least one processor, wherein one or more of the at least one processor may be configured individually and / or collectively in a distributed manner to perform the various functions described herein. As used herein, when "processor," "at least one processor," and "one or more processors" are described as being configured to perform a number of functions, these terms cover, for example, but not limited to, a situation where one processor performs some of the functions while another or more processors perform other functions, and a situation where a single processor can perform all of the functions. Additionally, at least one processor may include, for example, a combination of processors performing the various described / disclosed functions in a distributed manner. At least one processor may execute program instructions to implement or perform various functions.

[0076] The processor 130 can be a general-purpose processor such as a CPU, AP, or DSP; a dedicated graphics processor such as a GPU or VPU; or a dedicated AI processor such as an NPU. The processor 130 can control the processing of input data according to predefined operating rules or AI models. Optionally, when the processor 130 is a dedicated AI processor, the dedicated AI processor can be designed with a hardware architecture specifically for processing a particular AI model.

[0077] For example, memory 140 may include at least one type of storage medium, namely, flash memory, hard disk memory, multimedia card micro-memory, card memory (e.g., security digital (SD) card or extreme digital (XD) memory), random access memory (RAM), static RAM (SRAM), read-only memory (ROM), electrically erasable programmable ROM (EEPROM), PROM, magnetic storage, magnetic disk, or optical disk.

[0078] Memory 140 may store instructions relating to the following functions and / or operations performed by electronic device 100: obtaining spatial information from spatial images, obtaining object characteristic information from gesture input, voice input, and 2D guided images, and providing AR services by generating 3D virtual objects using spatial information and object characteristic information. In one or more embodiments of this disclosure, memory 140 may store at least one of instructions, algorithms, data structures, program code, or applications readable by processor 130. The instructions, algorithms, data structures, and program code stored in memory 140 may be implemented using programming or scripting languages ​​such as C, C++, Java, and assembly language.

[0079] The memory 140 may store instructions, algorithms, data structures, or program code related to the spatial information extraction module 141, gesture information extraction module 142, voice information extraction module 143, guidance information extraction module 144, information mixing module 145, generative AI model 146, and object modification module 147. The "modules" included in the memory 140 refer to units used to process functions or operations performed by the processor 130, and may be implemented as software such as instructions, algorithms, data structures, or program code.

[0080] The processor 130 can be implemented by executing instructions or program code stored in memory 140. (See below for further details.) Figure 3 and Figure 4 The processor 130 is described in detail as performing functions and / or operations by executing instructions or program code stored in memory 140 for each of a plurality of modules, as well as data input / output between the plurality of modules and components (e.g., camera 110, microphone 120, and display 150).

[0081] Figure 4 This is a diagram illustrating data input and output performed in an electronic device 100 between a software module stored in a memory 140 and each of a camera 110, a microphone 120, and a display 150, according to one or more embodiments of the present disclosure.

[0082] Refer to together Figure 3 and Figure 4The spatial information extraction module 141 comprises instructions or program code related to the function and / or operation of extracting spatial information about real-world space from spatial images acquired by the first camera 112. In one or more embodiments of this disclosure, "spatial information" may include information about at least one of the type, category, color, theme, or atmosphere of a real-world space. The type or category of a real-world space refers to classification information of a space obtained as a result of reasoning by an AI model, such as an office, café, school, library, playground, etc. The color, theme, or atmosphere of a real-world space is characteristic information based on the color or hue of the space, such as information like grayscale, brownscale, or greenscale. In one or more embodiments of this disclosure, the spatial information extraction module 141 may include an AI model trained to perform the following operations: identify the type of real-world space from a spatial image, classify the real-world space into a category, and identify colors, themes, etc. For example, an AI model could be a DNN model trained using a supervised learning method to obtain spatial information (such as the type, category, or atmosphere of the space) from spatial images. This supervised learning method applies multiple images of the space as input and applies label values ​​related to the type, category, etc., of the space as ground truth values. The DNN model could be, for example, a CNN model. However, the DNN model is not limited to this and can be implemented as any known DNN model, including RNNs, RBMs, DBNs, BRDNNs, DQNs, etc.

[0083] Processor 130 can execute instructions or program code of spatial information extraction module 141 to extract information about at least one of the type, category, color, theme, or atmosphere of real-world space from spatial images acquired by first camera 112. In one or more embodiments of this disclosure, processor 130 can obtain spatial information from spatial images by inputting spatial images into a DNN model and using the DNN model to infer information from them.

[0084] The gesture information extraction module 142 consists of instructions or program code related to functions and / or operations related to recognizing user gesture input from a hand image and extracting object characteristic information from the gesture input. The processor 130 can execute the instructions or program code of the gesture information extraction module 142 to recognize the area the user is pointing to and the trajectory of that area from a hand image obtained by the second camera 114, thereby recognizing the user's gesture input. However, this disclosure is not limited thereto, and the electronic device 100 may also include a touchscreen. The touchscreen receives touch input from the user, and the processor 130 can recognize the user's gesture input from the touch input. The processor 130 can obtain object characteristic information from the recognized gesture input for generating a virtual object according to the user's intent; the object characteristic information includes at least one of the object's position, size, or shape.

[0085] The speech information extraction module 143 comprises instructions or program code related to the function and / or operation of recognizing speech input and extracting object characteristic information from the speech input. In one or more embodiments of this disclosure, the speech information extraction module 143 may include an ASR model and an NLU model. The processor 130 may convert speech input (i.e., sound) obtained through the microphone 120 into an acoustic signal and remove noise (e.g., non-speech components) from the acoustic signal to obtain a speech signal. The processor 130 may execute the instructions or program code of the speech information extraction module 143 to obtain object characteristic information from the obtained speech signal. In one or more embodiments of this disclosure, the processor 130 may perform ASR to convert the speech signal into text and analyze the text using an NLU model to identify the user's intent and named entities. The processor 130 may obtain object characteristic information, including at least one of the object's type, shape, color, or theme, required to generate virtual objects based on the user's intent and named entities.

[0086] The guidance information extraction module 144 consists of instructions or program code related to the function and / or operation of extracting object characteristic information from the 2D guidance image. In this disclosure, a "2D guidance image" refers to an image used to provide guidance information about the type, shape, or theme of an object required by the electronic device 100 to generate a 3D virtual object. In one or more embodiments of this disclosure, the electronic device 100 may also include a communication interface and receive 2D guidance images from an external server or storage device (e.g., a web-based database) via the communication interface. However, this disclosure is not limited thereto, and the electronic device 100 may obtain a 2D guidance image 50 input via user input.

[0087] In one or more embodiments of this disclosure, the guidance information extraction module 144 may include an object recognition model composed of an AI model. The object recognition model may be a DNN model trained via supervised learning to recognize objects, wherein supervised learning applies bounding box images that can be recognized as objects from multiple training images as input data, and applies label values ​​for the type, shape, color, and subject of the object within the bounding box as ground truth values. The DNN model may be implemented as an object recognition model, such as, but not limited to, CNN, region-based CNN (R-CNN), YOLO v4, CenterNet, or MobileNet. The processor 130 may execute instructions or program code of the guidance information extraction module 144 to extract object characteristic information from the 2D guidance image, including at least one of the object's type, shape, color, or subject.

[0088] Although Figure 3 and Figure 4The memory 140 is shown to include a boot information extraction module 144, but according to one or more embodiments of the present disclosure, the boot information extraction module 144 may be omitted.

[0089] The information mixing module 145 consists of instructions or program code related to the function and / or operation of obtaining object generation information by mixing spatial information and object characteristic information. (See reference...) Figure 4 The information mixing module 145 can receive spatial information, including the type, theme, and atmosphere of the space, output by the spatial information extraction module 141. Furthermore, the information mixing module 145 can receive object characteristic information, such as shape, position, and size, from the gesture information extraction module 142, object characteristic information, such as type, shape, and theme, from the voice information extraction module 143, and object characteristic information, such as type, shape, and theme, from the guidance information extraction module 144. The processor 130 can execute instructions or program code from the information mixing module 145 to obtain object generation information for generating virtual objects by mixing the spatial information received from the spatial information extraction module 141 with multiple object characteristic information from the gesture information extraction module 142, voice information extraction module 143, and guidance information extraction module 144, respectively.

[0090] In one or more embodiments of this disclosure, processor 130 may perform embedding on spatial information and multiple object characteristic information respectively to convert them into multiple feature data, and obtain feature data representing object generation information by mixing the multiple feature data. (Refer to...) Figure 5 and Figure 6 This disclosure describes in detail specific embodiments of the processor 130 obtaining feature data representing object generation information.

[0091] In one or more embodiments of this disclosure, when a vector value representing embedded feature data indicating spatial information or object characteristic information is missing, the processor 130 may assign a random value, which is a randomly generated numerical value, as the missing vector value. (Refer to...) Figure 7 This disclosure describes in detail a specific embodiment of the processor 130 assigning random values ​​as missing vector values.

[0092] Generative AI model 146 is an AI model trained to generate 3D virtual objects based on spatial information and object characteristic information. In one or more embodiments of this disclosure, generative AI model 146 can be implemented as a GAN or a multimodal diffusion-based model. A multimodal diffusion-based model can be implemented as, for example, stable diffusion or SDFusion. However, generative AI model 146 is not limited to these and can include any known AI model in the art trained to generate 3D virtual objects based on spatial information and object characteristic information. Figure 3 Reference Figure 4 The processor 130 can input object generation information obtained by the information mixing module 145 into the generative AI model 146 and use the generative AI model 146 to perform inference, thereby generating a 3D virtual object corresponding to the object generation information. In one or more embodiments of this disclosure, the processor 130 can generate a 3D virtual object by performing inference by inputting a vector representing object generation information into the generative AI model 146. The processor 130 can display the 3D virtual object generated by the generative AI model 146 on the display 150.

[0093] Reference Figure 3 The object modification module 147 comprises instructions or program code related to functions and / or operations for modifying object generation information based on user input. In one or more embodiments of this disclosure, the electronic device 100 may receive gesture input for modifying object generation information. However, this disclosure is not limited thereto, and when the electronic device 100 is implemented as a mobile device including a touchscreen (such as a smartphone or tablet PC), the electronic device 100 may receive touch input from a user for modifying object generation information. In one or more embodiments of this disclosure, the electronic device 100 may receive user input for adjusting weight values ​​assigned to each of the multimodal inputs (i.e., spatial information obtained from a spatial image, object characteristic information obtained from gesture input, object characteristic information obtained from voice input, and object characteristic information obtained from a 2D guided image), and the processor 130 may modify the object generation information based on the user input. In one or more embodiments of this disclosure, the electronic device 100 may receive gesture or touch input from a user for changing at least one of the shape, size, position, color, type, or theme of an object, and the processor 130 may modify the object generation information based on the received gesture or touch input. (See also...) Figures 8 to 11 This disclosure describes in detail specific embodiments of the processor 130 that modify object generation information based on user input.

[0094] Display 150 is configured to display 3D virtual objects under the control of processor 130. When electronic device 100 is an AR device such as AR glasses, display 150 may consist of a lens optics system and include a waveguide and an optical engine. The optical engine may consist of a projector used to generate light for 3D virtual objects composed of virtual images and project that light onto the waveguide. The optical engine may include, for example, an image panel, an illumination optics system, a projection optics system, etc. In one or more embodiments of this disclosure, the optical engine may be disposed in the frame or temples of AR glasses. In one or more embodiments of this disclosure, the optical engine can display virtual objects by projecting 3D virtual objects onto the waveguide under the control of processor 130.

[0095] However, this disclosure is not limited thereto, and the display 150 may include at least one of, for example, an LCD, a TFT-LCD, an OLED display, a flexible display, a 3D display, or an electrophoretic display. In this case, the display 150 may be configured as a touchscreen including a touchpad.

[0096] Figure 5 This is a flowchart of a method performed by an electronic device 100 to obtain feature data representing object generation information, according to one or more embodiments of the present disclosure.

[0097] Figure 5 Operations S510 to S550 are Figure 2 The detailed operation of operation S250 is shown below. During execution... Figure 5 After operating the S550, you can execute... Figure 2 The operation S260 is shown.

[0098] Figure 6 This is a diagram illustrating the operation of an electronic device 100 according to one or more embodiments of the present disclosure to obtain feature data representing object generation information by embedding multimodal information.

[0099] Refer to together Figure 5 and Figure 6 The operation of electronic device 100 to obtain feature data representing object generation information by embedding multimodal information is described.

[0100] exist Figure 5 In operation S510, the electronic device 100 converts spatial information into first feature data by performing vector embedding on the spatial information. Here, "feature data" refers to data containing n-dimensional vectors, matrices, eigenvectors, etc., with one or more dimensions. Combined with... Figure 5 Reference Figure 6 Electronic device 100 can be accessed via camera ( Figure 1 and Figure 3The processor of the electronic device 100 obtains spatial information about the real-world space from the spatial image obtained by 110, and converts the obtained spatial information into first feature data 610 by performing vector embedding on the obtained spatial information. Figure 3 The 130) can execute the instructions or program code of the spatial information extraction module 141 to extract spatial information from the spatial image, including at least one of the types or themes of real-world space, and to perform vector embedding on the extracted spatial information. The vector embedding model is a model that digitizes information in text form into vector values, and can be implemented as a known embedding model, such as one-hot encoding model, label encoding model, etc.

[0101] However, this disclosure is not limited thereto, and the processor 130 can convert spatial information into tensors such as n-dimensional vectors, matrices, or eigenvectors having one or more dimensions. Because the functionality and / or operation of the vector embedding models included in the gesture information extraction module 142, the speech information extraction module 143, and the guidance information extraction module 144 are the same as those included in the spatial information extraction module 141, the description already provided above is omitted below. Figure 6 In the embodiments of this disclosure shown, processor 130 may execute a vector embedding module to obtain first feature data 610 by embedding the type of real-world space and the topic of real-world space included in the spatial information into vector values, respectively. The first feature data 610 may include first spatial data 611 obtained by converting information about the type of real-world space into vector values, and second spatial data 612 obtained by converting information about the topic of real-world space into vector values.

[0102] exist Figure 5 In operation S520, the electronic device 100 converts the object characteristic information obtained from the gesture input into second feature data by performing vector embedding on the object characteristic information. Combined with... Figure 5 Reference Figure 6 The processor 130 of the electronic device 100 can execute instructions or program code of the gesture information extraction module 142 to recognize the user's gesture input from a hand image, extract object characteristic information from the gesture input including at least one of the position, size, or shape of an object according to the user's intention, and perform vector embedding on the extracted object characteristic information. Figure 6In the embodiments of this disclosure shown, processor 130 may execute a vector embedding module to obtain second feature data 620 by embedding the position, size, and shape of an object included in object characteristic information, respectively. The second feature data 620 may include first object characteristic data 621 obtained by converting information about the object's position into vector values, second object characteristic data 622 obtained by converting information about the object's size into vector values, and third object characteristic data 623 obtained by converting information about the object's shape into vector values.

[0103] exist Figure 5 In operation S530, the electronic device 100 converts object characteristic information into third feature data by performing vector embedding on the object characteristic information obtained from the voice input. Combined with... Figure 5 Reference Figure 6 The processor 130 of the electronic device 100 can execute instructions or program code of the voice information extraction module 143 to extract information from the microphone ( Figure 1 and Figure 3 The 120) speech input is used to extract object characteristic information, including at least one of the object's type, shape, or theme based on the user's intent, and vector embedding is performed on the extracted object characteristic information. Figure 6 In the embodiments of this disclosure shown, processor 130 may execute a vector embedding module to obtain third feature data 630 by embedding the type, shape, and theme of an object included in object characteristic information, respectively. The third feature data 630 may include first object characteristic data 631 obtained by converting information about the object's type into vector values, second object characteristic data 632 obtained by converting information about the object's shape into vector values, and third object characteristic data 633 obtained by converting information about the object's theme into vector values.

[0104] exist Figure 5 In operation S540, the electronic device 100 converts object characteristic information into fourth feature data by performing vector embedding on the object characteristic information obtained from the 2D guide image. Combined with... Figure 5 Reference Figure 6 The processor 130 of the electronic device 100 can execute instructions or program code of the guidance information extraction module 144 to extract object characteristic information, including at least one of the object's type, shape, and subject, from the 2D guidance image, and perform vector embedding on the extracted object characteristic information. Figure 6In the embodiments of this disclosure shown, processor 130 may execute a vector embedding module to obtain fourth feature data 640 by embedding the type, shape, and theme of an object included in object characteristic information, respectively. The fourth feature data 640 may include first object characteristic data 641 obtained by converting information about the object's type into vector values, second object characteristic data 642 obtained by converting information about the object's shape into vector values, and third object characteristic data 643 obtained by converting information about the object's theme into vector values.

[0105] Figure 5 Operations S510 to S540 shown can be performed simultaneously, but are not limited thereto. In one or more embodiments of this disclosure, operations S510 to S540 can be performed sequentially in any order with time differences.

[0106] exist Figure 6 In this representation, the first feature data 610 is shown as an 8-dimensional (8D) vector, and the second feature data 620, the third feature data 630, and the fourth feature data 640 are all shown as 12-dimensional (12D) vectors. However, this is for ease of description, and one or more embodiments of this disclosure are not limited to this. Figure 6 The embodiments shown are illustrated. In one or more embodiments of this disclosure, the first feature data 610 to the fourth feature data 640 can all be implemented as an n-dimensional vector having one or more dimensions.

[0107] exist Figure 5 In operation S550, the electronic device 100 obtains a vector representing object generation information based on the first feature data 610 to the fourth feature data 640. Combined with... Figure 5 Reference Figure 6 The processor 130 of the electronic device 100 can input the first feature data 610, the second feature data 620, the third feature data 630, and the fourth feature data 640 into the information mixing module 145, and execute the instructions or program code of the information mixing module 145 to obtain mixed feature data 650 by concatenating the first feature data 610, the second feature data 620, the third feature data 630, and the fourth feature data 640. In one or more embodiments of this disclosure, the processor 130 can obtain mixed feature data 650 by using MLP technology to mix the first feature data 610 to the fourth feature data 640. However, this disclosure is not limited to this; the processor 130 can obtain mixed feature data 650 representing object generation information by inputting the first feature data 610 to the fourth feature data 640 into a CNN or RNN.

[0108] exist Figure 6In the embodiments of this disclosure shown, the mixed feature data 650 may consist of an n-dimensional vector, a matrix, or an eigenvector. In one or more embodiments of this disclosure, the mixed feature data 650 may be a vector in which the vector values ​​of the first feature data 610, the second feature data 620, the third feature data 630, and the fourth feature data 640 are arranged in a column according to the results of the MLP.

[0109] Electronic device 100 can input mixed feature data 650, including data values ​​(e.g., vector values) related to object generation information, into a generative AI model. Figure 1 and Figure 3 (146) and uses generative AI model 146 to perform inference to generate 3D virtual objects based on object generation information.

[0110] Figure 7 This is a diagram illustrating the operation of an electronic device 100 according to one or more embodiments of the present disclosure, which assigns random values ​​to missing vector values ​​in a vector 700 representing object generation information.

[0111] Reference Figure 7 Vector 700 may contain vector values, wherein object generation information, including spatial information and multiple object characteristic information, has been converted into said vector values ​​through vector embedding. In one or more embodiments of this disclosure, vector 700 may be generated by processing first feature data (via MLP) Figure 6 610), second feature data ( Figure 6 620), third feature data ( Figure 6 630) and the fourth feature data ( Figure 6 The vector obtained by concatenating 640 of the first feature data ( Figure 6 The 610) is obtained by performing vector embedding on spatial information extracted from spatial images, the second feature data ( Figure 6 The 620 is obtained by performing vector embedding on object feature information extracted from gesture input, the third feature data ( Figure 6 The 630th feature was obtained by vector embedding of object feature information extracted from speech input, the fourth feature data ( Figure 6 The 640) was obtained by performing vector embedding on object feature information extracted from 2D guided images.

[0112] Although vector 700 contains vector values, it may have some missing vector values ​​in the first element 701 and the second element 702. Figure 7 In the embodiments of this disclosure shown, the first element 701 and the second element 702 may not have vector values. In one or more embodiments of this disclosure, the processor of the electronic device 100 (Figure 3 The processor 130 can randomly generate random values ​​711 and 712, and assign the randomly generated random values ​​(i.e., the first random value 711 and the second random value 712) to the first element 701 and the second element 702, which have missing vector values. For example, the processor 130 can assign the first random value 711 to the first element 701 and the second random value 712 to the second element 702. By doing so, the processor 130 can obtain the reconstructed vector 710.

[0113] exist Figure 7 In the embodiments of this disclosure shown, in order to maintain the input to the generative AI model ( Figure 1 and Figure 3 Given the dimension of vector 700 (146), the electronic device 100 can fill empty elements (i.e., first element 701 and second element 702) with vector values ​​by assigning randomly generated random values ​​(i.e., first random value 711 and second random value 712) to first element 701 and second element 702 with missing vector values. Because the reconstructed vector 710 contains the first random value 711 and the second random value 712, when the reconstructed vector 710 is input to the generative AI model 146, it may generate 3D virtual objects with shapes, sizes, positions, types, themes, or atmospheres that do not correspond to the user's intent. In this case, the user can modify the object generation information.

[0114] Figure 8 This is a flowchart illustrating a method performed by an electronic device 100 to generate a 3D virtual object by modifying object generation information based on user input, according to one or more embodiments of the present disclosure.

[0115] Can be executed Figure 2 The operation S230 shown is executed afterward. Figure 8 Operations S810 to S830. During execution... Figure 8 After operating the S830, it can execute... Figure 2 The operation S250 is shown.

[0116] In operation S810, electronic device 100 receives user input for modifying object generation information. In one or more embodiments of this disclosure, electronic device 100 may receive user input for adjusting the weights applied to each of the multiple pieces of multimodal information. For example, electronic device 100 may receive user input for adjusting the values ​​of the weights assigned to each of spatial information obtained from a spatial image, object characteristic information obtained from gesture input, object characteristic information obtained from voice input, and object characteristic information obtained from a 2D guiding image.

[0117] In one or more embodiments of this disclosure, electronic device 100 may receive user input for changing at least one of the shape, size, position, color, type, or theme of an object. For example, electronic device 100 may receive gesture input for modifying the shape, size, position, etc. of an object.

[0118] In operation S820, electronic device 100 modifies object generation information based on received user input. In one or more embodiments of this disclosure, electronic device 100 may change the weights assigned to each of the spatial information obtained from the spatial image, the object characteristic information obtained from the gesture input, the object characteristic information obtained from the voice input, and the object characteristic information obtained from the 2D guided image, based on user input for adjusting the weights applied to each of the multiple pieces of multimodal information. Referring below... Figure 10 This disclosure describes in detail a specific embodiment of an electronic device 100 that modifies the weights applied to each of a plurality of multimodal messages based on user input.

[0119] In one or more embodiments of this disclosure, the electronic device 100 can modify at least one of the shape, size, position, color, type, or theme of an object included in the object generation information based on user input. (Refer to below) Figure 11 This disclosure describes in detail specific embodiments of the electronic device 100 that generates information based on user input to modify objects.

[0120] In operation S830, electronic device 100 can generate 3D virtual objects by inputting modified object generation information into a generative AI model and using the generative AI model to perform inference. Because the object generation information has been modified, the shape, size, position, color, type, or theme of the 3D virtual object obtained by generative AI model 146 can be changed.

[0121] Figure 9 This is a diagram illustrating components of an electronic device 100 for generating 3D virtual objects by modifying object generation information, according to one or more embodiments of the present disclosure.

[0122] Reference Figure 9 Multiple pieces of multimodal information are input to the information mixing module 145. In one or more embodiments of this disclosure, "multiple pieces of multimodal information" may include spatial information obtained from spatial images, object characteristic information obtained from gesture input, object characteristic information obtained from voice input, and object characteristic information obtained from 2D guide images.

[0123] The object modification module 147 receives multiple pieces of multimodal information and user modification input. In one or more embodiments of this disclosure, the user modification input may be user input for adjusting the weights applied to each piece of multimodal information, or user input for changing at least one of the object's shape, size, position, color, type, or theme. The processor of the electronic device 100 ( Figure 3 The object modification module 147 (130) can execute instructions or program code to modify object generation information based on user input. The object modification module 147 can output object modification information.

[0124] Processor 130 can execute instructions or program code of information mixing module 145 to modify object generation information based on multiple pieces of multimodal information by using object modification information, and obtain the modified object generation information. In one or more embodiments of this disclosure, processor 130 can modify object generation information by performing vector embedding on multiple pieces of multimodal information respectively via MLP concatenation and modifying the vector values ​​of the vectors obtained as concatenation results.

[0125] The processor 130 can generate a 3D virtual object based on the modified object generation information obtained as a result of modification by inputting a vector representing the modified object generation information into the generative AI model 146.

[0126] Figure 10 This is a diagram illustrating the operation of an electronic device 100 according to one or more embodiments of the present disclosure to modify object generation information by receiving user input for adjusting weights for multiple pieces of multimodal information (e.g., 1010, 1020, 1030, and 1040).

[0127] Reference Figure 10 The electronic device 100 receives user input (operation ①) for adjusting the weights of spatial information and object characteristic information obtained based on multiple pieces of multimodal information. The electronic device 100 can receive user input for respectively changing the weights applied to the multiple pieces of multimodal information (i.e., object characteristic information 1010 obtained from gesture input, spatial information 1020 obtained from spatial images, object characteristic information 1030 obtained from voice input, and object characteristic information 1040 obtained from 2D guided images). Figure 10In the embodiments of this disclosure shown, electronic device 100 is implemented as a mobile device including a touchscreen, such as a smartphone or tablet PC, and can receive touch input from a user for changing weights. However, this disclosure is not limited thereto; when electronic device 100 is implemented as an AR device such as AR glasses or an HMD device, electronic device 100 can receive finger input from a user for changing weights. In one or more embodiments of this disclosure, electronic device 100 may include an eye-tracking sensor and can receive user input for modifying weights applied to multiple pieces of multimodal information based on the user's gaze direction obtained using the eye-tracking sensor.

[0128] Electronic device 100 adjusts the weights applied to multiple pieces of multimodal information respectively (operation ②). In one or more embodiments of this disclosure, the processor of electronic device 100 ( Figure 3 (130) can change the weights of the object characteristic information 1010 obtained from gesture input, the spatial information 1020 obtained from spatial image, the object characteristic information 1030 obtained from voice input, and the object characteristic information 1040 obtained from 2D guided image, respectively, based on user input. Figure 10 In the embodiments of this disclosure shown, object characteristic information 1010 obtained from gesture input may include information about the size, position, and shape of the object, and may consist of vector values ​​transformed via vector embedding. A first weight α may be applied to the object characteristic information 1010 via user input. Spatial information 1020 obtained from a spatial image may include information about the type and subject of the real-world space, and may consist of vector values ​​transformed via vector embedding. A second weight β may be applied to the spatial information 1020 via user input. Object characteristic information 1030 obtained from voice input may include information about the shape, type, and subject of the object, and may consist of vector values ​​transformed via vector embedding. A third weight γ may be applied to the object characteristic information 1030 via user input. Object characteristic information 1040 obtained from a 2D guided image may include information about the shape, type, and subject of the object, and may consist of vector values ​​transformed via vector embedding. A fourth weight δ may be applied to the object characteristic information 1040 via user input.

[0129] Electronic device 100 modifies object generation information (operation ③) by adjusting the weights applied to multiple pieces of multimodal information. Figure 10In the embodiments of this disclosure shown, the processor 130 of the electronic device 100 can modify object generation information by applying a first weight α to object characteristic information 1010 obtained from gesture input, applying a second weight β to spatial information 1020 obtained from spatial image, applying a third weight γ to object characteristic information 1030 obtained from voice input, and applying a fourth weight δ to object characteristic information 1040 obtained from 2D guide image.

[0130] The electronic device 100 obtains modified object generation information 1050 (operation ④) by mixing spatial information with adjusted weights and multiple object characteristic information. In one or more embodiments of this disclosure, the processor 130 of the electronic device 100 can obtain modified object generation information 1050 by concatenating the vector values ​​of object characteristic information 1010 with applied first weight α, spatial information 1020 with applied second weight β, object characteristic information 1030 with applied third weight γ, and object characteristic information with applied fourth weight δ via an MLP. However, this disclosure is not limited to this; the processor 130 can obtain modified object generation information 1050 by inputting the vector values ​​of object characteristic information 1010 with applied first weight α, spatial information 1020 with applied second weight β, object characteristic information 1030 with applied third weight γ, and object characteristic information with applied fourth weight δ into a CNN or RNN.

[0131] The processor 130 of the electronic device 100 can input the modified object generation information 1050 into the generative AI model. Figure 1 and Figure 3 (146), and generates 3D virtual objects modified according to user intent by reasoning using generative AI model 146.

[0132] Figure 11 This is a diagram illustrating the operation of an electronic device 100 according to one or more embodiments of the present disclosure to modify object generation information by receiving user input for modifying the shape, size, and position of an object.

[0133] Reference Figure 11 The electronic device 100 receives user input (operation ①) for changing at least one of the shape, size, position, color, type, or theme of an object. In one or more embodiments of this disclosure, the electronic device 100 may display a 3D virtual object 1100 and receive gesture input 1110 from the user for changing at least one of the shape, size, position, color, type, or theme of the displayed 3D virtual object. Figure 11In the embodiments of this disclosure shown, electronic device 100 may receive gesture input 1110 from a user drawing a tilted "∩" shape. However, this disclosure is not limited thereto, and in one or more embodiments of this disclosure, when electronic device 100 is implemented as a mobile device (such as a smartphone or tablet PC) including a touchscreen, electronic device 100 may receive user touch input for changing at least one of the shape, size, position, color, type, or theme of a 3D virtual object 1100 displayed on the touchscreen. In one or more embodiments of this disclosure, electronic device 100 may input a spatial image obtained by capturing space other than the space used to generate the 3D virtual object 1100.

[0134] Electronic device 100 modifies at least one of spatial information or object characteristic information based on user input (operation ②). Figure 11 In the embodiments of this disclosure shown, object generation information 1120 may include object characteristic information 1121 obtained from gesture input, spatial information 1122 obtained from spatial image, object characteristic information 1123 obtained from voice input, and object characteristic information 1124 obtained from 2D guide image. In one or more embodiments of this disclosure, the processor of electronic device 100 ( Figure 3 Processor 130 can execute instructions or program code from gesture information extraction module 142 to obtain object modification information 1130 by performing vector embedding on gesture input 1110 received from the user. Processor 130 can replace object characteristic information 1121 included in object generation information 1120 with the obtained object modification information 1130.

[0135] Electronic device 100 obtains modified object generation information 1140 (operation ③) by mixing modified spatial information and object characteristic information. In one or more embodiments of this disclosure, processor 130 of electronic device 100 can obtain modified object generation information 1140 by concatenating object modification information 1130, spatial information 1122, and multiple object characteristic information 1123 and 1124 via MLP. Modified object generation information 1140 may be a vector including vector values ​​of object modification information 1130, vector values ​​of spatial information 1122, and vector values ​​of multiple object characteristic information 1123 and 1124. However, this disclosure is not limited to this, and processor 130 can obtain modified object generation information 1140 by inputting the vector values ​​representing object modification information 1130, spatial information 1122, and multiple object characteristic information 1123 and 1124 into CNN or RNN.

[0136] Electronic device 100 generates 3D virtual object 1150 by using modified object generation information 1140 (operation ④). In one or more embodiments of this disclosure, processor 130 of electronic device 100 can generate 3D virtual object 1150 by inputting modified object generation information 1140 into generative AI model 146 and performing inference using generative AI model 146.

[0137] exist Figure 11 In the embodiments of this disclosure shown, the previously generated 3D virtual object 1100 is a table including a circular top plate and a circular bottom plate and having a leg connecting the top plate and the bottom plate, and can be modified into a 3D virtual object 1150 based on gesture input 1110 received from the user, which is a "∩"-shaped table with a top plate and two legs.

[0138] exist Figures 8 to 11 In the embodiments of this disclosure shown, electronic device 100 can receive user input for adjusting weights applied to multiple pieces of multimodal information, or for changing at least one of the shape, size, position, color, type, or theme of an object, and modify the 3D virtual object based on the received user input. According to one or more embodiments of this disclosure, by allowing users to further modify the initially generated 3D virtual object, electronic device 100 not only enables users to interact with AR services but also enables multiple users to share the experience of AR services.

[0139] Figure 12a This is a diagram illustrating the operation of an electronic device 100 according to one or more embodiments of the present disclosure to generate a 3D virtual object 1250a based on multiple pieces of multimodal information (e.g., 1210, 1220, 1230 and 1240) and display the generated 3D virtual object 1250a.

[0140] exist Figure 12a In the embodiments of this disclosure shown, the electronic device 100 can obtain spatial information such as "the brown hue of a café" from a spatial image 1210, object characteristic information from a voice input 1220 "make a table suitable for a café", object characteristic information about the object's shape, size, position, etc. from a gesture input 1230 for drawing the table shape, and object characteristic information about the object's shape, size, type, color, theme, etc. from a 2D guided image 1240 including a white table with drawers. The processor of the electronic device 100 ( Figure 3 (130) can convert multiple pieces of information into vector values ​​by performing vector embedding, and then input the vector values ​​as the result of the embedding into a generative AI model ( Figure 1 and Figure 3(146) to generate 3D virtual object 1250a. Figure 12a In the embodiments of this disclosure shown, the 3D virtual object 1250a may be a 3D graphic of a table having a shape (e.g., a wooden rounded shape) and color (e.g., brown) that matches the hue, theme, or atmosphere of a real-world space (e.g., a coffee shop).

[0141] The electronic device 100 can display the generated 3D virtual object 1250a by overlaying it onto the real-world space 1200a. Although Figure 12a An electronic device 100, implemented as a mobile device, is shown displaying a real-world space 1200a and a 3D virtual object 1250a on a display, but this disclosure is not limited thereto. In one or more embodiments of this disclosure, the electronic device 100 may be implemented as an AR device, such as AR glasses, and display the real-world space 1200a transmitted through an optical lens and the 3D virtual object 1250a projected through a waveguide in an AR or mixed reality format.

[0142] Figure 12b This is a diagram illustrating the operation of an electronic device 100 according to one or more embodiments of the present disclosure to generate a 3D virtual object 1250b based on multiple pieces of multimodal information (e.g., 1210, 1220, 1230 and 1240) and display the generated 3D virtual object 1250b.

[0143] Aside from the fact that the real-world space 1200b is a gray-toned office rather than a café, and that the electronic device 100 receives voice input 1220 stating "Make a desk suitable for this office," Figure 12b The illustrated embodiments and Figure 12a The embodiments shown are the same as those in this disclosure, therefore the description already provided above is omitted.

[0144] refer to Figure 12b The electronic device 100 can input a mixture of multimodal information, including spatial image 1210, voice input 1220, gesture input 1230, and 2D guide image 1240, into a generative AI model. Figure 1 and Figure 3 (146) and perform inference on their mixture to generate 3D virtual objects 1250b. Figure 12b In the embodiments of this disclosure shown, the 3D virtual object 1250b may be a 3D graphic of a table having a shape (e.g., a steel right-angled shape) and color (e.g., gray) that matches the gray tone, theme, or atmosphere of the real-world space 1200b (e.g., an office). The electronic device 100 can display the generated 3D virtual object 1250b by overlaying it onto the real-world space 1200b.

[0145] Figure 12c This is a diagram illustrating the operation of an electronic device 100 according to one or more embodiments of the present disclosure to generate a 3D virtual object 1250c based on multiple pieces of multimodal information (e.g., 1210, 1220, 1230 and 1240) and display the generated 3D virtual object 1250c.

[0146] In addition to the 2D guide image 1240, which includes an image of a table made of wood with a brown hue, Figure 12c The embodiments of this disclosure shown are similar to Figure 12b The embodiments shown are the same as those in this disclosure, so the description already provided above is omitted.

[0147] Reference Figure 12c The electronic device 100 can input a mixture of multimodal information, including spatial image 1210, voice input 1220, gesture input 1230, and 2D guide image 1240, into a generative AI model. Figure 1 and Figure 3 146) and perform reasoning on their mixture to generate 3D virtual objects 1250c. In Figure 12c In the embodiments of this disclosure shown, the 3D virtual object 1250c may be a 3D graphic of a table having a shape (e.g., a wooden right-angled shape) and color (e.g., brown) that matches the hue, theme, or atmosphere of the real-world space 1200c (e.g., an office). The electronic device 100 can display the generated 3D virtual object 1250c by overlaying it onto the real-world space 1200c.

[0148] Figure 12d This is a diagram illustrating the operation of an electronic device 100 according to one or more embodiments of the present disclosure to generate a 3D virtual object 1250d based on multiple pieces of multimodal information (e.g., 1210, 1220, 1230 and 1240) and display the generated 3D virtual object 1250d.

[0149] In addition to the electronic device 100 receiving voice input 1220 "Make a bookcase suitable for this office", Figure 12d The illustrated embodiments and Figure 12c The embodiments shown are the same as those in this disclosure, therefore the description already provided above is omitted.

[0150] Reference Figure 12d The electronic device 100 can input a mixture of multimodal information, including spatial image 1210, voice input 1220, gesture input 1230, and 2D guide image 1240, into a generative AI model. Figure 1 and Figure 3(146) and perform inference on their mixture to generate 3D virtual objects 1250d. In Figure 12d In the embodiments of this disclosure shown, the 3D virtual object 1250d may be a 3D graphic of a bookshelf having a shape (e.g., a wooden right-angled shape) and color (e.g., brown) that matches the hue, theme, or atmosphere of the real-world space 1200d (e.g., an office). The electronic device 100 can display the generated 3D virtual object 1250d by overlaying it onto the real-world space 1200d.

[0151] Figure 12e This is a diagram illustrating the operation of an electronic device 100 according to one or more embodiments of the present disclosure to generate a 3D virtual object 1250e based on multiple pieces of multimodal information (e.g., 1210, 1220, 1230 and 1240) and display the generated 3D virtual object 1250e.

[0152] In addition to the user's gesture input 1230, which represents drawing a rectangular top panel and four legs, Figure 12e The embodiments of this disclosure shown are similar to Figure 12d The embodiments shown are the same as those in this disclosure, so the description already provided above is omitted.

[0153] Reference Figure 12e The electronic device 100 can input a mixture of multimodal information, including spatial image 1210, voice input 1220, gesture input 1230, and 2D guide image 1240, into a generative AI model. Figure 1 and Figure 3 146) and perform reasoning on their mixture to generate 3D virtual objects 1250e. Figure 12e In the embodiments of this disclosure shown, the 3D virtual object 1250c may be a 3D graphic of a table having a shape (e.g., a shape with a rectangular top panel and four legs at right angles) and color (e.g., brown) that matches the hue, theme, or atmosphere of the real-world space 1200e (e.g., an office). The electronic device 100 can display the generated 3D virtual object 1250e by overlaying it onto the real-world space 1200e.

[0154] exist Figures 12a to 12e In the embodiments of this disclosure shown, electronic device 100 can modify each of the 3D virtual objects 1250a, 1250b, 1250c, 1250d, and 1250e based on user input received for adjusting weights applied to multiple pieces of multimodal information or for changing at least one of the shape, size, position, color, type, or theme of an object.

[0155] According to one aspect of this disclosure, an electronic device 100 is provided for generating virtual objects based on multimodal information. According to one or more embodiments of this disclosure, the electronic device 100 may include: a camera 110 configured to acquire images; a memory 140 storing at least one instruction; and at least one processor 130 configured to execute at least one instruction. The at least one processor 130 may be configured to acquire spatial information about real-world space based on the images acquired by the camera 110. The at least one processor 130 may be configured to acquire user input based on the images acquired by the camera 110. The at least one processor 130 may be configured to acquire multiple object characteristic information from the acquired user input. The at least one processor 130 may be configured to acquire object generation information for generating virtual objects based on the acquired spatial information and the multiple object characteristic information. The at least one processor 130 may be configured to generate virtual objects for the object generation information by inputting the acquired object generation information into a generative AI model trained to generate 3D virtual objects based on information about space and objects.

[0156] In one or more embodiments of this disclosure, camera 110 may include a first camera 112 configured to acquire spatial images of real-world space by capturing images of real-world space. At least one processor 130 may be configured to obtain spatial information from the spatial images acquired by the first camera 112 regarding at least one of the following: type, category, color, theme, or atmosphere of the real-world space.

[0157] In one or more embodiments of this disclosure, camera 110 may include a second camera 114 configured to acquire a hand image by capturing an image of a user's hand. At least one processor 130 may be configured to recognize gesture input from the user in the hand image acquired by the second camera 114, and extract object characteristic information about at least one of the object's shape, position, or size from the recognized user gesture input.

[0158] In one or more embodiments of this disclosure, the second camera 114 may be configured as a depth camera including at least one of a ToF camera, a stereo vision camera, or a LiDAR sensor, and obtain a depth image by capturing an image of the user's hand. At least one processor 130 may be configured to recognize gesture input from the user in the depth image obtained by the second camera 114.

[0159] In one or more embodiments of this disclosure, the electronic device 100 may further include a touchscreen for receiving touch input from a user. At least one processor 130 may be configured to recognize gesture input from the touch input received via the touchscreen and extract object characteristic information from the recognized gesture input regarding at least one of the shape, position, or size of an object.

[0160] In one or more embodiments of this disclosure, the electronic device 100 may further include a microphone 120 for receiving voice input from a user. At least one processor 130 may be configured to obtain a voice signal from the voice input received through the microphone 120. At least one processor 130 may be configured to convert the voice signal into text and analyze the text using an NLU model to obtain object characteristic information including at least one of the following: object type, shape, color, or theme.

[0161] In one or more embodiments of this disclosure, at least one processor 130 may be configured to obtain a 2D guide image and extract object characteristic information from the 2D guide image, including at least one of the following: object type, shape, color, or theme.

[0162] In one or more embodiments of this disclosure, at least one processor 130 may be configured to convert spatial information into first feature data by performing vector embedding on the spatial information. At least one processor 130 may be configured to convert object characteristic information into second feature data by performing vector embedding on object characteristic information obtained from gesture input. At least one processor 130 may be configured to convert object characteristic information into third feature data by performing vector embedding on object characteristic information obtained from voice input. At least one processor 130 may be configured to convert object characteristic information into fourth feature data by performing vector embedding on object characteristic information obtained from a 2D guide image. At least one processor 130 may be configured to obtain feature data representing object generation information based on the first to fourth feature data.

[0163] In one or more embodiments of this disclosure, at least one processor 130 may be configured to assign a randomly generated random value as the missing vector value when a vector value representing spatial information or object characteristic information is missing in a feature vector including first feature data to fourth feature data.

[0164] In one or more embodiments of this disclosure, at least one processor 130 may be configured to modify object generation information based on user input.

[0165] In one or more embodiments of this disclosure, at least one processor 130 may be configured to modify object generation information by adjusting weight values ​​assigned to each of spatial information obtained from a spatial image, object feature information obtained from gesture input, object feature information obtained from voice input, and object feature information obtained from a 2D guide image based on user input.

[0166] In one or more embodiments of this disclosure, at least one processor 130 may be configured to modify object generation information based on user input by changing at least one of shape, size, position, color, type, or theme.

[0167] According to another aspect of this disclosure, a method for generating virtual objects based on multimodal information, performed by an electronic device 100, is provided. In one or more embodiments of this disclosure, the method performed by the electronic device 100 may include acquiring an image via a camera 110 (S210). The method performed by the electronic device 100 may include acquiring user input based on the image acquired via the camera 110 (S220). The method performed by the electronic device 100 may include acquiring spatial information about real-world space based on the acquired image (S230). The method performed by the electronic device 100 may include acquiring multiple object characteristic information from the acquired user input (S240). The method performed by the electronic device 100 may include acquiring object generation information for generating virtual objects based on the acquired spatial information and the multiple object characteristic information. The method performed by the electronic device 100 may include generating virtual objects for the acquired object generation information by inputting the object generation information into a generative AI model trained to generate 3D virtual objects based on information about space and objects.

[0168] In one or more embodiments of this disclosure, the step of obtaining spatial information (S230) may include: obtaining a spatial image by using a first camera 112, and obtaining spatial information about at least one of the following: type, category, color, theme, or atmosphere of the real-world space from the obtained spatial image, wherein the first camera 112 is configured to obtain a spatial image of the real-world space by capturing an image of the real-world space.

[0169] In one or more embodiments of this disclosure, the step of obtaining object characteristic information (S240) may include: obtaining a hand image via a second camera 114, and recognizing a gesture input from a user in the obtained hand image, wherein the second camera 114 is configured to obtain the hand image by capturing an image of the user's hand. The step of obtaining multiple pieces of object characteristic information (S240) may include extracting object characteristic information about at least one of the shape, position, or size of an object from the recognized gesture input.

[0170] In one or more embodiments of this disclosure, the second camera 114 may be configured as a depth camera including at least one of a ToF camera, a stereo vision camera, or a LiDAR sensor. When acquiring a hand image, the electronic device 100 can obtain a depth image by capturing an image of the user's hand. When recognizing gesture input from the user, the electronic device 100 can recognize the gesture input from the user in the depth image acquired by the second camera 114.

[0171] In one or more embodiments of this disclosure, the step of obtaining object characteristic information (S240) may include: receiving touch input from a user via a touchscreen; and recognizing gesture input from the touch input received via the touchscreen. The step of obtaining multiple pieces of object characteristic information (S240) may include extracting object characteristic information about at least one of the object's shape, position, or size from the recognized user gesture input.

[0172] In one or more embodiments of this disclosure, the step of obtaining multiple pieces of object characteristic information (S240) may include: receiving voice input from a user via microphone 120, obtaining a voice signal from the received voice input, and converting the voice signal into text. The step of obtaining multiple pieces of object characteristic information (S240) may include analyzing the text using an NLU model to obtain object characteristic information including at least one of the following: object type, shape, color, or theme.

[0173] In one or more embodiments of this disclosure, the method performed by the electronic device 100 may further include obtaining a 2D guide image. The step of obtaining multiple object characteristic information (S240) may include extracting object characteristic information from the 2D guide image, including at least one of the following: object type, shape, color, or theme.

[0174] In one or more embodiments of this disclosure, the step of obtaining multiple object characteristic information (S240) may include converting spatial information into first feature data by performing vector embedding on the spatial information. The step of obtaining multiple object characteristic information (S240) may include converting object characteristic information into second feature data by performing vector embedding on object characteristic information obtained from gesture input. The step of obtaining multiple object characteristic information (S240) may include converting object characteristic information into third feature data by performing vector embedding on object characteristic information obtained from voice input. The step of obtaining object characteristic information (S240) may include converting object characteristic information into fourth feature data by performing vector embedding on object characteristic information obtained from a 2D guide image. The step of obtaining object generation information (S250) may include obtaining feature data representing object generation information based on the first to fourth feature data.

[0175] In one or more embodiments of this disclosure, the method performed by the electronic device 100 may further include: receiving user input for modifying object generation information (S810); and modifying the object generation information based on the received user input (S820).

[0176] In one or more embodiments of this disclosure, when modifying object generation information (S820), the electronic device 100 can modify the object generation information by adjusting the weight values ​​assigned to each of the spatial information obtained from the spatial image, the object characteristic information obtained from the gesture input, the object characteristic information obtained from the voice input, and the object characteristic information obtained from the 2D guide image based on user input.

[0177] In one or more embodiments of this disclosure, when modifying object generation information (S820), electronic device 100 can modify the object generation information based on user input by changing at least one of shape, size, position, color, type, or theme.

[0178] According to another aspect of this disclosure, a computer program product including a computer-readable storage medium is provided. The computer-readable storage medium may include instructions readable by an electronic device 100 for performing the following operations: acquiring an image via a camera 110; acquiring user input based on the image acquired via the camera 110; acquiring spatial information about real-world space based on the acquired image; acquiring multiple object characteristic information from the acquired user input; acquiring object generation information for generating virtual objects based on the acquired spatial information and the multiple object characteristic information; and generating virtual objects for the acquired object generation information by inputting the object generation information into a generative AI model, said AI model being trained to generate 3D virtual objects based on information about space and objects.

[0179] The program executed by the electronic device 100 described in this disclosure can be implemented as a hardware component, a software component, and / or a combination of hardware and software components. The program can be executed by any system capable of executing computer-readable instructions.

[0180] Software may include one or more of computer programs, code segments, instructions, or computer programs, code segments, and instructions, and configures processing means to operate as needed or to independently or jointly instruct processing means.

[0181] Software can be implemented as a computer program that includes instructions stored in a computer-readable storage medium. Examples of computer-readable recording media include magnetic storage media (e.g., ROM, RAM, floppy disks, hard disks, etc.), optical recording media (e.g., optical disc ROM (CD-ROM), digital versatile disc (DVD), etc.). Computer-readable recording media can be distributed across computer systems connected via a network, allowing computer-readable code to be stored and executed in a distributed manner. The medium can be read by a computer, stored in memory, and executed by a processor.

[0182] Computer-readable storage media may be provided in the form of non-transitory storage media. In this respect, the term "non-transitory" means only that the storage medium does not include signals and is a tangible device, and the term does not distinguish between data that is stored semi-permanently in the storage medium and data that is temporarily stored in the storage medium. For example, a "non-transitory storage medium" may include a buffer for temporarily storing data.

[0183] Furthermore, programs according to embodiments of this disclosure may be included in a computer program product when provided. The computer program product may be traded as a product between a seller and a buyer.

[0184] Computer program products may include software programs and computer-readable storage media on which the software programs are stored. For example, a computer program product may include a product in the form of a software program (e.g., a downloadable application) distributed electronically by the manufacturer of electronic device 100 or through an electronic marketplace (e.g., Samsung Galaxy Store™). For such electronic distribution, at least a portion of the software program may be stored in the storage medium or may be temporarily generated. In this case, the storage medium may be a server of the manufacturer of electronic device 100, a server of the electronic marketplace, or a relay server used for temporary storage of the software program.

[0185] In a system comprising electronic device 100 and / or a server, the computer program product may include the storage medium of the server or the storage medium of electronic device 100. Optionally, in the presence of a third device (e.g., a wearable device) communicatively connected to electronic device 100, the computer program product may include the storage medium of the third device. Optionally, the computer program product may include the software program itself transmitted from electronic device 100 to the third device or from the third device to electronic device 100.

[0186] In this configuration, one of the electronic device 100 and the third device may execute a computer program product to perform the method according to an embodiment of the present disclosure. Optionally, at least one of the electronic device 100 or the third device may execute a computer program product to perform the method according to an embodiment of the present disclosure in a distributed manner.

[0187] For example, electronic device 100 can execute functions stored in memory ( Figure 3 The computer program product in (140) is used to control another electronic device (e.g., a wearable device) communicatively connected to electronic device 100 to perform the method according to embodiments of the present disclosure.

[0188] In another example, the third device may execute a computer program product to control an electronic device communicatively connected to the third device to perform a method according to one or more embodiments of the present disclosure.

[0189] When a third device executes a computer program product, the third device can download the computer program product from the electronic device 100 and execute the downloaded computer program product. Optionally, the third device can execute a computer program product pre-loaded therein to perform a method according to an embodiment of this disclosure.

[0190] While embodiments of this disclosure have been described above with reference to limited examples and accompanying drawings, those skilled in the art will understand that various modifications and changes in form and detail can be made from the above description. For example, sufficient effects may be achieved even if the above-described techniques are performed in a different order than described above, and / or the aforementioned components, such as computer systems or modules, are coupled or combined in a different form and mode than described above, or are replaced or supplemented by other components or their equivalents.

Claims

1. An electronic device comprising: monitor; The camera is configured to acquire images; Memory, storing at least one instruction; and At least one processor is configured to execute the at least one instruction to cause the electronic device to: Spatial information about the real-world space is obtained based on the images acquired through the camera; User input is obtained based on the image acquired through the camera; Obtain object characteristic information from the user input; Based on the spatial information and the object characteristic information, object generation information for generating virtual objects is obtained; The virtual object is generated by inputting the object generation information into a generative artificial intelligence (AI) model, wherein the generative AI model is trained to generate a three-dimensional (3D) virtual object based on information about space and object. and Control the display to show the virtual object.

2. The electronic device according to claim 1, in, The camera includes a first camera configured to acquire a spatial image of the real-world space by capturing images of the real-world space, and The at least one processor is further configured to execute the at least one instruction to cause the electronic device to: Spatial information is obtained from the spatial image acquired through the first camera regarding at least one of the following: type, category, color, theme, or atmosphere of the real-world space.

3. The electronic device according to claim 1, in, The camera includes a second camera configured to acquire a hand image by capturing an image of the user's hand, and The at least one processor is further configured to execute the at least one instruction to cause the electronic device to: Recognize gesture input from the user in the hand image obtained through the second camera; and Extract object characteristic information about at least one of the object's shape, position, or size from the gesture input.

4. The electronic device according to claim 3, in, The second camera is configured as a depth camera including at least one of a time-of-flight (ToF) camera, a stereo vision camera, or a light detection and ranging (LiDAR) sensor, and is configured to acquire a depth image by capturing an image of the user's hand. The at least one processor is further configured to execute the at least one instruction to cause the electronic device to: The gesture input from the user is identified in the depth image obtained through the second camera.

5. The electronic device according to claim 1, further comprising: The touchscreen is configured to receive touch input from the user. The at least one processor is further configured to execute the at least one instruction to cause the electronic device to: Recognize gesture input from the touch input received via the touchscreen; and Extract object characteristic information about at least one of the object's shape, position, or size from the gesture input.

6. The electronic device according to claim 1, further comprising: The microphone is configured to receive voice input from the user. The at least one processor is further configured to execute the at least one instruction to cause the electronic device to: A voice signal is obtained from the voice input received through the microphone; Convert the speech signal into text; and The text is analyzed using a Natural Language Understanding (NLU) model to extract object characteristic information, including at least one of the object's type, shape, color, or theme.

7. The electronic device according to claim 1, wherein, The at least one processor is also configured to execute the at least one instruction to cause the electronic device to: Obtain a two-dimensional (2D) guide image; and Extract object characteristic information, including at least one of the object's type, shape, color, or theme, from the 2D guide image.

8. The electronic device according to claim 1, wherein, The object characteristic information includes at least one of the following: first object characteristic information obtained from spatial information, second object characteristic information obtained from gesture input, third characteristic information obtained from voice input, or fourth characteristic information obtained from 2D guide images. The at least one processor is further configured to execute the at least one instruction to cause the electronic device to: By performing vector embedding on the spatial information, the first object characteristic information is converted into first feature data; By performing vector embedding on the second object characteristic information obtained from the gesture input, the second object characteristic information is converted into second feature data; By performing vector embedding on the third object characteristic information obtained from the speech input, the third object characteristic information is converted into third feature data. The fourth object characteristic information is converted into fourth feature data by performing vector embedding on the fourth object characteristic information obtained from the 2D guide image; and Feature data representing the object generation information is obtained based on the first feature data to the fourth feature data.

9. The electronic device according to claim 8, wherein, The at least one processor is further configured to execute the at least one instruction to cause the electronic device to: The object generation information is modified based on user input.

10. The electronic device according to claim 9, wherein, The at least one processor is further configured to execute the at least one instruction to cause the electronic device to: The object generation information is modified by adjusting the weight values ​​of each of the first object characteristic information, the second object characteristic information extracted from the gesture input, the third object characteristic information obtained from the voice input, and the fourth object characteristic information obtained from the 2D guide image based on the user input.

11. A method for generating virtual objects, performed by an electronic device, the method comprising: Images are acquired through a camera; User input is obtained based on the image acquired through the camera; Based on the image, spatial information about the real-world space is obtained; Obtain object characteristic information from the user input; Based on the spatial information and the object characteristic information, obtain object generation information for generating the virtual object; The virtual object is generated by inputting the object generation information into a generative artificial intelligence (AI) model, wherein the generative AI model is trained to generate a three-dimensional (3D) virtual object based on information about space and object. and The virtual object is displayed on a monitor.

12. The method according to claim 11, wherein, The object characteristic information includes at least one of the following: first object characteristic information obtained from spatial information, second object characteristic information obtained from gesture input, third characteristic information obtained from voice input, or fourth characteristic information obtained from 2D guide images. The steps for generating the object generation information include: By performing vector embedding on the spatial information, the first object characteristic information is converted into first feature data; By performing vector embedding on the second object characteristic information obtained from the gesture input, the second object characteristic information is converted into second feature data; By performing vector embedding on the third object characteristic information obtained from the speech input, the third object characteristic information is converted into third feature data; The fourth object characteristic information is converted into fourth feature data by performing vector embedding on the fourth object characteristic information obtained from the 2D guide image; and Feature data representing the object generation information is obtained based on the first feature data to the fourth feature data.

13. The method of claim 12, further comprising: Receive user input for modifying the object generation information; and Modify the object generation information based on the user input.

14. The method according to claim 13, wherein, The step of modifying the object generation information includes: modifying the object generation information by adjusting the weight values ​​of each of the first object characteristic information, the second object characteristic information obtained from the gesture input, the third object characteristic information obtained from the voice input, and the fourth object characteristic information obtained from the 2D guide image based on the user input.

15. A computer program product comprising a computer-readable storage medium, wherein, The computer-readable storage medium includes instructions that can be read by an electronic device to perform the following operations: Images are acquired through a camera; User input is obtained based on the image acquired through the camera; Based on the image, spatial information about the real-world space is obtained; Obtain object characteristic information from the user input; Based on the spatial information and the object characteristic information, object generation information for generating virtual objects is obtained; The virtual object is generated by inputting the object generation information into a generative artificial intelligence (AI) model, wherein the generative AI model is trained to generate a three-dimensional (3D) virtual object based on information about space and object. and The virtual object is displayed on a monitor.

Images