Method and electronic device for providing information related to arrangement of objects in space

Abstract

A method for providing information related to an arrangement of an object in a space is provided. The method may include obtaining three-dimensional space data corresponding to a space and object-related data of a first object in the space, obtaining a space map including a positional relationship of the first object in the space based on the three-dimensional space data and the object-related data, receiving a user input for changing an arrangement of the object in the space, on the basis of a user input, a null node indicating a second object to be arranged in a blank area in which the first object is not arranged in the space is added to the space graph, and the space graph to which the null node has been added is applied to a graph neural network (GNN) in order to update the space graph, and outputting information relating to a change in the arrangement of the object in the space based on the updated space map.

Description

Technical Field

[0001] The present disclosure relates to an electronic device for providing a user with information related to placement of an object in a space by using a space map, and an operating method of the electronic device. Background Art

[0002] In many cases, it is advantageous to be able to place home appliances / furniture in a space in a particular arrangement, but it is difficult to move bulky products in a real space. When placing a new home appliance / furniture in a space, it is difficult to actually try to place the home appliance / furniture in the real space, and therefore, it is difficult to determine whether the new home appliance / furniture is suitable for the real space.

[0003] To address these issues, various technologies / techniques have been developed for placing objects in a virtual reality / augmented reality space. Algorithms are used to provide information related to optimized object placement in a virtual reality / augmented reality space. Summary of the invention

[0004] According to one aspect of the present disclosure, a method performed by an electronic device includes: obtaining three-dimensional spatial data corresponding to a space and object-related data of multiple first objects in the space; based on the three-dimensional spatial data and the object-related data, obtaining a spatial graph including a positional relationship between multiple first objects in the space, the spatial graph including nodes corresponding to attributes of the multiple first objects and edges representing the positional relationship between the multiple first objects; receiving user input for changing the placement of an object in the space; based on the user input, adding an empty node to the spatial graph, the empty node representing a second object to be placed in an empty region in the space where the multiple first objects are not placed; updating the spatial graph by applying the spatial graph to which the empty node has been added to a graph neural network (GNN); and based on the updated spatial graph, outputting information related to the object placement change in the space.

[0005] According to one aspect of the present disclosure, an electronic device includes: a display; a memory storing one or more instructions; and at least one processor configured to access the memory and execute the one or more instructions stored in the memory to obtain at least three-dimensional spatial data corresponding to a space and object-related data of a plurality of first objects in the space, obtain a spatial graph including positional relationships between the plurality of first objects in the space based on the three-dimensional spatial data and the object-related data, the spatial graph including nodes corresponding to attributes of the plurality of first objects and edges representing positional relationships between the plurality of first objects, receive user input for changing the placement of an object in the space, add an empty node to the spatial graph based on the user input, the empty node representing a second object to be placed in a blank area in the space where the plurality of first objects are not placed, update the spatial graph by applying the spatial graph to which the empty node has been added to a graph neural network (GNN), and output information related to the object placement change in the space based on the updated spatial graph through the display.

[0006] According to an aspect of the present disclosure, there may be provided a computer-readable recording medium having recorded thereon a program for causing a computer to execute methods performed by an electronic device and providing information related to placing an object in a space, the methods being described above or below. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a diagram schematically illustrating an operation of an electronic device for providing information related to placement of an object in a space.

[0008] Figure 2 is a flowchart illustrating a method of providing information related to placement of an object in a space, performed by an electronic device according to an embodiment of the present disclosure.

[0009] Figure 3a is a diagram illustrating an operation of obtaining data for generating a spatial map, performed by an electronic device according to an embodiment of the present disclosure.

[0010] Figure 3b is a diagram illustrating an operation of generating a space map performed by an electronic device according to an embodiment of the present disclosure.

[0011] Figure 4a is a diagram showing a space diagram according to an embodiment of the present disclosure.

[0012] Figure 4b is a diagram showing types of space diagrams according to an embodiment of the present disclosure.

[0013] Figure 4c is a diagram illustrating an operation of training a spatial graph by using a graph neural network, performed by an electronic device according to an embodiment of the present disclosure.

[0014] Figure 5 is a diagram illustrating operations using a graph neural network performed by an electronic device according to an embodiment of the present disclosure.

[0015] Figure 6a is a diagram illustrating an operation of performing reasoning by using a graph neural network, performed by an electronic device according to an embodiment of the present disclosure.

[0016] Figure 6b is a diagram illustrating an operation of outputting object placement change related information of a space, performed by an electronic device according to an embodiment of the present disclosure.

[0017] Figure 6c It is used to further describe Figure 6b .

[0018] Figure 7a is a diagram illustrating an operation of performing reasoning by using a graph neural network, performed by an electronic device according to an embodiment of the present disclosure.

[0019] Figure 7b is a diagram illustrating an operation of outputting object placement change related information of a space, performed by an electronic device according to an embodiment of the present disclosure.

[0020] Figure 8 is a diagram illustrating an operation of outputting object placement change related information of a space based on a type of the space, performed by an electronic device according to an embodiment of the present disclosure.

[0021] Figure 9a is a diagram illustrating an operation of generating a personalized space map performed by an electronic device according to an embodiment of the present disclosure.

[0022] Figure 9b is a diagram illustrating an operation of outputting personalized recommendation information about a change in placement of an object in a space based on a personalized space map, performed by an electronic device according to an embodiment of the present disclosure.

[0023] Fig.10a is a diagram illustrating an operation of generating a metaverse spatial graph performed by an electronic device according to an embodiment of the present disclosure.

[0024] Fig.10b is a diagram illustrating an operation of updating a metaverse space map performed by an electronic device according to an embodiment of the present disclosure.

[0025] Fig.11 is a diagram illustrating an operation of recommending information related to object placement based on a feature of a real space, performed by an electronic device according to an embodiment of the present disclosure.

[0026] Fig.12is a block diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure.

[0027] Fig.13 is a block diagram showing a configuration of a server according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] Throughout this disclosure, the expression “at least one of a, b, or c” may indicate only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0029] Although the terms used herein are selected from common terms currently widely used in consideration of their functions in the present disclosure, these terms may be different according to the intention of a person of ordinary skill in the art, precedents, or the emergence of new technologies. In addition, in some cases, there are also terms arbitrarily selected by the applicant, and in this case, their meanings will be defined in detail in the specification. Therefore, the terms used herein are not merely designations of terms, but rather, the terms are defined based on the meanings of the terms and content throughout the present disclosure.

[0030] Singular expressions may also include plural meanings as long as they do not contradict the context. All terms used herein, including technical and scientific terms, may have the same meanings as those generally understood by those skilled in the art. In addition, although terms such as "first" or "second" may be used in this specification to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element.

[0031] Throughout the specification, when a part "includes" a component, it means that the part may further include other components rather than exclude other components, as long as there is no special description to the contrary. In addition, as used herein, terms such as "...er(or)", "...unit", "...module" and the like represent a unit that performs at least one function or operation, which can be implemented as hardware or software or a combination thereof.

[0032] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings to allow those skilled in the art to easily perform embodiments of the present disclosure. However, the present disclosure can be implemented in many different forms and should not be construed as being limited to the embodiments of the present disclosure set forth herein. In addition, in order to clearly describe the present disclosure, parts that are irrelevant to the description of the present disclosure are omitted, and similar reference numerals are assigned to similar elements throughout the specification. In addition, the reference numerals used in the various drawings are only used to describe the drawings, and the different reference numerals used in different drawings are not used to indicate different elements. Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

[0033] Figure 1is a diagram schematically illustrating an operation of an electronic device for providing information related to placement of an object in a space.

[0034] In an embodiment of the present disclosure, the electronic device 2000 may be a device including a display. For example, the electronic device 2000 may include, but is not limited to, a smart phone, a head-mounted display (e.g., an augmented reality (AR) or virtual reality (VR) device), a smart television (TV), a tablet personal computer (PC), a laptop computer, etc. The electronic device 2000 may be implemented as a device without a display (e.g., a desktop computer) and may be connected to a display device (e.g., a monitor).

[0035] In the present disclosure, "space" refers to a real space. The space may include, for example, a room, a living room, a study, an office, etc. In the present disclosure, "virtual space" refers to a space that is implemented to correspond to the real space. That is, the virtual space in the present disclosure is a virtual environment obtained by implementing the real space, and may be the result of reducing or enlarging the real space in the same / similar proportion considering the layout of the real space. As used herein, "metaverse space" is the same as the above-mentioned virtual space in that it implements a virtual environment, but unlike the above-mentioned virtual space, the term "metaverse space" refers to a space in which a separate virtual world different from the real space is implemented as a virtual environment.

[0036] In an embodiment of the present disclosure, the electronic device 2000 may display a virtual space corresponding to a real space on the screen. The virtual space may be a virtual environment obtained by realizing the user's real space (hereinafter referred to as "space"). In order to change the placement of an object in the space, the user may try to change the placement of the object in the virtual space by using the electronic device 2000. For example, a user of the electronic device 2000 may input an object to the electronic device 2000 to receive a recommendation for a location where the object is to be placed, or input a location to receive a recommendation for an object to be placed at that location. In the case where the electronic device 2000 is capable of providing an AR mode, the electronic device 2000 may provide an AR mode in which virtual objects, etc. are displayed and overlaid on the real space. In the present disclosure, for ease of description, the virtual space is described as an example, but the embodiments of the present disclosure may be similarly applied to the AR space.

[0037] refer to Figure 1 , the electronic device 2000 may receive a user input 100 for selecting a blank area in the virtual space from a user.

[0038] The electronic device 2000 may perform calculations for object placement based on user input and output object placement change related information 110. For example, because the user has selected a blank area in the virtual space, information for recommending an object to be placed in the blank area may be output as the object placement change related information 110.

[0039] The electronic device 2000 may use a spatial graph 120 and a graph neural network (GNN) to provide object placement change related information 110. The spatial graph 120 may include nodes and edges. The nodes correspond to objects (hereinafter, also referred to as "first objects") present in the space, and the edges represent the positional relationship between the objects present in the space. For example, in the spatial graph 120, the nodes may be shown as circles, and the edges may be shown as lines between the circles.

[0040] The user of the electronic device 2000 may place a new object (hereinafter, also referred to as a "second object") in the virtual space. Since the nodes of the spatial graph 120 correspond to the objects in the space, the electronic device 2000 may generate a new node 130 corresponding to the new object and add the new node 130 to the spatial graph 120. The new node 130 may be an empty node including only position information, and an object feature vector related to the object attributes in the node may not have been determined.

[0041] The electronic device 2000 may apply the spatial graph 120 to which the new node 130 has been added to the graph neural network. The graph neural network may be pre-trained. When the spatial graph 120 to which the new node 130 has been added is applied to the graph neural network, the feature vector of the new node 130 may be determined based on the existing nodes and edges included in the spatial graph 120 (this process is also referred to as "node embedding"). The electronic device 2000 may output object placement change related information 110 related to a new object to be placed in the virtual space (e.g., the location where the new object is to be placed, a new object to be placed at a specified location, etc.) based on the feature vector embedded in the new node 130.

[0042] Detailed operations performed by the electronic device 2000 to process object placement in a virtual environment by using the spatial graph 120 and the graph neural network and to provide the user with information related to a change in object placement in the space will be described below with reference to corresponding drawings.

[0043] Figure 2 is a flowchart illustrating a method of providing information related to placement of an object in a space, performed by an electronic device according to an embodiment of the present disclosure.

[0044] In the description Figure 2In order to distinguish between an object currently existing in the space and a new object to be placed, the object currently existing in the space will be referred to as a first object, and the new object will be referred to as a second object. In addition, in the following description, when describing an object that does not require distinction between the first object and the second object, the ordinal numbers "first" and "second" will be omitted.

[0045] In operation S210, the electronic device 2000 obtains three-dimensional space data corresponding to the space and object-related data of a first object in the space. For example, the space may have a plurality of first objects existing in the space.

[0046] In an embodiment of the present disclosure, the electronic device 2000 may obtain three-dimensional space data. For example, the electronic device 2000 may obtain three-dimensional space data including a three-dimensional space image obtained by three-dimensionally scanning the space using a camera. The three-dimensional space data may include, but is not limited to, a three-dimensional image of the space, a layout of the space, a size of the space (e.g., width, length, and height), a position of one or more objects in the space, and a size of one or more objects in the space.

[0047] In an embodiment of the present disclosure, the electronic device 2000 may obtain object-related data of a first object in space. The object-related data may include but is not limited to the position of the object in space, the size of the object in space, the type of the object, the identification information of the object, the orientation of the object, etc.

[0048] In operation S220, the electronic device 2000 obtains a space map including a positional relationship between first objects in a space based on the three-dimensional space data and the object-related data of the first objects.

[0049] In an embodiment of the present disclosure, the spatial graph may include nodes corresponding to attributes of the first objects and edges representing positional relationships between the first objects.

[0050] The nodes of the spatial graph may include object feature vectors representing object properties. The object feature vectors representing object properties may include various features related to the object, such as the location, size, category, color, or style of the object. The nodes of the spatial graph may include feature vectors related to the type of space (e.g., room) and the layout of the space (e.g., wall).

[0051] The edges of the spatial graph may be an adjacency matrix representing the positional relationship between adjacent objects. The positional relationship may be defined as various types related to the positional relationship between objects, such as "co-occurrence" positional relationship, "support" positional relationship, "supported" positional relationship, "surround" positional relationship, "surrounded" positional relationship, "next-to" positional relationship, etc. At the same time, for ease of description, it has been exemplarily described that a spatial graph can be defined using vectors and adjacency matrices, but this is only an example, and the data format representing the spatial graph is not limited to the above example.

[0052] In operation S230, the electronic device 2000 receives a user input for changing placement of an object in a space.

[0053] In an embodiment of the present disclosure, a user input for changing the placement of an object in a space may be an input corresponding to a “position in a space where the object is to be placed” (hereinafter, referred to as a position input). The position input may be received in various ways.

[0054] In an embodiment of the present disclosure, the electronic device 2000 may generate a virtual space corresponding to a space and display the virtual space on a screen. In this case, the virtual space may be generated based on three-dimensional space data and object-related data, and may refer to data that can be visually provided to a user by rendering the layout of the real space and the objects in the space into the virtual space in the same configuration as the real space. The electronic device 2000 may display the virtual space on a screen and receive user input for specifying a specific position in the space from a user viewing the virtual space on the screen. In detail, the electronic device 2000 may receive user input for selecting one or more positions in a blank area where an object is not placed.

[0055] In an embodiment of the present disclosure, the electronic device 2000 can receive natural language input from a user. The natural language input can be text data or voice data, and can be analyzed by using a natural language processing (NLP) model. In this case, when voice data is received, an automatic speech recognition (ASR) model can be used. For example, when the natural language input is "near TV", the electronic device 2000 can perform semantic analysis on the natural language input by using the NLP model, and identify the coordinates of the available blank area in the space next to the TV, the size of the blank area, etc. based on the three-dimensional space data and the object-related data.

[0056] In an embodiment of the present disclosure, the electronic device 2000 may receive raw data related to a position from a user. For example, the electronic device 2000 may receive user input representing the coordinates of a specific position in a space, the size of the position, etc. from a user.

[0057] The position input received in various ways may be converted into data for updating the spatial map. For example, the position input may be converted into the coordinates of the position, the size of the position (eg, width, length, and height), etc., but the present disclosure is not limited thereto.

[0058] In an embodiment of the present disclosure, a user input for changing the placement of an object in a space may be an input corresponding to an “object to be placed in the space”. The input corresponding to the “object to be placed in the space” (hereinafter referred to as the object input) may be an input for selecting a specific object so that the user requests a recommendation for a location in the space where the specific object is to be placed, but is not limited thereto, and may be received in various ways. The object input may include, for example, an input of an object category (e.g., TV), an input of object identification information (e.g., a TV model name), a natural language input (e.g., “a location recommended for placing the TV”), etc., but is not limited thereto. The detailed method of receiving the object input performed by the electronic device 2000 is the same as the example of the position input described above, so its description will be omitted.

[0059] The object input received in various ways may be converted into data for updating the spatial map. For example, the object input may be converted into an object category, an object size, etc., but the present disclosure is not limited thereto.

[0060] In operation S240, the electronic device 2000 adds an empty node to the spatial graph based on the user input, the empty node representing the second object to be placed in the blank area in the space where the first object is not placed. The empty node is a node representing the second object to be newly placed in the space, but is in a state where the second object has not yet been determined. That is, the object feature vector of the empty node representing the object attribute may not have been determined yet.

[0061] In an embodiment of the present disclosure, the type of user input received in operation S230 may be a position input. In this case, the electronic device 2000 may determine the position of the spatial graph to which the empty node has been added based on the position information of the position input (e.g., the coordinates in the position input, the size in the position input, etc.). The nodes of the spatial graph may include the coordinates of the object and the size of the object. For example, based on the position input "first position (x, y, z)", the electronic device 2000 may identify at least one first object (x, y, z) within a certain distance from the first position. 1 ,y 1 , z 1 ). In detail, when the position input corresponds to the position of a table in the space, the electronic device 2000 may recognize the table, which is the first object below the position, based on the position information.

[0062] In this case, the electronic device 2000 may add an empty node representing the second object to be adjacent to at least one node corresponding to the first object within a certain distance from the first location.

[0063] In an embodiment of the present disclosure, the type of user input received in operation S230 may be an object input. In this case, the electronic device 2000 may add one or more empty node candidates to the spatial graph. As an example, three empty node candidates may be added for one object input, and in this case, a spatial graph to which a first empty node candidate has been added, a spatial graph to which a second empty node candidate has been added, and a spatial graph to which a third empty node candidate has been added may be generated.

[0064] In this case, the electronic device 2000 can determine the position in the spatial map where one or more empty node candidates are to be added based on the position information of the nodes in the spatial map (e.g., the position of the first object, the size of the first object, etc.). For example, one or more empty node candidates may be added to the position in the spatial map corresponding to the blank area in the space. For example, based on the size of the second object input as an object, one or more empty node candidates may be added to the position corresponding to the blank area in the space, where the position has a size that the second object can be placed. That is, one or more empty node candidates may be added to the position corresponding to the empty node area in the space, where the position has a size sufficient to accommodate the placement of the second object.

[0065] When an empty node representing a second object is added in operation S240, an edge representing a connection relationship with a node corresponding to an adjacent first object may be determined. In this case, the type of edge represents the positional relationship between the objects. For example, when an empty node is added, the type of edge may be determined in a rule-based manner based on the positional information of the empty node. For example, the type of edge may be determined by user input. For example, the type of edge may be predicted by a graph neural network (GNN) in operation S250 described below.

[0066] In operation S250 , the electronic device 2000 updates the spatial graph by applying the spatial graph to which the empty node has been added to the GNN.

[0067] The electronic device 2000 can infer the object feature vector of the empty node in the spatial graph by using the GNN. For example, the electronic device 2000 can predict the edge to be connected to the empty node. The edge prediction may include, for example, whether the edge is connected and / or the type of the edge.

[0068] In an embodiment of the present disclosure, when the type of user input received in operation S230 is a position input, the electronic device 2000 can infer the object feature vector of the empty node by using GNN (this process is also referred to as "node embedding"). When the object feature vector is embedded in the empty node, the empty node can be referred to as a vector-embedded node. The spatial graph including the vector-embedded nodes can be referred to as an updated spatial graph. The object feature vector filled in the empty node as a result of the node embedding is determined based on the attributes of the first object. For example, based on various object-related features, such as the position, size, category, color, and style of the first object, the empty node can be filled with feature vectors related to various object features related to the object, such as the position, size, category, color, and style of the object.

[0069] In an embodiment of the present disclosure, when the type of user input received in operation S230 is an object input, the electronic device 2000 can infer object feature vectors of one or more empty node candidates by using a GNN. When multiple empty node candidates are generated, the object feature vector can be embedded in each of the multiple empty node candidates. The object feature vector filled in each empty node candidate as a result of node embedding is determined based on the attributes of the first object. For example, based on various object-related features, such as the position, size, category, color, and style of the first object, the empty node can be filled with feature vectors related to various object features related to the object, such as the position, size, category, color, and style of the object.

[0070] The electronic device 2000 may select one or more of the vector embedding node candidates based on the object feature vector of the vector embedding node candidate. In the example, embedding may be performed on a first empty node candidate in a spatial graph including a first empty node candidate, embedding may be performed on a second empty node candidate in a spatial graph including a second empty node candidate, and embedding may be performed on a third empty node candidate in a spatial graph including a third empty node candidate. In this case, the first empty node candidate and the second empty node candidate may be selected. However, although the selection of empty node candidates is described above, the selected nodes are nodes in which the object feature vector is embedded, that is, the first vector embedding node and the second vector embedding node may be selected. The spatial graph including the selected vector embedding nodes may be referred to as an updated spatial graph.

[0071] In operation S260, the electronic device 2000 outputs object placement change related information of the space based on the updated space map.

[0072] The electronic device 2000 may output the object placement change related information of the space based on the object feature vector in the vector embedding node of the updated space graph.

[0073] In an embodiment of the present disclosure, when the type of user input received in operation S230 is a position input, the electronic device 2000 may determine and output one or more object categories corresponding to a second object that may be placed at the input position. In detail, the electronic device 2000 may output "object A, object B, object C, etc.", which may be placed at the input position.

[0074] In an embodiment of the present disclosure, when the type of user input received in operation S240 is an object input, the electronic device 2000 can determine and output one or more positions where the input second object is to be placed. In detail, the electronic device 2000 can output the positions where the input object can be placed "position A, position B, position C, etc."

[0075] Figure 3a is a diagram illustrating an operation of obtaining data for generating a spatial map, performed by an electronic device according to an embodiment of the present disclosure.

[0076] Three-dimensional spatial data may include, but is not limited to, a three-dimensional image of the space, the layout of the space, the dimensions of the space (e.g., width, length, and height), the position of one or more objects in the space, and the dimensions of one or more objects in the space.

[0077] In an embodiment of the present disclosure, the electronic device 2000 may perform a three-dimensional scan of a space under the control of a user. In an embodiment, the electronic device 2000 may perform a three-dimensional scan of a space under the control of a processor. In this case, the electronic device 2000 may include one or more sensors for three-dimensional scanning. For example, one or more sensors may use but are not limited to red-green-blue (RGB)-depth (RGB-D) sensors, time of flight (ToF) sensors, light detection and ranging (LiDAR) sensors, radio detection and ranging (RADAR) sensors, and the like. The electronic device 2000 may generate three-dimensional space data 310 based on sensor data obtained by three-dimensional scanning.

[0078] In an embodiment of the present disclosure, three-dimensional spatial data 310 may be received from an external device (e.g., a server or other electronic device). For example, three-dimensional spatial data 310 obtained by another electronic device (e.g., a robot cleaner) that performs a three-dimensional scan of a space may be received by the electronic device 2000. For example, the electronic device 2000 may receive three-dimensional spatial data 310 obtained by a three-dimensional precision measuring instrument (e.g., a LiDAR scanner). For example, pre-stored three-dimensional spatial data 310 may be received from a server. For example, the three-dimensional spatial data 310 may be obtained by receiving direct input of the dimensions of the space. In this case, accurate three-dimensional spatial data 310 based on the user's actual measurement of the space may be received by the electronic device.

[0079] The object-related data 320 may include, but is not limited to, the position of the object in space, the size of the object in space, the type of the object, identification information of the object, the orientation of the object, and the like.

[0080] In an embodiment of the present disclosure, the electronic device 2000 may generate object-related data 320 based on the three-dimensional space data 310 .

[0081] The electronic device 2000 can obtain the position and size of the object by performing a three-dimensional scan of the space. For example, the electronic device 2000 can obtain the position and size of the object by detecting one or more objects in the space from a three-dimensional image of the space. In this case, an object detection model can be used, which is an artificial intelligence model for detecting objects in the space. For example, the electronic device 2000 can combine the position and size of the object obtained from the three-dimensional scan of the space with the position and size of the object obtained by using the object detection model.

[0082] The electronic device 2000 can obtain the type of the object and the identification information of the object by identifying one or more objects in the three-dimensional image (or two-dimensional image) of the space. In this case, an object recognition model can be used, which is an artificial intelligence model for identifying objects in the space. The object recognition model may include the algorithm of the object detection model described above for detecting the position, size, etc. of the object, and thus the object detection result may also be output.

[0083] In an embodiment of the present disclosure, object-related data 320 may be received from an external device. For example, the electronic device 2000 may receive pre-stored information of an object in a space as the object-related data 320 from a server. For example, the electronic device 2000 may receive object-related data 320 obtained by detecting / recognizing an object by another electronic device from another electronic device. For example, the object-related data 320 may be obtained by receiving a direct input of the dimensions of the object. In this case, accurate object-related data 320 based on actual measurement of the object by the user may be received by the electronic device 2000.

[0084] The method of obtaining the three-dimensional space data 310 and the object-related data 320 performed by the electronic device 2000 is not limited to the above-mentioned example. For example, two or more of the methods of obtaining the three-dimensional space data 310 and the object-related data 320 according to the above-mentioned examples may be used complementary to each other, and thus, accurate three-dimensional space data 310 and object-related data 320 may be obtained.

[0085] Figure 3b is a diagram illustrating an operation of generating a space map performed by an electronic device according to an embodiment of the present disclosure.

[0086] In an embodiment of the present disclosure, the electronic device 2000 can generate a spatial graph 330 by using the three-dimensional spatial data 310 and the object-related data 320. In this case, based on the three-dimensional spatial data 310 and the object-related data 320, a feature vector of a node representing an object attribute can be determined, and an edge representing an object connection relationship can be determined. In this case, a graph generation model as an artificial intelligence model for generating the spatial graph 330 can be used.

[0087] In an embodiment of the present disclosure, the electronic device 2000 may receive a spatial map 330 defining objects in a space and positional relationships between objects from an external device (eg, a server). In this case, the spatial map received from the external device may be generated by using the spatial data 310 and the object-related data 320.

[0088] In an embodiment of the present disclosure, the electronic device 2000 may generate or update the spatial graph 330. For example, the electronic device 2000 may generate or update the spatial graph 330 based on user input. For example, the electronic device 2000 may receive user input and generate a spatial graph 330 including nodes representing objects and edges representing positional relationships between objects. The electronic device 2000 may receive user input and modify the nodes and / or edges of the generated spatial graph 330 based on the received user input.

[0089] Figure 4a is a diagram showing a space diagram according to an embodiment of the present disclosure.

[0090] Figure 4a The spatial graph 400 is visually depicted. The nodes of the spatial graph may be data including eigenvectors, and the edges of the spatial graph may be data represented by an adjacency matrix. For example, the nodes of the spatial graph may be data consisting of eigenvectors, and the edges of the spatial graph may be data represented by an adjacency matrix, but are not limited thereto. Figure 4a The spatial graph 400 shown in is the initial graph and can be updated by the GNN to the final graph. This will be referred to below Figure 4c Further description.

[0091] In an embodiment of the present disclosure, the nodes of the space graph 400 may correspond to the type of space, the layout of the space, or objects present in the space. Hereinafter, the first node 410, the second node 420, and the third node 430 will be described as examples.

[0092] The first node 410 may be a node corresponding to the type of space "room". In this case, the first node 410 may include a space feature vector related to the type of space. The space feature vector related to the type of space may include, for example, the type of space, the main objects in the space, the style of the space, the color of the space, etc., but is not limited thereto.

[0093] The second node 420 may be a node corresponding to the layout "wall" of the space. In this case, the second node 420 may include a spatial feature vector related to the layout of the space. The spatial feature vector related to the layout of the space may include, for example, the style of the layout, the color of the layout, the presence or absence of a door, the presence or absence of a wall socket, etc., but is not limited thereto.

[0094] The third node 430 may be a node corresponding to a "cabinet", which is an object (also referred to as a first object) existing in the space. In this case, the third node 430 may include an object feature vector related to the attributes of the object. The object feature vector may include, for example, the position of the object, the size of the object, the category of the object, the color of the object, the style of the object, etc., but is not limited thereto.

[0095] In an embodiment of the present disclosure, the edges of the spatial graph 400 may represent the positional relationship between adjacent objects and / or layouts in the space. The positional relationship may be defined as various types related to the positional relationship between objects, such as a "co-occurrence" positional relationship, a "supporting" positional relationship, a "supported" positional relationship, a "surrounding" positional relationship, a "surrounded" positional relationship, a "closely adjacent" positional relationship, etc.

[0096] The detailed and exemplary definitions of the positional relationships are as follows:

[0097] "Support" positional relationship: When an object is placed on top of another object, the lower object is defined as supporting the higher object. For example, in the case of "table->vase", the table is positioned to support the vase, so the edge from the table to the vase is defined as type "support".

[0098] "Supported" positional relationship: When an object is located below another object, the higher object is defined as supported by the lower object. For example, in the case of "vase->table", the vase is positioned to be supported by the table, so the edge from the vase to the table is defined as type "Supported".

[0099] "Surround" positional relationship: When objects of the same or similar size are positioned to surround a central object, the peripheral objects are defined to surround the central object. For example, in the case of "lamp A->bed<-lamp B", the lamps are positioned to surround the bed, and therefore, the edges from each of the lamps to the bed are defined as type "surround".

[0100] "Surrounded" positional relationship: When a central object is positioned to be surrounded by objects of the same or similar size, the central object is defined to be surrounded by peripheral objects. For example, in the case of "lamp A <- bed -> lamp B", the bed is positioned to be surrounded by the lamps, and therefore, the edges from the bed to each lamp are defined as the "surrounded" type.

[0101] "Proximity" positional relationship: When two objects at the same or similar height are adjacent to each other within a certain distance, the two objects are defined as being adjacent to each other. The edge between two objects can be undirected. For example, in the case of "vase-TV", the vase and TV are adjacent to each other, so the edge connecting the vase to the TV is defined as the "proximity" type.

[0102] "Co-occurrence" positional relationship: All objects existing in the same room are defined as co-occurrence. However, the "co-occurrence" positional relationship has the lowest priority, and objects that do not correspond to the above positional relationship are indicated as co-occurrence. For example, in a spatial graph corresponding to a room, a directed edge of 'room->wall->object' is generated, and the type of the edge is defined as type 'co-occurrence'.

[0103] Hereinafter, the positional relationship among the first node 410 , the second node 420 , and the third node 430 will be described as an additional example.

[0104] For example, a room requires the presence of a wall, and therefore, a first node 410 corresponding to the room may be defined as being in a "co-occurrence" positional relationship with a second node 420 corresponding to the wall.

[0105] For example, the cabinet is placed in direct contact with the wall, and therefore, the second node 420 corresponding to the wall and the third node 430 corresponding to the cabinet may be defined as being in a “co-occurrence” position relationship.

[0106] However, the above positional relationship is only an example, and the edges may be Figure 4a The space diagram 400 shown in FIG. 4 connects nodes differently. For example, the second node 420 corresponding to the wall and the third node 430 corresponding to the cabinet can be defined as being in a "close proximity" position relationship.

[0107] Nodes corresponding to objects around the cabinet (eg, lamp, vase, and TV) may be connected to the third node 430 corresponding to the cabinet through an edge. In this case, the type of the edge may be determined according to the positional relationship between the objects.

[0108] Figure 4b is a diagram showing types of space diagrams according to an embodiment of the present disclosure.

[0109] In an embodiment of the present disclosure, the spatial diagram 400 may be divided into multiple types of spatial diagrams. Each of the multiple types of spatial diagrams may correspond to the above reference Figure 4a Each of the multiple types of positional relationships described.

[0110] Figure 4b A spatial graph corresponding to a "co-occurrence" positional relationship is shown. In this case, the edges connecting the nodes represent the "co-occurrence" positional relationship. For example, for a room type space, walls, wardrobes, cabinets, etc. can be defined as being in a "co-occurrence" relationship with each other.

[0111] although Figure 4b Only the spatial graphs corresponding to the “co-occurrence” positional relationship are shown, but there may be other types of spatial graphs corresponding to the above-mentioned positional relationships.

[0112] In an embodiment of the present disclosure, the spatial graph 400 may be a combination of multiple types of spatial graphs. In this case, edges representing all types of positional relationships may be included in one spatial graph 400. In the case where multiple types of edges are included in one spatial graph 400, each type of edge may be identified by using an index or the like.

[0113] Figure 4c is a diagram illustrating an operation performed by an electronic device of training a spatial graph by using a GNN according to an embodiment of the present disclosure.

[0114] In an embodiment of the present disclosure, the electronic device 2000 can train the spatial graph 400 by applying the spatial graph 400 to a graph neural network (GNN). The GNN can update the state of each node by using the edges representing the connection relationship between the nodes and the states of the adjacent nodes. Here, the final state of the node after the update can be referred to as a node embedding.

[0115] Reference Figure 4c An example of performing node embedding on the TV node 440, which is one of the nodes of the spatial graph 400, is described. The node embedding of the TV node 440 can be equally applied to other nodes of the spatial graph 400, and therefore, for the sake of brevity, redundant description will be omitted.

[0116] In an embodiment of the present disclosure, the TV node 440 may receive a message from an adjacent node. As an example of a message, in the case where the w node is adjacent to the v node, the message received by the v node refers to information generated based on the hidden state and feature vector of the v node, and the hidden state and feature vector of the w node are aggregated and then delivered to the v node. That is, the message is defined as a function including the hidden state and feature vector of the v node and the hidden state and feature vector of the w node as variables.

[0117] For example, node A and node B may be adjacent to TV node 440. In this case, TV node 440 may receive a message from node A and a message from node B. In this case, node A delivers the message to TV node 440 based on an aggregation of messages from other nodes adjacent to node A, and node B delivers the message to TV node 440 based on an aggregation of messages from other nodes adjacent to node B.

[0118] In the embodiments of the present disclosure, as shown above, Figure 4b As described above, the spatial diagram 400 may include multiple types of spatial diagrams. In this case, a message may be received from each type of spatial diagram.

[0119] For example, in the "support" type space graph 450-1, the message may be received by the TV node 440. In this case, when the message is aggregated and then delivered to the neighboring nodes in the "support" type space graph 450-1, there may be nodes in the "support" type space graph 450-1 that have no edges connected to them (see Figure 4b , which shows an example of a "co-occurrence" type spatial graph). In this case, for nodes that have no edges connected to them, it is possible to Figure 4a The connection relationship of the spatial graph 400 delivers the message to the TV node 440.

[0120] In the same manner, in each of the “supported” type spatial graph 450-2, the “surrounded” type spatial graph 450-3, the “surrounded” type spatial graph 450-4, the “adjacent” type spatial graph 450-5 and the “co-occurrence” type spatial graph 450-6, a message reflecting each type of positional relationship can be delivered to the TV node 440.

[0121] In an embodiment of the present disclosure, the GNN may aggregate messages delivered to the TV node 440. For such message aggregation, the GNN may include a convolutional layer, but is not limited thereto.

[0122] The GNN may aggregate messages delivered to the TV node 440, and based on the result of the cascaded aggregated messages, update the state of the TV node 440 from the current state to the next state (e.g., from t to t+1). For such an update, the GNN may include a multilayer perceptron (MLP), but is not limited thereto. The update function of the v node may be defined as including the hidden state (t) of the v node and the message received by the v node as variables, and the v node may be updated to the next hidden state (t+1) by the update function. That is, based on the current state of the TV node 440 and the message received by the TV node 440, the TV node 440 may be updated to the next state.

[0123] In an embodiment of the present disclosure, the final feature vector of the TV node 440 may be filled by iterating the above-mentioned update. This iteration is referred to as TV node embedding 470.

[0124] When the electronic device 2000 applies the spatial graph 400 to the GNN, each node in the spatial graph 400 is embedded, as in the example of the TV node embedding 470 .

[0125] When the electronic device 2000 trains the spatial graph 400 by using the GNN so that embedding is performed on all nodes in the spatial graph 400, then adds new nodes to the trained spatial graph, and applies the spatial graph to the GNN, the newly added nodes and edges can be inferred. The operation is described below with reference to the corresponding drawings.

[0126] Figure 5 is a diagram illustrating an operation using a GNN performed by an electronic device according to an embodiment of the present disclosure.

[0127] In the description Figure 5 When placing a new object in the space, in order to distinguish between an object currently existing in the space and a new object to be placed, the object currently existing in the space will be referred to as a first object, and the new object will be referred to as a second object.

[0128] In an embodiment of the present disclosure, the electronic device 2000 may receive a user input for changing the placement of an object in space. As described above, the user input may be a position input or an object input. When the position input is received, the electronic device 2000 may infer a second object to be placed at the input position. When the object input is received, the electronic device 2000 may infer the position in the space where the input object (e.g., the second object) is to be placed. Figure 5 An example is shown where the user input is a location input.

[0129] In an embodiment of the present disclosure, the electronic device 2000 may receive an input for selecting the first position 510 from among a blank area in a space where the first object is not placed.

[0130] The electronic device 2000 may add an empty node 520 to the space graph based on the coordinates of the first position 510 .

[0131] When adding the empty node 520 to the space graph, the electronic device 2000 may determine the edge of the empty node 520 to be adjacent to the nodes corresponding to the first objects within a certain distance from the first position 510 among the first objects.

[0132] The electronic device 2000 may update the spatial graph by applying the spatial graph to which the empty node 520 has been added to the GNN 530. In this case, the object feature vector of the empty node 520 may be inferred, and the type of the edge to be connected to the empty node 520 may be predicted.

[0133] When the empty node embedding 540 is completed, the electronic device 2000 may determine one or more object categories of the second object that may be placed at the first position 510 based on the inferred object feature vector, and output the determined object categories.

[0134] In an embodiment, the electronic device 2000 may receive an object input from a user and provide a position prediction. Figures 6a to 7b An operation performed by the electronic device 2000 to infer an object based on a position input from a user or to infer a position where an object is to be placed based on an object input from a user is described in more detail.

[0135] Figure 6a is a diagram illustrating an operation of performing reasoning by using a GNN, performed by an electronic device according to an embodiment of the present disclosure.

[0136] In the description Figures 6a to 6c When placing a new object in the space, in order to distinguish between an object currently existing in the space and a new object to be placed, the object currently existing in the space will be referred to as a first object, and the new object will be referred to as a second object.

[0137] In an embodiment of the present disclosure, the electronic device 2000 may receive an input for selecting the first position from among a blank area in a space where the first object is not placed.

[0138] The electronic device 2000 may add the empty node 610 to the spatial graph based on the coordinates of the first position. Here, the spatial graph may have been generated by the GNN 620 in the same manner as in the above reference. Figure 4c The empty node 610 may be a node representing a second object to be placed at the first position, and the object feature vector may not have been embedded in the node.

[0139] When adding the empty node 610 to the spatial graph, the electronic device 2000 may connect an edge to the empty node 610 so that the empty node 610 is adjacent to a node corresponding to the first object within a certain distance from the first position among the first objects. For example, an edge may be generated between the empty node 610 and the wall node 612.

[0140] The electronic device 2000 may update the spatial graph by applying the spatial graph to which the empty node 610 has been added to the GNN 620. In this case, the object feature vector of the empty node 610 may be inferred. In addition, the type of the edge to be connected to the empty node 610 may be predicted.

[0141] The inference of the object feature vector and the prediction of the type of the edge are determined by GNN 620 applying the attributes of the first object to the empty node 610. In detail, GNN 620 may aggregate messages received by the wall node 612 from other nodes adjacent to the wall node 612, and deliver the messages from the wall node 612 to the empty node 610. Therefore, when the feature vector is embedded therein, the empty node 610 may become a vector embedding node 630. This operation is described above, and therefore, for the sake of brevity, redundant descriptions will be omitted.

[0142] When the spatial graph is updated to include the vector embedding node 630, the electronic device 2000 can determine one or more object categories of the second object that can be placed at the first position based on the object feature vector of the vector embedding node 630, and output at least one of the determined object category, object color, or object size. For example, the electronic device 2000 can output the probability 640 of the object category of the second object placed at the first position. In this case, the GNN 620 has learned various object attributes, such as the style, color, type, position, direction, or size of the objects in the space and the relationship between the objects. Therefore, the probability of the object category of the second object can be a result reflecting the characteristics of the space. According to the result of outputting the probability 640 of the object category of the second object, it is inferred that the second object suitable for being placed at the first position can be a lamp, a TV, and a vase in the order of probability.

[0143] Figure 6b is a diagram illustrating an operation of outputting object placement change related information of a space, performed by an electronic device according to an embodiment of the present disclosure.

[0144] In an embodiment of the present disclosure, the electronic device 2000 may generate a virtual space corresponding to the space and display the virtual space on the screen. The electronic device 2000 may receive an input for selecting a first position from among a blank area where the first object is not placed.

[0145] For example, the electronic device 2000 may receive a user input for selecting the blank area 650 displayed on the screen. The electronic device 2000 may update the spatial graph by adding an empty node corresponding to the blank area 650 to the spatial graph and applying the spatial graph to which the empty node has been added to the GNN.

[0146] Based on the updated spatial map, the electronic device 2000 may output the object placement change related information 660 of the space. For example, because the user input is a position input, the object placement change related information 660 of the space may include a recommendation for an object to be placed at the input position. In detail, "laptop computer", "book", "vase", etc. may be output as objects to be placed in the blank area 650. The object placement change related information 660 may be displayed based on the priority order of the object categories, but is not limited thereto.

[0147] The electronic device 2000 may display a virtual space in which an object is placed. The electronic device 2000 may receive a user input for testing placement of an object in a virtual space based on the information 660 related to object placement changes. For example, when a user selects "laptop computer" based on the object placement change related information 660, the electronic device 2000 may place a virtual object 670 representing a laptop computer in the virtual space. When a user selects "vase" based on the object placement change related information 660, the electronic device 2000 may place a virtual object 672 representing a vase in the virtual space.

[0148] Figure 6c It is used to further describe Figure 6b .

[0149] In an embodiment of the present disclosure, the position input may include the size of the position. For example, the user may touch and drag a blank area in the virtual space displayed by the electronic device 2000 to specify the position in the blank area and the size of the position.

[0150] The electronic device 2000 can recommend an object to be placed at the input position based on the coordinates and size of the position in the position input from the user. When the position input is received from the user, the electronic device 2000 can add an empty node corresponding to the input position to the spatial graph. In this case, the position and size of the empty node can be included as the initial feature vector of the empty node. As a result of applying the spatial graph to which the empty node has been added to the GNN by the electronic device 2000, the object feature vector embedded in the empty node can reflect the size of the position. In detail, feature vectors related to objects having a size within or outside a certain range based on the size of the input position can be embedded in the empty node, or features related to objects having a size less than or equal to the size of the input position can be embedded in the empty node.

[0151] For example, the user input for the same space may be a small-size position input 680 or a large-size position input 690. In this case, an object may be recommended based on the size of the position. For example, when a small-size position input 680 is received, the electronic device 2000 may output a "vase" as a recommended object. When a large-size position input 690 is received, the electronic device 2000 may output a "TV" as a recommended object.

[0152] Figure 7a is a diagram illustrating an operation of performing reasoning by using a GNN, performed by an electronic device according to an embodiment of the present disclosure.

[0153] In the description Figure 7a to Figure 7b When placing a new object in the space, in order to distinguish between an object currently existing in the space and a new object to be placed, the object currently existing in the space will be referred to as a first object, and the new object will be referred to as a second object.

[0154] In an embodiment of the present disclosure, the electronic device 2000 may receive a request for a recommendation for a location of a second object that is not currently placed in the space. For example, when a user inputs information related to the second object (e.g., a category or model name), the electronic device 2000 may recommend a location in the space where the second object is to be placed.

[0155] The electronic device 2000 may add one or more empty node candidates to the spatial graph based on an object input indicating a second object (e.g., an object category of the second object or a model name of the second object). For example, a spatial graph to which a first empty node candidate 710 has been added and a spatial graph to which a second empty node candidate 720 has been added may be generated, and a spatial graph to which both the first empty node candidate 710 and the second empty node candidate 720 have been added may also be generated. The empty node candidates may be generated to correspond to a blank area in space, and may include position information of the blank area. For example, the first empty node candidate 710 may include position information of a first position of a blank area in space, and the second empty node candidate 720 may include position information of a second position of another blank area in space. A node representing a position that is most suitable for placing the second object may be ultimately selected from the empty node candidates.

[0156] The electronic device 2000 can update the spatial graph by applying the spatial graph to which the empty node candidate has been added to the GNN. In this case, the object feature vector of the empty node can be inferred. In addition, the type of edge to be connected to the empty node can be predicted. The inference of the object feature vector and the prediction of the edge type are determined by the GNN applying the attributes of the first object to the empty node. For example, the electronic device 2000 can perform embedding on the first empty node candidate 710 by applying the spatial graph to which the first empty node candidate 710 has been added to the GNN. The electronic device 2000 can perform embedding on the second empty node candidate 720 by applying the spatial graph to which the second empty node candidate 720 has been added to the GNN. The electronic device 2000 can perform embedding on both the first empty node candidate 710 and the second empty node candidate 720 by applying the spatial graph to which both the first empty node candidate 710 and the second empty node candidate 720 have been added to the GNN.

[0157] The electronic device 2000 may select one or more of the empty node candidates based on the result of performing embedding on the empty node candidates.For example, the electronic device 2000 may compare the object feature vector embedded in the first empty node candidate 710 with the object category of the second object.

[0158] In detail, an example will be described in which the object category of the second object is "TV" when the user inputs the object input "TV" to place the TV. The object categories inferred from the first empty node candidate 710 may be "TV", "vase", and "laptop" in order, which are the results obtained by the electronic device 2000 updating the spatial graph using the GNN and determining the object category based on the object feature vector embedded in the first empty node candidate 710. This order means that the order of suitability of the category of the object to be placed at the position corresponding to the first empty node candidate 710 is "TV", "vase", and "laptop". In the same way, the category of the object inferred from the second empty node candidate 720 may be "laptop", "vase", and "TV" in order. This order means that the order of suitability of the category of the object to be placed at the position corresponding to the second empty node candidate 720 is "laptop", "vase", and "TV". In this case, the electronic device 2000 can select the first empty node candidate 710 from among the first empty node candidate 710 and the second empty node candidate 710 as the final data of the spatial graph. Therefore, the finally updated spatial graph may include the selected node (ie, the node obtained by embedding the object feature vector in the first empty node candidate 710).

[0159] When the spatial graph is updated to include the selected node, the electronic device 2000 may determine one or more locations in the space where the second object is to be placed based on the object feature vector of the selected node, and output the determined locations. For example, the electronic device 2000 may output the coordinates of the location where the second object is to be placed. When the electronic device 2000 is capable of displaying a virtual space corresponding to the space on the screen, the electronic device 2000 may display the location where the second object is to be placed on the screen.

[0160] Figure 7b is a diagram illustrating an operation of outputting object placement change related information of a space, performed by an electronic device according to an embodiment of the present disclosure.

[0161] In an embodiment of the present disclosure, the electronic device 2000 may generate a virtual space corresponding to the space and display the virtual space on the screen.The electronic device 2000 may receive a request for a recommendation for a position of a second object that is not currently placed in the space.

[0162] For example, the electronic device 2000 may receive a user input for selecting the TV 730 as the object category as the second object. In order to find the position in the space where the TV 730 is to be placed, the electronic device 2000 may add one or more empty node candidates to the spatial graph, and update the spatial graph by applying the spatial graph to which the one or more empty node candidates are added to the GNN.

[0163] Based on the updated spatial map, the electronic device 2000 may output the object placement change related information 740 of the space. For example, because the user input is an object input, the object placement change related information 740 of the space may include a recommendation for a location where the input object is to be placed. In detail, location A, location B, location C, etc. may be recommended as the location where the TV 730 is to be placed. The object placement change related information 740 for recommending the location where the object is to be placed may be displayed based on the priority order of the location, but is not limited thereto.

[0164] The electronic device 2000 may display the position where the object is to be placed in the virtual space. The electronic device 2000 may receive a user input for testing the placement of the object in the virtual space based on the object placement change related information 740. For example, when the user selects position A based on the object placement change related information 740, the electronic device 2000 may place the virtual object representing the TV 730 at a position 750 corresponding to the position A in the virtual space. When the user selects position B, the electronic device 2000 may place the virtual object representing the TV 730 at a position corresponding to the position B in the virtual space.

[0165] Figure 8 is a diagram illustrating an operation of outputting object placement change related information of a space based on a type of the space, performed by an electronic device according to an embodiment of the present disclosure.

[0166] In an embodiment of the present disclosure, a spatial graph may include a node corresponding to a type of space. The type of space may be, for example, a room, a living room, an office, etc., but is not limited thereto. A node corresponding to the type of space may include a feature vector related to the type of space. The feature vector related to the type of space may include, for example, the type of space, the main objects in the space, the style of the space, the color of the space, etc., but is not limited thereto.

[0167] Since spaces can be classified into multiple types, multiple space maps may exist. For example, different space maps may be obtained for spaces with different usage purposes, such as (ordinary) rooms, bedrooms, study rooms, living rooms, dressing rooms, etc. in a house. In this case, each of the multiple space maps may be generated based on the three-dimensional spatial data of each space and data related to objects in each space.

[0168] In an embodiment of the present disclosure, since the spatial graph includes nodes corresponding to the types of spaces, the electronic device 2000 can use GNN to train the spatial graph to reflect the characteristics of the space. That is, the spatial graph may correspond to the type of space. For example, there may be a spatial graph corresponding to the office type 810 and a spatial graph corresponding to the room type 820. In this case, as shown in the above reference Figure 4cAs described above, by applying the spatial graph corresponding to the office type 810 to the GNN, embedding can be performed on the nodes in the spatial graph, and the objects existing in the space of the office type 810 and the positional relationships between the objects can be reflected in the feature vectors and edges in the spatial graph. In addition, by applying the spatial graph corresponding to the room type 820 to the GNN, embedding can be performed on the nodes in the spatial graph, and the objects existing in the space of the office type 820 and the positional relationships between the objects can be reflected in the feature vectors and edges in the spatial graph.

[0169] In an embodiment of the present disclosure, the electronic device 2000 may identify the type of space based on a user input, and load a space map corresponding to the identified type of space.

[0170] For example, when the user specifies the office type 810 as the space type, the space map corresponding to the office type 810 may be loaded. In this case, the electronic device 2000 may generate a virtual space corresponding to the office space and display the virtual space on the screen. According to the above-described embodiment of the present disclosure, when an object input or a position input of the virtual space is received from the user, the electronic device 2000 may apply the space map corresponding to the office type 810 to the GNN. The electronic device 2000 may update the space map corresponding to the office type 810, and output object placement change related information 812 of the office space based on the updated space map. For example, as an output in response to the position input from the user, "laptop", "vase", "book", etc. may be output as objects to be placed on the table in the virtual space corresponding to the office type 810. Although not in Figure 8 , but as an output in response to an object input from a user, a position where the object input by the user is to be placed in a virtual space corresponding to the office type 810 may be output. In this case, a space map corresponding to the office type 810 may also be used. Here, object placement change related information 812 for the office space indicates that an object suitable for the office space (or a position where the object is to be placed) is recommended by using the space map corresponding to the office type 810.

[0171] As an additional example, when the user specifies room type 820 as the space type, a space graph corresponding to room type 820 may be loaded. Therefore, the electronic device 2000 may apply the space graph corresponding to room type 820 to the GNN. The electronic device 2000 may output object placement change related information 822 for the room space. For example, as an output in response to a position input from the user, object placement change related information 822 for the room space indicating "TV", "laptop", "plant", etc. as objects to be placed on a TV stand in the room space may be output. Here, the object configuration change related information 822 for the room space indicates that objects suitable for the room space (or locations where objects are to be placed) are recommended by using a space graph corresponding to the room type 820.

[0172] Figure 9a is a diagram illustrating an operation of generating a personalized space map performed by an electronic device according to an embodiment of the present disclosure.

[0173] In the description Figure 9a When the user preference style space 901 is used, it refers to a space in which interior design elements are configured in a style preferred by the user. For example, the interior design elements of the space may include various elements that can be used as a standard for user preferences, such as the style of the space (e.g., modern or retro) and the color of the space (e.g., white, wood, or gray), and the preference may be selected or input by the user.

[0174] In an embodiment of the present disclosure, the electronic device 2000 may obtain a user-preferred style space image (hereinafter, referred to as a preferred space image) including a user-preferred style. For example, the electronic device 2000 may receive a preferred space image from the user.

[0175] The electronic device 2000 can extract various features for generating a spatial graph from the preferred spatial image. For example, the electronic device 2000 can identify the type, style, etc. of the user's preferred style space 901 from the preferred spatial image. In this case, a scene classification model can be used, which is an artificial intelligence model for detecting / recognizing scene features from an image. For example, the electronic device 2000 can detect / recognize one or more objects present in the user's preferred style space 901 from the preferred spatial image. In this case, an object detection / recognition model can be used, which is an artificial intelligence model for detecting / recognizing objects in space. The electronic device 2000 can generate a preferred spatial graph 910 by converting the features of the user's preferred style space 901 into data based on the features extracted from the preferred spatial image. The preferred spatial graph 910 may include nodes representing objects present in the user's preferred style space 901, and edges representing positional relationships between objects. In some embodiments, the electronic device 2000 can train the preferred spatial graph 910 by applying the preferred spatial graph 910 to the GNN so that the feature vector of the preferred spatial graph 910 is supplemented and / or modified.

[0176] In an embodiment of the present disclosure, the electronic device 2000 may obtain a spatial map 920-1 corresponding to the user's real space 902. The spatial map 920-1 corresponding to the user's real space 902 may be generated based on three-dimensional spatial data and object-related data, which is described above, and therefore, for the sake of brevity, redundant descriptions will be omitted.

[0177] In an embodiment of the present disclosure, the electronic device 2000 may personalize the spatial graph 920-1 by modifying at least one of the nodes or edges of the spatial graph 920-1 corresponding to the user's real space 902 based on the result of comparing the preferred spatial graph 910 with the spatial graph 920-1 corresponding to the user's real space 902. Figure 9b The personalization of the spatial map 920-1 is further described.

[0178] In addition to obtaining the preferred space map 910 from the preferred space image as described above, the electronic device 2000 may also obtain the preferred space map 910 in various ways.

[0179] For example, the electronic device 2000 may provide a spatial data store through which a user may select and / or purchase a preferred style. The storage may be provided in the form of an application by the electronic device 2000, but is not limited thereto. The spatial data store may include two-dimensional and / or three-dimensional images of a space and a spatial map corresponding thereto. The spatial data store may be a platform provided by using a separate server, where various users may share and / or sell spatial data, where the space is decorated in their own style. Therefore, a spatial map 920-1 corresponding to the user's real space 902 may also be uploaded to the spatial data store to be shared.

[0180] The user can use the electronic device 2000 to view a two-dimensional and / or three-dimensional image of a space in a style preferred by the user from the spatial data storage, and download the preferred spatial map 910. The electronic device 2000 can compare the downloaded preferred spatial map 910 with the spatial map 920-1 corresponding to the user's real space 902, and personalize the spatial map 920-1 corresponding to the user's real space 902.

[0181] In an embodiment, the electronic device 2000 may obtain the preferred spatial map 910 by using short-range / long-range wireless communication (eg, Bluetooth or Wi-Fi Direct) between the electronic device 2000 and another electronic device.

[0182] Figure 9b is a diagram illustrating an operation of outputting personalized recommendation information about a change in placement of an object in a space based on a personalized space map, performed by an electronic device according to an embodiment of the present disclosure.

[0183] In an embodiment of the present disclosure, the electronic device 2000 may compare the preferred spatial graph 910 with a spatial graph 920-1 (hereinafter, for ease of description, referred to as "spatial graph 920-1") corresponding to the user's real space 902. The electronic device 2000 may obtain a personalized spatial graph 920-2 by performing a graph-level comparison / analysis of similarities, differences, etc. between nodes and edges of the preferred spatial graph 910 and the spatial graph 920-1 corresponding to the user's real space 902, and updating the spatial graph 920-1. The personalized spatial graph 920-2 may be a spatial graph obtained by reflecting the features of the preferred spatial graph 910 in the spatial graph 920-1.

[0184] In an embodiment of the present disclosure, the electronic device 2000 may output personalized recommendation information related to a change in placement of an object in the user's real space 902 based on the personalized space map 920 - 2 .

[0185] Personalized recommendation information may include at least one of recommendations to change the positions of at least some of the objects present in the user's real space 902, recommendations to replace any one of the objects present in the user's real space 902 with another object, recommendations to newly place another object, or recommendations for the style of the user's real space 902, but is not limited thereto.

[0186] For example, the personalized spatial map 920-2 may include nodes corresponding to the newly added objects (e.g., TV node 922 and home audio node 924-1). The electronic device 2000 may recommend changing the positions of at least some objects present in the user's real space 902 based on the personalized spatial map 920-2. In detail, based on including the TV node 922 in the personalized spatial map 920-2, the electronic device 2000 may recommend changing the position of the sofa to a position suitable for watching TV, considering the possibility that the TV is placed in the user's real space 902. As another example, the electronic device 2000 may recommend switching the position of an object present in the user's real space 902. However, the present disclosure is not limited to the above examples, and the electronic device 2000 may provide the user with various recommendable ways of changing the position of an object based on the personalized spatial map 920-2.

[0187] For example, among the nodes included in the preferred spatial graph 910, the sofa node 912-1 may include a feature vector 914-1, which is "color: navy blue" and "style: modern, minimalist". In this case, when the spatial graph 920-1 is updated to the personalized spatial graph 920-2, the personalized spatial graph 920-2 may include the updated feature vector 914-2 of the sofa node. The electronic device 2000 may recommend changing the color of the sofa existing in the user's real space 902 to a navy blue color based on the feature vector 914-2 of the sofa node of the personalized spatial graph 920-2.

[0188] For example, the TV node and the home audio node are included in the preference space map 910, but the TV node and the home audio node may not be included in the space map 920-1. In this case, when the space map 920-1 is updated to the personalized space map 920-2, the newly added TV node 922 and the home audio node 924-1 may be included in the personalized space map 920-2. The electronic device 2000 can recommend the TV and home audio system as new objects to be placed in the user's real space 902 based on the feature vector of the newly added node of the personalized space map 920-2. In this case, the electronic device 2000 can further recommend the location where the new object is to be placed, the positional relationship between the new object and other existing objects, the style of the new object, etc. In detail, based on the feature vector 924-2 of the newly added home audio node 924-1, the electronic device 2000 can recommend placing a white modern style product as a newly added home audio system.

[0189] For example, although not in Figure 9b , but the preferred spatial graph 910 may include nodes corresponding to the spatial types. In detail, the preferred spatial graph 910 may include a node corresponding to the spatial type "living room", and the spatial feature vector 918-1 may be included in the living room node. In this case, when the spatial graph 920-1 is updated to the personalized spatial graph 920-2, the node (not shown) of the personalized spatial graph corresponding to the "living room" may include the updated spatial feature vector 918-2. The electronic device 2000 may recommend the overall spatial style of the user's real space 902 based on the spatial feature vector of the personalized spatial graph 920-2. In detail, the electronic device 2000 may recommend an interior design with a modern style and navy blue and white for the user's real space 902. That is, the electronic device 2000 may provide the user with a recommendation for making the user's real space 902 similar to the style space 901 preferred by the user in terms of object placement, spatial color, interior design, etc.

[0190] The personalized space map 920-2 may be continuously updated. For example, when a second preference space image is obtained after the space map 920-1 is updated to the personalized space map 920-2 when the first preference space image is obtained, the personalized space map 920-2 may be additionally updated based on the second preference image.

[0191] Fig.10a is a diagram illustrating an operation of generating a metaverse spatial graph performed by an electronic device according to an embodiment of the present disclosure.

[0192] In the description Fig.10a and Fig.10bIn order to distinguish the metaverse objects currently existing in the metaverse space from the metaverse objects to be newly placed, the metaverse objects currently existing in the metaverse space will be referred to as the first metaverse objects and the new objects will be referred to as the second metaverse objects.

[0193] In an embodiment of the present disclosure, a user of the electronic device 2000 may perform activities in a virtual world such as the metaverse space 1002. For example, the user may create his / her own metaverse space 1002, decorate the metaverse space 1002, and interact with the metaverse space 1002. In this case, the electronic device 2000 may apply the above-described embodiment to the metaverse space 1002 to provide the user with information related to changes in object placement of the metaverse space 1002.

[0194] In the present disclosure, the metaverse space 1002 refers to a space in which a separate virtual world different from the real space is implemented as a virtual environment. That is, the physical space layout of the metaverse space 1002 may have unique characteristics different from those of the real space. For example, it may not be possible to place certain objects (e.g., vases) on the wall surface, ceiling, etc. in the real space, while it may be possible to place those certain objects on the wall surface, ceiling, etc. in the metaverse space 1002.

[0195] In an embodiment of the present disclosure, the metaverse space 1002 may be generated by the electronic device 2000. For example, the electronic device 2000 may generate the metaverse space 1002 based on metaverse space-related data (e.g., metaverse space data and metaverse objects) stored in the electronic device 2000 or received from the outside. The electronic device 2000 may display the layout of the metaverse space 1002 on the screen, and generate the metaverse space 1002 based on user input to the displayed metaverse space 1002 (e.g., for changing the metaverse space layout or object placement).

[0196] In an embodiment of the present disclosure, the electronic device 2000 may obtain a metaverse space map 1020 corresponding to the metaverse space 1002. For example, the data corresponding to the metaverse space 1002 may include metaverse space data and object-related data of a metaverse object. The electronic device 2000 may obtain metaverse space data representing the metaverse space 1002 and object-related data of a first metaverse object in the metaverse space.

[0197] The metaverse space graph 1020 may reflect the characteristics of the metaverse space, where the physical space layout is different from the layout of the real space. For example, in the metaverse space graph 1020, the desk node 1022 corresponding to the object "desk" existing indoors and the tree node 1024 corresponding to the object "tree" existing outdoors may be adjacent nodes connected to each other by edges.

[0198] In an embodiment of the present disclosure, the electronic device 2000 may update the metaverse space map 1020 by applying the metaverse space map to the GNN. When the metaverse space map 1020 is updated, the electronic device 2000 may reflect the characteristics of the user's real space 1001. To this end, the electronic device 2000 may train a space map 1010 corresponding to the user's real space 1001 by using the GNN to which the metaverse space map 1020 is input. The space map 1010 corresponding to the user's real space 1001 may be generated based on the three-dimensional space data and the object-related data, which is described above, and therefore, redundant descriptions will be omitted.

[0199] The electronic device 2000 may apply the space map 1010 corresponding to the user's real space 1001 to the GNN as training data so that the GNN learns the characteristics of the user's real space 1001 .

[0200] The following will refer to Fig.10b The operations performed by the electronic device 2000 to obtain and use an updated metaverse spatial graph by applying the metaverse spatial graph 1020 to the GNN are further described.

[0201] Fig.10b is a diagram illustrating an operation of updating a metaverse space map performed by an electronic device according to an embodiment of the present disclosure.

[0202] In an embodiment of the present disclosure, the electronic device 2000 may update the metaverse spatial map 1020 by using the GNN 1030. The GNN 1030 may have been trained based on a training data set 1032 including a plurality of spatial maps. In this case, the spatial map 1010 corresponding to the user's real space 1001 may be included in the training data set 1032, so that a result reflecting the characteristics of the user's real space 1001 is output from the GNN 1030. The electronic device 2000 may train the GNN 1030 by using the training data set 1032 including the spatial map 1010 corresponding to the user's real space 1001.

[0203] In an embodiment of the present disclosure, the electronic device 2000 may receive a user input for placing a second metaverse object in the metaverse space. The electronic device 2000 may add an empty node to the metaverse space graph 1020 based on the user input and perform node embedding. For example, a tree node 1024 may be included in the metaverse space graph, and an empty node may be added adjacent to the tree node 1024 based on the user input. Thereafter, as a result of node embedding performed by the electronic device 2000 using the GNN 1030, the empty node may become a tea table node 1025. The tea table node 1025 may be a node embedded with the object feature vector 1026.

[0204] In detail, the user input may be a position input for inputting a position adjacent to a tree in the metaverse space 1002. In this case, when an empty node is added and node embedding is performed according to the above embodiment, the tea table node 1025 may be generated to be adjacent to the tree node 1024. The user input may be an object input for inputting a tea table object in the metaverse space 1002. In this case, when an empty node is added and node embedding is performed according to the above embodiment, the tea table node 1025 may be generated to be adjacent to the tree node 1024.

[0205] The updated metaverse space map 1020 is a space map that reflects the user's preferences while having no spatial and physical constraints. The electronic device 2000 may output information for placing a second metaverse object in the metaverse space 1002 based on the updated metaverse space map 1020. For example, the electronic device 2000 may recommend placing a tea table next to a tree in the metaverse space 1002 based on the feature vector 1026 of the tea table node 1025 of the updated metaverse space map 1020. In this case, based on the feature vector 1026 of the tea table node 1025, the electronic device 2000 may recommend placing a tea table with a natural style and beige color to float, but the present disclosure is not limited thereto.

[0206] When a user wants to organize a space within a virtual environment based on metaverse features, the electronic device 2000 according to an embodiment can provide the user with a metaverse space design that reflects the features of all users' preferences and unique features of the metaverse space (for example, removing the exterior walls of a building) by using the metaverse space map 1020 and the user's space map 1010.

[0207] Fig.11 is a diagram illustrating an operation of recommending information related to object placement based on a feature of a real space, performed by an electronic device according to an embodiment of the present disclosure.

[0208] In an embodiment of the present disclosure, the three-dimensional spatial data may further include data related to the spatial design specifications. The data related to the spatial design specifications may include, for example, the location of a wall socket, the location of a lamp, the location of a door, the location of a communication port, the location of a water supply unit, etc., but is not limited thereto. Since the three-dimensional spatial data includes data related to the spatial design specifications, the spatial graph generated based on the three-dimensional spatial data may also include data related to the spatial design specifications. The electronic device 2000 may train the GNN by using a spatial graph including data related to the spatial design specifications, and use the trained spatial graph.

[0209] In an embodiment of the present disclosure, the electronic device 2000 may output information related to the object placement change of the space based on the properties of the object and the design specifications of the space. For example, when the information related to the object placement change of the space is generated according to the above-mentioned embodiment, the electronic device 2000 may further generate information related to the object placement change of the space based on data related to the design specifications of the space. That is, when the electronic device 2000 receives user input (e.g., object input and / or position input) and applies the spatial graph to the GNN, information related to the object placement change reflecting the design specifications of the space and the properties of the object may be generated.

[0210] For example, the TV 1110 is a home appliance that needs power. In this case, the location 1120 of the wall socket included in the data related to the design specifications of the space can be used. The data representing the attributes of the TV 1110 may include information that the TV 1110 is a home appliance that needs power.

[0211] For example, when the electronic device 2000 provides information for placing the TV 1110 in the space, the location 1120 of the wall socket may be reflected. In detail, the electronic device 2000 may recommend placing the TV 1110 at a location within a certain distance from the location 1120 of the wall socket.

[0212] For example, when the electronic device 2000 provides information for placing the TV 1000 in a space, information about the location 1120 of the wall socket may be provided together. In detail, when the electronic device 2000 recommends that the TV 1110 be placed at a location farther than a specific distance from the location 1120 of the wall socket, the electronic device 2000 may further recommend that the TV 1110 be powered by using a power strip.

[0213] As an additional example, the sofa 1130 is a piece of furniture that does not require power supply. In this case, the door position 1140 included in the data related to the design specifications of the space can be used. The data representing the attributes of the sofa 1130 may include information that the sofa 1130 is a piece of furniture that needs to be positioned so as not to overlap with the door position 1140 when the sofa 1130 is placed in direct contact with the wall.

[0214] For example, when the electronic device 2000 provides information for placing the sofa 1130 in the space, the position of the door 1140 may be reflected. In detail, the electronic device 2000 may recommend placing the sofa 1130 at a position that does not interfere with the opening / closing of the door.

[0215] For example, when the electronic device 2000 provides information for placing the sofa 1130 in the space, information about the object associated with the sofa 1130 may be reflected. In detail, the object associated with the sofa 1130 may be the TV 1110. When recommending the position where the sofa 1130 is to be placed, the electronic device 2000 may recommend placing the sofa 1130 to face the TV 1110. Based on the screen size of the TV 1110, the electronic device 2000 may recommend a separation distance between the sofa 1130 and the TV 1110 (e.g., 2 meters for a 50-inch TV, or 3 meters for a 75-inch TV). Based on the resolution of the TV 1110, the electronic device 2000 may recommend a separation distance between the sofa 1130 and the TV 1110 (e.g., 3 meters for a 50-inch FHD TV, or 2 meters for a 75-inch UHD TV).

[0216] The above examples are not to be implemented independently, and all possible combinations of the above examples may be performed.

[0217] According to an embodiment of the present disclosure, the electronic device 2000 can perform calculations to place various types of objects in optimal locations in the space based on the design specifications of the space and the attributes of the objects, and provide recommendations to the user based on the calculation results, thereby accurately and conveniently assisting the user in the interior design of the real space.

[0218] Fig.12 is a block diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure.

[0219] The electronic device 2000 according to an embodiment of the present disclosure may include a communication interface 2100 , a display 2200 , a memory 2300 , and a processor 2400 .

[0220] The communication interface 2100 may perform data communication with other electronic devices under the control of the processor 2400 .

[0221] The communication interface 2100 may include a communication circuit. The communication interface 2100 may include a communication circuit capable of performing data communication between the electronic device 2000 and other devices by using at least one of a data communication scheme (e.g., a wired local area network (LAN), a wireless LAN, Wi-Fi, Bluetooth, ZigBee, Wi-Fi Direct (WFD), an infrared data association (IrDA), Bluetooth low energy (BLE), near field communication (NFC), wireless broadband (WiBro), world interoperability for microwave access (WiMAX), shared wireless access protocol (SWAP), wireless Gigabit Alliance (WiGig), or radio frequency (RF) communication).

[0222] According to an embodiment of the present disclosure, the communication interface 2100 can send data for controlling the operation of the electronic device 2000 to an external electronic device and receive data for controlling the operation of the electronic device 2000 from an external electronic device. For example, the communication interface 2100 can send and receive an artificial intelligence model (e.g., an object detection model, an object recognition model, a graph generation model, a graph neural network, or a scene classification model) used by the electronic device 2000 to a server, etc. In addition, the electronic device 2000 can receive three-dimensional space data and object-related data from a server, etc. In addition, the electronic device 2000 can receive a spatial map corresponding to a space from a server, etc. In addition, the electronic device 2000 can send various data for generating / displaying a virtual space / metaverse space to a server, etc., and receive various data for generating / displaying a virtual space / displaying a virtual space / metaverse space from a server, etc.

[0223] The display 2200 may output information processed by the electronic device 2000. Meanwhile, in the case where the display and the touch panel constitute a layer structure to form a touch screen, the display may be used as an input device in addition to an output device. The display may include at least one of a liquid crystal display (LCD), a thin film transistor liquid crystal display (TFT LCD), an organic light emitting diode (OLED) display, a flexible display, a three-dimensional display, a micro display, or a head mounted display (HMD).

[0224] The memory 2300 may store instructions, data structures, and program codes that can be read by the processor 2400. A plurality of memories 2300 may be provided. In an embodiment of the present disclosure, the operations performed by the processor 2400 may be implemented by executing instructions or codes of a program stored in the memory 2300.

[0225] Memory 2300 may include nonvolatile memory, such as read-only memory (ROM) (e.g., programmable ROM (PROM), erasable programmable ROM (EPROM), or electrically erasable programmable ROM (EEPROM)), flash memory (e.g., memory card, solid-state drive (SSD)), or analog recording type memory (e.g., hard disk drive (HDD), magnetic tape, or optical disk), or volatile memory, such as random access memory (RAM) (e.g., dynamic RAM (DRAM) or static RAM (SRAM)).

[0226] The memory 2300 according to an embodiment of the present disclosure may store one or more instructions and programs for operating the electronic device 2000 to provide spatial object placement change related information. For example, the memory 2300 may store a data processing module 2310, a graph generation module 2320, and an artificial intelligence module 2330.

[0227] The processor 2400 may control the overall operation of the electronic device 2000. For example, the processor 2400 may execute one or more instructions of a program stored in the memory 2300 to control the overall operation of the electronic device 2000 for providing spatial object placement change related information. A plurality of processors 2400 may be provided.

[0228] According to the present disclosure, one or more processors 2400 may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), an integrated many-core (MIC) processor, a digital signal processor (DSP), or a neural processing unit (NPU). One or more processors 2400 may be implemented in the form of an integrated system on chip (SoC) including one or more electronic components. Each of the one or more processors may be implemented as separate hardware (H / W).

[0229] The processor 2400 can generally process data related to object placement changes by using the data processing module 2310. For example, the processor 2400 can store three-dimensional spatial data obtained by performing a three-dimensional scan on the space, as well as object-related data, and perform preprocessing for generating a spatial map. By using the data processing module 2310, the processor 2400 can manage the spatial map generated for each space (e.g., a room or a living room), and store the spatial map updated by performing calculations for updating the spatial map. The processor 2400 can process the obtained / received user input by using the data processing module 2310. The processor 2400 can generate a virtual space corresponding to the space by using the data processing module 2310, and display the virtual space and virtual objects on the display 2200. The detailed operations related to the data processing module 2310 are described in detail above with reference to the accompanying drawings, and therefore, redundant descriptions will be omitted.

[0230] The processor 2400 may execute the graph generation module 2320 to generate a spatial graph corresponding to the space. The processor 2400 may generate the spatial graph based on the three-dimensional spatial data and the object-related data. The processor 2400 may generate the spatial graph based on a user input to a user interface for generating a spatial graph. The detailed operations related to the graph generation module 2320 are described in detail above with reference to the accompanying drawings, and therefore, redundant descriptions will be omitted.

[0231] By using the artificial intelligence module 2330, the processor 2400 can execute various artificial intelligence models for generating information related to the object placement change of the space, and process data obtained from each of the various artificial intelligence models. The artificial intelligence module 2330 may include one or more artificial intelligence models. For example, the artificial intelligence module 2330 may include an object detection model, an object recognition model, a graph generation model, a graph neural network, a scene classification model, etc., but is not limited thereto.

[0232] The above modules stored in the memory 2300 and executed by the processor 2400 are provided for convenience of description, and the present disclosure is not necessarily limited thereto. Other modules may be added to implement the above embodiments, one module may be divided into multiple separate modules according to detailed functions, and some of the above modules may be combined to implement as one module.

[0233] In the case where the method according to an embodiment of the present disclosure includes multiple operations, the multiple operations can be performed by one processor or multiple processors. For example, when the first operation, the second operation, and the third operation are performed by the method according to an embodiment of the present disclosure, the first operation, the second operation, and the third operation can all be performed by the first processor, and the first operation and the second operation can be performed by the first processor (e.g., a general-purpose processor), and the third operation can be performed by the second processor (e.g., a dedicated artificial intelligence processor). Here, a dedicated artificial intelligence processor as an example of a second processor can perform operations for learning / reasoning of an artificial intelligence model. However, embodiments of the present disclosure are not limited to this.

[0234] One or more processors according to the present disclosure may be implemented as a single-core processor or a multi-core processor.

[0235] In the case where the method according to the embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one core or a plurality of cores included in one or more processors.

[0236] The electronic device 2000 may further include one or more cameras 2500 and / or one or more sensors 2600. The electronic device 2000 may perform a three-dimensional scan of a space by using the one or more cameras 2500 and / or the one or more sensors 2600. For example, the one or more cameras 2500 and / or the one or more sensors 2600 may include an RGB-D sensor, a ToF sensor, a LiDAR sensor, a RADAR sensor, etc., but are not limited thereto.

[0237] The electronic device 2000 may further include an input / output interface 2700. The input / output interface 2700 may include an input interface for receiving an input from a user, and an output interface for outputting a signal other than an image / video signal output from the display 2200.

[0238] The input interface is used to receive input from the user. The input interface may include, but is not limited to, at least one of a keypad, a dome switch, a touch pad (e.g., a touch-type capacitive touch pad, a pressure-type resistive cover touch pad, an infrared sensor touch pad, a surface acoustic wave conduction touch pad, an integrated tension measurement touch pad, a piezoelectric effect touch pad), a jog wheel, or a knob switch.

[0239] The input interface may include a microphone, and therefore, the speech recognition module may be stored in the memory of the electronic device 2000. The electronic device 2000 may receive a speech signal as an analog signal through a microphone, and process the speech signal by using a speech recognition module. For example, the electronic device 2000 may convert the speech portion into a computer-readable text by using an ASR model. The electronic device 2000 may interpret the text by using a natural language understanding (NLU) model to obtain the user's utterance intention. Here, the ASR model or the NLU model may be an artificial intelligence model. Language understanding is a technology for recognizing and applying / processing human language / characters, and may include natural language processing, machine translation, a dialogue system, question-answering, speech recognition / synthesis, and the like.

[0240] The output interface may include a speaker. The speaker may output an audio signal received from the communication interface 2100 or stored in the memory 2300.

[0241] Fig.13 is a block diagram showing a configuration of a server according to an embodiment of the present disclosure.

[0242] In an embodiment of the present disclosure, at least some of the above operations of the electronic device 2000 may be performed by the server 3000 .

[0243] The server 3000 according to an embodiment of the present disclosure may include a communication interface 3100, a memory 3200, and a processor 3300. The communication interface 3100, the memory 3200, and the processor 3300 of the server 3000 correspond to Fig.12 The communication interface 2100, the memory 2300, and the processor 2400 of the electronic device 2000 shown in FIG. 2 are shown in FIG. 1 , and therefore, redundant descriptions will be omitted for the sake of brevity.

[0244] The server 3000 according to an embodiment of the present disclosure may be a device having a higher computing performance than the computing performance of the electronic device 2000 and thus capable of performing a larger amount of computing. The server 3000 may perform training of an artificial intelligence model, which requires a relatively larger amount of computing than reasoning. The server 3000 may perform reasoning by using an artificial intelligence model and send the result of the reasoning to the electronic device 2000.

[0245] The present disclosure describes a method for inferring the category of an object to be placed in a space and / or the position of an object to be placed in a space by using a GNN and a spatial graph including objects in the space and positional relationships between objects in the space, so as to provide a user with information related to object placement changes in the space.

[0246] The technical objectives of the present disclosure are not limited to those mentioned above, and other technical objectives not mentioned herein can be clearly understood by those skilled in the art from the following description.

[0247] According to one aspect of the present disclosure, a method for providing information related to placing an object in a space performed by an electronic device may be provided. The method may include obtaining three-dimensional spatial data corresponding to the space and object-related data of a first object in the space. The method may also include: based on the three-dimensional spatial data and the object-related data of the first object, obtaining a spatial graph including a positional relationship between the first objects in the space, the spatial graph including nodes corresponding to the attributes of the first object and edges representing the positional relationship between the first objects. The method may also include receiving user input for changing the placement of objects in the space. The method may also include: based on the user input, adding an empty node to the spatial graph, the empty node representing a second object to be placed in a blank area in the space where the first object is not placed. The method may also include updating the spatial graph by applying the spatial graph to which the empty node has been added to the GNN. The method may also include outputting information related to the object placement change in the space based on the updated spatial graph.

[0248] Obtaining three-dimensional spatial data corresponding to the space and object-related data of the first object in the space may include obtaining three-dimensional spatial data including a three-dimensional image of the space by performing a three-dimensional scan on the space.

[0249] Obtaining three-dimensional space data corresponding to the space and object-related data of a first object in the space may also include: detecting the first object from a three-dimensional image of the space.

[0250] The spatial graph may include multiple types of spatial graphs.

[0251] Each of the plurality of types of spatial graphs may correspond to each of the plurality of types of positional relationships between the first objects.

[0252] The multiple types of positional relationships may include two or more of a “co-occurrence” positional relationship, a “supporting” positional relationship, a “supported” positional relationship, a “surrounding” positional relationship, a “surrounded by” positional relationship, and a “close to” positional relationship.

[0253] The updating of the spatial graph may include inferring an object feature vector of the empty node by applying the spatial graph including the empty node to the GNN.

[0254] The updating of the spatial graph may also include predicting edges to be connected to empty nodes.

[0255] The inferred object feature vector and the predicted edges may be determined by the GNN based on attributes of the first object.

[0256] Receiving the user input may include receiving an input for selecting a first position in a blank area where the first object is not placed.

[0257] The adding of the empty node may include adding the empty node representing the second object to be adjacent to at least one node corresponding to at least one first object within a certain distance from the first position among the first objects.

[0258] The outputting of the information related to the object placement change of the space may include determining one or more object categories corresponding to the second object that can be placed at the first position based on the inferred object feature vector.

[0259] The outputting of information related to the change in the placement of objects in the space may also include outputting the determined one or more object categories.

[0260] Receiving the user input may include receiving a request for a recommendation for a location for a second object that is not currently placed in the space.

[0261] The adding of the empty node may include adding one or more empty node candidates to the spatial graph.

[0262] The updating of the spatial graph may also include inferring an object feature vector for each of the one or more empty node candidates by applying the spatial graph to the GNN.

[0263] The updating of the spatial graph may also include selecting one or more of the one or more empty node candidates based on the object feature vector of each of the empty node candidates and the object category of the second object.

[0264] The outputting of the information related to the object placement change of the space may include determining one or more locations in the space where the second object is to be placed based on the object feature vectors of the selected one or more nodes.

[0265] The output of information related to the object placement change of the space may also include outputting the determined one or more locations.

[0266] The method may also include obtaining a preferred spatial image including features preferred by the user.

[0267] The method may further include obtaining a spatial map of user preference by using the preferred spatial image.

[0268] The method may further include personalizing the spatial graph by changing at least one of a node or an edge of the spatial graph based on a result of comparing the spatial graph of the user preference with the spatial graph.

[0269] The method may further include outputting personalized recommendation information related to a change in placement of the object in the space based on the personalized space map.

[0270] The personalized recommendation information may include at least one of a recommendation to change the position of at least some of the first objects, a recommendation to replace any of the first objects with a third object, a recommendation to newly place a third object, or a recommendation for a style of the space.

[0271] The method may also include obtaining Metaverse space data representing the Metaverse space and object-related data of a first Metaverse object in the Metaverse space.

[0272] The method may also include obtaining a metaverse space map reflecting the characteristics of a metaverse space whose physical space layout is different from the real space based on the metaverse space data and the object-related data of the first metaverse object.

[0273] The method may also include receiving user input for placing a second metaverse object in the metaverse space.

[0274] The method may also include updating the metaverse spatial map by applying the metaverse spatial map to the GNN so that the metaverse spatial map includes features of the real space.

[0275] GNNs may have further learned a spatial map corresponding to the real space.

[0276] The method may also include outputting information for placing a second Metaverse object in the Metaverse space based on the updated Metaverse space map.

[0277] The generating of the spatial map may include obtaining the spatial map by inputting the three-dimensional spatial data and the object related data of the first object into a spatial map generating model.

[0278] According to an aspect of the present disclosure, an electronic device 2000 for providing information related to placing an object in a space may be provided. The electronic device 2000 may include a display 2200, a memory 2300 storing one or more instructions, and at least one processor 2400 configured to execute one or more instructions stored in the memory 2300. The at least one processor 2400 may also be configured to execute one or more instructions to obtain three-dimensional spatial data corresponding to the space and object-related data of a first object in the space. The at least one processor 2400 may also be configured to execute one or more instructions to obtain a spatial graph including a positional relationship between first objects in the space based on the three-dimensional spatial data and the object-related data of the first object. The spatial graph may include nodes corresponding to the attributes of the first object and edges representing the positional relationship between the first objects. The at least one processor 2400 may also be configured to execute one or more instructions to receive user input for changing the placement of an object in the space. The at least one processor 2400 may also be configured to execute one or more instructions to add an empty node to the spatial graph based on the user input, the empty node representing a second object to be placed in a blank area in the space where the first object is not placed. At least one processor 2400 may also be configured to execute one or more instructions to update the spatial graph by applying the spatial graph to which the empty node has been added to the GNN. At least one processor 2400 may also be configured to execute one or more instructions to output information related to the object placement change of the space through the display based on the updated spatial graph.

[0279] The electronic device 2000 may further include a camera.

[0280] The at least one processor 2400 may also be configured to execute one or more instructions to obtain three-dimensional spatial data including a three-dimensional spatial image of the space by performing a three-dimensional spatial scan of the space using a camera.

[0281] The at least one processor 2400 may also be configured to execute one or more instructions to detect a first object from the three-dimensional image of the space.

[0282] The spatial graph may include multiple types of spatial graphs.

[0283] Each of the plurality of types of spatial graphs may correspond to each of the plurality of types of positional relationships between the first objects.

[0284] The multiple types of positional relationships may include two or more of a “co-occurrence” positional relationship, a “supporting” positional relationship, a “supported” positional relationship, a “surrounding” positional relationship, a “surrounded by” positional relationship, and a “close to” positional relationship.

[0285] At least one processor 2400 may also be configured to execute one or more instructions to infer an object feature vector of an empty node by applying a spatial graph including the empty node to the GNN.

[0286] At least one processor 2400 may also be configured to execute one or more instructions to predict edges to be connected to the empty node.

[0287] The inferred object feature vector and the predicted edges may be determined by the GNN based on attributes of the first object.

[0288] The at least one processor 2400 may also be configured to execute one or more instructions to receive an input for selecting a first position in a blank area where the first object is not placed.

[0289] The at least one processor 2400 may also be configured to execute one or more instructions to add an empty node representing the second object adjacent to at least one node corresponding to at least one first object within a certain distance from the first position among the first objects.

[0290] The at least one processor 2400 may also be configured to execute one or more instructions to determine one or more object categories corresponding to the second object that can be placed at the first location based on the inferred object feature vector.

[0291] The at least one processor 2400 may also be configured to execute one or more instructions to output the determined one or more object categories.

[0292] The at least one processor 2400 may also be configured to execute one or more instructions to receive a request for a recommendation for a location for a second object that is not currently placed in the space.

[0293] The at least one processor 2400 may also be configured to execute one or more instructions to add one or more empty node candidates to the spatial graph.

[0294] At least one processor 2400 may also be configured to execute one or more instructions to infer an object feature vector for each of the one or more empty node candidates by applying the spatial graph to the GNN.

[0295] The at least one processor 2400 may also be configured to execute one or more instructions to select one or more of the one or more empty node candidates based on the object feature vector of each of the empty node candidates and the object category of the second object.

[0296] The at least one processor 2400 may also be configured to execute one or more instructions to determine one or more locations in the space where the second object is to be placed based on the object feature vectors of the selected one or more nodes.

[0297] The at least one processor 2400 may also be configured to execute one or more instructions to output the determined one or more locations.

[0298] The at least one processor 2400 may also be configured to execute one or more instructions to obtain a preferred spatial image including features preferred by the user.

[0299] The at least one processor 2400 may also be configured to execute one or more instructions to obtain a user-preferred spatial map by using the preferred spatial image.

[0300] The at least one processor 2400 may also be configured to execute one or more instructions to personalize the spatial graph by changing at least one of the nodes or edges of the spatial graph based on a result of comparing the spatial graph of the user's preference with the spatial graph.

[0301] The at least one processor 2400 may also be configured to execute one or more instructions to output personalized recommendation information related to changes in placement of objects in the space based on the personalized spatial map.

[0302] The personalized recommendation information may include at least one of a recommendation to change the position of at least some of the first objects, a recommendation to replace any of the first objects with a third object, a recommendation to newly place a third object, or a recommendation for a style of the space.

[0303] At least one processor 2400 may also be configured to execute one or more instructions to obtain metaverse space data representing the metaverse space and object-related data of a first metaverse object in the metaverse space.

[0304] At least one processor 2400 can also be configured to execute one or more instructions to obtain a metaverse space map reflecting the characteristics of the metaverse space whose physical space layout is different from the real space based on the metaverse space data and the object-related data of the first object of the metaverse.

[0305] The at least one processor 2400 may also be configured to execute one or more instructions to receive user input for placing a second Metaverse object in the Metaverse space.

[0306] At least one processor 2400 may also be configured to execute one or more instructions to update the metaverse spatial map by applying the metaverse spatial map to the GNN so that the metaverse spatial map includes features of the real space.

[0307] GNNs may have further learned a spatial map corresponding to the real space.

[0308] At least one processor 2400 may also be configured to execute one or more instructions to output information for placing a second Metaverse object in the Metaverse space based on the updated Metaverse space map.

[0309] Embodiments of the present disclosure may be implemented as a recording medium including computer executable instructions such as computer executable program modules. Computer readable media may be any available media that can be accessed by a computer, and may include volatile or non-volatile media and removable or non-removable media. Computer readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. Communication media may typically include computer readable instructions, data structures or other data such as modulated data signals of program modules.

[0310] The computer-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" refers to a tangible device and does not include a signal (e.g., an electromagnetic wave), and the term "non-transitory storage medium" does not distinguish between a case where data is semi-permanently stored in the storage medium and a case where data is temporarily stored. For example, a non-transitory storage medium may include a buffer in which data is temporarily stored.

[0311] According to an embodiment of the present disclosure, the method according to various embodiments disclosed herein may be included in a computer program product and then provided. The computer program product may be traded between a seller and a buyer as a commodity. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc (CD) ROM (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or distributed online directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored in a machine-readable storage medium, such as a memory of a manufacturer's server, an application store's server, or a relay server.

[0312] Although the present disclosure has been specifically shown and described, it will be appreciated by those skilled in the art that various changes may be made in form and detail without departing from the spirit and scope of the present disclosure. Therefore, it should be understood that the above embodiments do not limit the scope of the present disclosure. For example, each element described in a single type may be performed in a distributed manner, and the elements described in a distributed manner may also be performed in an integrated form.

[0313] The scope of the present disclosure is not limited by the detailed description of the present disclosure but by the appended claims, and all modifications or substitutions derived from the scope and spirit of the claims and their equivalents fall within the scope of the present disclosure.

Claims

1. A method performed by an electronic device (2000), the method comprising: Obtaining three-dimensional space data corresponding to a space and object-related data of a plurality of first objects in the space (S210); Based on the three-dimensional spatial data and the object-related data, a spatial graph including positional relationships between the plurality of first objects in the space is obtained, the spatial graph including nodes corresponding to attributes of the plurality of first objects and edges representing positional relationships between the plurality of first objects (S220); receiving a user input for changing the placement of the object in the space (S230); Based on the user input, adding an empty node to the space graph, the empty node representing a second object to be placed in a blank area in the space where the plurality of first objects are not placed (S240); updating the spatial graph by applying the spatial graph to which the empty node has been added to a graph neural network (GNN) (S250); as well as Based on the updated space map, information related to the object placement change of the space is output (S260).

2. The method according to claim 1, wherein: The spatial graph includes multiple types of spatial graphs. Each of the plurality of types of spatial diagrams corresponds to each of the plurality of types of positional relationships between the plurality of first objects, and The multiple types of positional relationships include two or more of a "co-occurrence" positional relationship, a "supporting" positional relationship, a "supported" positional relationship, a "surrounding" positional relationship, a "surrounded by" positional relationship, and a "close to" positional relationship.

3. The method according to any one of claims 1 to 2, wherein: Updating the spatial map includes: Inferring a plurality of object feature vectors of the empty node by applying the spatial graph including the empty node to the GNN; and predict the edges that are to be connected to the empty node, and The multiple object feature vectors and the edges are determined by the GNN based on the attributes of the multiple first objects.

4. The method according to claim 3, wherein: Receiving the user input includes: receiving an input for selecting a first position in the blank area where the plurality of first objects are not placed, Adding the empty node includes adding the empty node adjacent to at least one node, the at least one node corresponding to at least one first object within a certain distance from the first position among the plurality of first objects, and Outputting the information related to the object placement change in the space includes: determining one or more object categories corresponding to the second object that can be placed at the first location based on the plurality of object feature vectors; and The one or more object categories are output.

5. The method according to claim 3, wherein: receiving the user input includes receiving a request for a recommendation for a location for the second object that is not currently placed in the space, Adding the empty node includes adding one or more empty node candidates to the spatial graph, Updating the spatial map further includes: inferring a plurality of object feature vectors for each of the one or more empty node candidates by applying the spatial graph to the GNN; and selecting one or more nodes of the one or more empty node candidates based on the plurality of object feature vectors of the one or more empty node candidates and the object category of the second object; and Outputting the information related to the object placement change in the space includes: determining one or more locations in the space where the second object is to be placed based on the plurality of object feature vectors of the selected one or more nodes; and The one or more locations are output.

6. The method according to any one of claims 1 to 2, further comprising: obtaining a spatial image including features preferred by the user; obtaining a spatial map of user preference based on the spatial image; obtaining a personalized spatial graph by changing at least one of the nodes or the edges of the spatial graph based on a result of comparing the spatial graph of the user preference with the spatial graph; as well as Based on the personalized space map, personalized recommendation information related to the change of the placement of the object in the space is output.

7. The method according to claim 6, wherein: The personalized recommendation information includes at least one of a recommendation to change the position of at least some of the multiple first objects, a recommendation to replace any one of the multiple first objects with a third object, a recommendation to newly place the third object, or a recommendation for the style of the space.

8. An electronic device (2000) for providing information related to placing an object in a space, the electronic device comprising: display(2200); A memory (2300) storing one or more instructions; and At least one processor (2400) is configured to execute the one or more instructions stored in the memory (2300) to: obtaining three-dimensional spatial data corresponding to a space and object-related data of a plurality of first objects in the space, Based on the three-dimensional spatial data and the object-related data, a spatial graph including positional relationships between the plurality of first objects in the space is obtained, the spatial graph including nodes corresponding to attributes of the plurality of first objects and edges representing positional relationships between the plurality of first objects. receiving user input for changing the placement of the object in the space, adding an empty node to the space graph based on the user input, the empty node representing a second object to be placed in an empty area in the space where the plurality of first objects are not placed, updating the spatial graph by applying the spatial graph to which the empty node has been added to a graph neural network (GNN), and Based on the updated spatial map, information related to the object placement change of the space is outputted through the display.

9. The electronic device according to claim 8, wherein: The spatial graph includes multiple types of spatial graphs. Each of the plurality of types of spatial diagrams corresponds to each of the plurality of types of positional relationships between the plurality of first objects, and The multiple types of positional relationships include two or more of a "co-occurrence" positional relationship, a "supporting" positional relationship, a "supported" positional relationship, a "surrounding" positional relationship, a "surrounded by" positional relationship, and a "close to" positional relationship.

10. The electronic device according to any one of claims 8 to 9, wherein: The at least one processor is further configured to execute the one or more instructions to infer a plurality of object feature vectors of the empty node by applying the spatial graph including the empty node to the GNN, and predict edges to be connected to the empty node, and The multiple object feature vectors and the predicted edges are determined by the GNN based on the attributes of the multiple first objects.

11. The electronic device according to claim 10, wherein: The at least one processor is further configured to execute the one or more instructions to: receiving an input for selecting a first position in the blank area where the plurality of first objects are not placed, adding the empty node adjacent to at least one node, wherein the at least one node corresponds to at least one first object within a certain distance from the first position among the plurality of first objects, One or more object categories corresponding to the second object that can be placed at the first position are determined based on the plurality of object feature vectors, and the one or more object categories are output.

12. The electronic device according to claim 10, wherein: The at least one processor is further configured to execute the one or more instructions to: receiving a request for a recommendation for a location for the second object that is not currently placed in the space, adding one or more empty node candidates to the spatial graph, inferring a plurality of object feature vectors for each of the one or more empty node candidates by applying the spatial graph to the GNN, selecting one or more nodes of the one or more empty node candidates based on the plurality of object feature vectors of the one or more empty node candidates and the object category of the second object, determining one or more locations in the space where the second object is to be placed based on the plurality of object feature vectors of the selected one or more nodes, and The one or more locations are output.

13. The electronic device according to any one of claims 8 to 9, wherein: The at least one processor is further configured to execute the one or more instructions to: obtaining a spatial image including features preferred by the user, obtaining a spatial map of user preference based on the spatial image, obtaining a personalized spatial graph by changing at least one of the nodes or the edges of the spatial graph based on a result of comparing the user preference spatial graph with the spatial graph, and Based on the personalized space map, personalized recommendation information related to the change of the placement of the object in the space is output.

14. The electronic device according to claim 13, wherein: The personalized recommendation information includes at least one of a recommendation to change the position of at least some of the multiple first objects, a recommendation to replace any one of the multiple first objects with a third object, a recommendation to newly place the third object, or a recommendation for the style of the space. 15 . A computer-readable recording medium having recorded thereon a program for causing a computer to execute the method according to claim 1 .

Images