Method and server for generating spatial map

Abstract

A method for a server to provide a personalized space map may comprise the steps of: acquiring a first space map generated in advance and including a first map feature representing space information about a space; obtaining a second spatial diagram generated based on the sensor data of the spatial measurement, the second spatial diagram including a second diagram feature; selecting a graph feature for graph alignment from the first graph feature and the second graph feature; aligning the first spatial diagram and the second spatial diagram based on the selected diagram features; identifying a local region in the space where there is a difference between the first space map and the second space map; and generating a personalized spatial map by determining spatial information about the local region based on the second map feature.

Description

Technical Field

[0001] The present disclosure relates to a server for providing a spatial graph including spatial information optimized for a user by using a graph analysis algorithm and an operation method thereof. Background Art

[0002] Various technologies / processes have been developed to represent a user space as a virtual space by using virtual reality (VR), augmented reality (AR), computer graphics, etc. A virtual space implemented to create an immersive experience can be used in various ways, such as real estate sales and interior design for home appliances and furniture.

[0003] As an example for providing an immersive virtual space corresponding to a user space, a space map may be used. In this case, a plurality of space maps corresponding to the user space may be used. For example, there may be a common space map corresponding to the user space, but the space map does not include detail changes in the user's real-world space. That is, since the spatial information included in the space map differs depending on the time when the space map is generated, etc., there is a limitation in reflecting the user's real-world space as it is.

[0004] To address the above shortcomings, a spatial graph generation / update algorithm can be used to reflect personal spatial information, such as the actual layout of the user's space and real-world objects. Summary of the invention

[0005] According to one aspect of the present disclosure, a method for providing a personalized spatial map performed by a server includes: obtaining a first spatial map including a first map feature representing spatial information about a space; obtaining a second spatial map based on sensor data obtained by measuring the space, wherein the second spatial map includes a second map feature; identifying a third map feature for map alignment from the first map feature and the second map feature; aligning the first spatial map and the second spatial map based on the identified third map feature; identifying a local area in the space where there is at least one difference between the first spatial map and the second spatial map; and generating a personalized spatial map by identifying multiple pieces of spatial information about the local area based on the second map feature.

[0006] The graph features including the first graph feature and the second graph feature may include at least one of spatial layout information, information related to an object in a space, or spatial three-dimensional (3D) information.

[0007] The method may include projecting a higher-dimensional spatial map of the first spatial map and the second spatial map to correspond to a spatial dimension of a lower-dimensional spatial map based on the spatial dimension of the first spatial map being different from the spatial dimension of the second spatial map.

[0008] Identifying a third map feature for map alignment may include: marking a first map feature and a second map feature with multiple feature attribute information, and wherein, aligning the first spatial map and the second spatial map may include: aligning the first spatial map and the second spatial map based on the marked first map feature and the marked second map feature so that the first spatial map and the second spatial map overlap.

[0009] Aligning the first spatial graph and the second spatial graph may include: identifying a confidence score representing the similarity between each first graph feature in the first spatial graph and each second graph feature corresponding to each first graph feature in the second spatial graph; and removing fourth graph features having a confidence score less than a predetermined threshold from the first graph features and the second graph features.

[0010] Identifying a local area in a space where there is at least one difference between a first spatial map and a second spatial map may include: comparing the first spatial map with the second spatial map; and classifying the local area that has changed in the second spatial map relative to the first spatial map, and wherein generating a personalized spatial map may include: generating a personalized spatial map by identifying multiple pieces of spatial information about the local area based on second map features and the classified local area.

[0011] The changed local region may be classified as at least one of a local region where the object is changed or a local region containing measurement noise.

[0012] Identifying a local region in the space where at least one difference exists between the first spatial map and the second spatial map may include identifying the local region by using a spatial change detection model that takes the first spatial map and the second spatial map as input and outputs information about the local region.

[0013] The method may include obtaining a 3D spatial map of at least one local area among a plurality of local areas in the space; and updating spatial information about the at least one local area within the personalized spatial map based on a fifth map feature in the 3D spatial map.

[0014] The method may include generating a personalized virtual space corresponding to the personalized space map based on the personalized space map.

[0015] According to one aspect of the present disclosure, an electronic device for providing a personalized spatial map includes: at least one memory configured to store one or more instructions; and at least one processor configured to execute one or more instructions stored in the at least one memory to perform the following operations: obtain a first spatial map including a first map feature representing spatial information about a space; obtain a second spatial map based on sensor data obtained by measuring the space, wherein the second spatial map includes a second map feature; identify a third map feature for map alignment from the first map feature and the second map feature; align the first spatial map and the second spatial map based on the identified third map feature; identify a local area in the space where there is at least one difference between the first spatial map and the second spatial map; and generate a personalized spatial map by identifying multiple pieces of spatial information about the local area based on the second map feature.

[0016] According to one aspect of the present disclosure, a server for providing a personalized spatial map includes: a communication interface; at least one memory configured to store one or more instructions; and at least one processor configured to execute one or more instructions stored in the at least one memory to perform the following operations: obtaining a first spatial map including a first map feature representing spatial information about a space via the communication interface; obtaining a second spatial map based on sensor data obtained by measuring the space via the communication interface, wherein the second spatial map includes a second map feature; identifying a third map feature for map alignment from the first map feature and the second map feature; aligning the first spatial map and the second spatial map based on the identified third map feature; identifying a local area in the space where there is at least one difference between the first spatial map and the second spatial map; and generating a personalized spatial map by identifying multiple pieces of spatial information about the local area based on the second map feature.

[0017] The graph features including the first graph feature and the second graph feature may include at least one of spatial layout information, information related to an object in a space, or spatial three-dimensional (3D) information.

[0018] At least one processor may also be configured to execute one or more instructions to perform the following operations: based on the fact that the spatial dimension of the first spatial map is different from the spatial dimension of the second spatial map, project the higher-dimensional spatial map in the first spatial map and the second spatial map to correspond to the spatial dimension of the lower-dimensional spatial map.

[0019] At least one processor may also be configured to run one or more instructions to perform the following operations: mark the first image feature and the second image feature with multiple feature attribute information, and align the first spatial image and the second spatial image based on the marked first image feature and the marked second image feature so that the first spatial image and the second spatial image overlap.

[0020] At least one processor may also be configured to run one or more instructions to perform the following operations: identifying a confidence score representing the similarity between each first image feature in the first spatial map and each second image feature corresponding to each first image feature in the second spatial map, and removing fourth image features having a confidence score less than a predetermined threshold from the first image features and the second image features.

[0021] At least one processor may also be configured to execute one or more instructions to perform the following operations: compare the first spatial map with the second spatial map, classify local areas that have changed in the second spatial map relative to the first spatial map, and generate a personalized spatial map by identifying multiple pieces of spatial information about the local areas based on second map features and the classified local areas.

[0022] The changed local region may be classified as at least one of a local region where the object is changed or a local region containing measurement noise.

[0023] The at least one processor may be further configured to execute one or more instructions to perform the following operations: identifying the local region by using a spatial change detection model, wherein the spatial change detection model takes the first spatial map and the second spatial map as input and outputs information about the local region.

[0024] At least one processor may also be configured to execute one or more instructions to perform the following operations: obtain a 3D spatial map of at least one local area among multiple local areas in the space, and update spatial information about at least one local area within the personalized spatial map based on map features in the 3D spatial map.

[0025] The at least one processor may be further configured to execute one or more instructions to perform the following operations: generating a personalized virtual space corresponding to the personalized space map based on the personalized space map.

[0026] According to an aspect of the present disclosure, there may be provided a computer-readable recording medium having recorded thereon a program for executing any one of the methods of providing a personalized space map performed by a server as described above and below.

[0027] According to one aspect of the present disclosure, an electronic device for providing a personalized spatial map includes: at least one memory configured to store one or more instructions; and at least one processor configured to execute one or more instructions stored in the at least one memory to perform the following operations: obtain a first spatial map including a first map feature representing spatial information about a space; obtain a second spatial map based on sensor data obtained by measuring the space, wherein the second spatial map includes a second map feature; identify a third map feature for map alignment from the first map feature and the second map feature; align the first spatial map and the second spatial map based on the identified third map feature; identify a local area in the space where there is at least one difference between the first spatial map and the second spatial map; and generate a personalized spatial map by identifying multiple pieces of spatial information about the local area based on the second map feature.

[0028] The graph features including the first graph feature and the second graph feature may include at least one of spatial layout information, information related to an object in a space, or spatial three-dimensional (3D) information.

[0029] At least one processor may also be configured to execute one or more instructions to perform the following operations: based on the fact that the spatial dimension of the first spatial map is different from the spatial dimension of the second spatial map, project the higher-dimensional spatial map in the first spatial map and the second spatial map to correspond to the spatial dimension of the lower-dimensional spatial map.

[0030] At least one processor may also be configured to run one or more instructions to perform the following operations: mark the first image feature and the second image feature with multiple feature attribute information, and align the first spatial image and the second spatial image based on the marked first image feature and the marked second image feature so that the first spatial image and the second spatial image overlap. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 An example in which a server provides a personalized space map according to an embodiment of the present disclosure is schematically shown.

[0032] Figure 2a is a diagram for describing a first space map used by a server according to an embodiment of the present disclosure.

[0033] Figure 2b is a diagram for describing a second space map used by a server according to an embodiment of the present disclosure.

[0034] Figure 2c is a diagram for describing a personalized space map generated by a server according to an embodiment of the present disclosure.

[0035] Figure 3 is a flowchart illustrating an operation of a server generating a personalized space map according to an embodiment of the present disclosure.

[0036] Figure 4 It is a diagram for describing graph features in a spatial graph according to an embodiment of the present disclosure.

[0037] Figure 5a is a flow chart illustrating the operation of a server projecting a spatial graph according to an embodiment of the present disclosure.

[0038] Figure 5b is a diagram for describing an operation of a server projecting a spatial graph according to an embodiment of the present disclosure.

[0039] Figure 6 is a diagram for describing an operation of a server performing graph alignment according to an embodiment of the present disclosure.

[0040] Figure 7a 2 is a diagram for describing an operation of a server evaluating confidence for aligned graphs according to an embodiment of the present disclosure.

[0041] Figure 7b is a flowchart illustrating the operation of a server evaluating confidences between graph features for aligned graphs according to an embodiment of the present disclosure.

[0042] Figure 7c is a diagram for describing an example in which a server determines confidence for aligned graphs according to an embodiment of the present disclosure.

[0043] Figure 8a is a flowchart illustrating an operation of a server identifying a local area in a space where a difference exists between a first spatial map and a second spatial map according to an embodiment of the present disclosure.

[0044] Figure 8b is a diagram for describing an artificial intelligence (AI) model used by a server according to an embodiment of the present disclosure.

[0045] Figure 8c 2 is a diagram for describing an operation of a server generating training data for an AI model according to an embodiment of the present disclosure.

[0046] Figure 9a is a diagram for describing an example in which a server generates a personalized space graph according to an embodiment of the present disclosure.

[0047] Figure 9b is a diagram for describing an example in which a server generates a personalized space graph according to an embodiment of the present disclosure.

[0048] Fig.9c is a diagram for describing an example in which a server generates a personalized space graph according to an embodiment of the present disclosure.

[0049] Fig.10is a flowchart for describing an operation of a server updating a local area in a personalized space map according to an embodiment of the present disclosure.

[0050] Fig.11a is a diagram for describing an operation of a server providing a virtual space based on a personalized space graph according to an embodiment of the present disclosure.

[0051] Fig.11b is a diagram for describing an operation of a server providing a virtual space based on a personalized space graph according to an embodiment of the present disclosure.

[0052] Fig.12 is a block diagram of a configuration of a server according to an embodiment of the present disclosure.

[0053] Fig.13 is a block diagram of a configuration of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0054] Throughout the disclosure, the expression "at least one of a, b, or c" refers to only a, only b, only c, a and b, a and c, b and c, and all of a, b, and c or variations thereof.

[0055] The terms used in the present disclosure may be general terms currently widely used in the art based on the functions described in the present disclosure, but may be changed according to the intention of technicians engaged in the art, precedents, the emergence of new technologies, etc. In addition, some specific terms may be arbitrarily selected by the applicant, and in this case, the meaning of the selected terms will be described in detail in the detailed description of the present disclosure. Therefore, the terms used herein should not be defined by their simple titles, but should be defined based on the meaning of the terms and the overall description of the present disclosure.

[0056] Unless the context clearly indicates otherwise, the singular expressions used herein are also intended to include plural expressions. All terms used herein (including technical terms or scientific terms) may have the same meaning as those generally understood by those of ordinary skill in the art. In addition, although terms including ordinal numbers such as "first", "second" may be used herein to describe various elements or components, these elements or components should not be limited by these terms. These terms are only used to distinguish an element or component from another element or component.

[0057] Throughout the specification, when a component "includes" or "contains" an element, unless there is a specific description to the contrary, it should be understood that the component may also include another element, and other elements are not excluded. In addition, terms such as "part", "module", etc. described in the specification refer to a unit for processing at least one function or operation, and can be implemented as hardware or software, or a combination of hardware and software.

[0058] Embodiments of the present disclosure will be described more fully below with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement them. However, the present disclosure can be implemented in different forms and should not be construed as being limited to the embodiments of the present disclosure set forth herein. In addition, parts irrelevant to the description of the present disclosure are omitted to clearly explain the present disclosure in the accompanying drawings, and similar reference numerals always represent similar elements. In addition, the reference numerals used in the various drawings are only used to describe the various drawings, and the different reference numerals used in the different drawings are not intended to represent different elements. Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

[0059] Figure 1 An example in which a server provides a personalized space map according to an embodiment of the present disclosure is schematically shown.

[0060] In the present disclosure, “user” refers to a user who receives and uses the personalized space map 100 generated by the server 2000 .

[0061] In the present disclosure, "spatial map" refers to map data about an interior space, and may provide the location, direction, browsing, search, etc. of an indoor space. A spatial map may be composed of a hierarchical structure of multiple pieces of data for representing indoor space information, and various known indoor map data formats may be used. Data representing the attributes of spatial information included in a spatial map may be referred to as map features. Map feature definitions may represent various attributes of indoor space information, such as spatial layout information, information related to objects in the space, spatial three-dimensional (3D) information, and other spatial map related information (e.g., building address and floor). The map features in a spatial map may be input by a map maker, or may be input or updated based on sensor data obtained by measuring the indoor space. A spatial map may be provided to a user in various forms. For example, a spatial map may be provided to a user in the form of a plan view, and an indoor space may be rendered and provided to a user in the form of a 360-degree panoramic view based on the spatial map.

[0062] In an embodiment of the present disclosure, the server 2000 may generate and provide a personalized space map 100 to the user. The personalized space map 100 may be generated based on personalized space information such as layout information and object information in the user space. In other words, the personalized space map 100 may be a representation of the user's real-world space as graph data.

[0063] In an embodiment of the present disclosure, a personalized spatial map 100 may be provided to a user via the user's electronic device 3000. The user's electronic device 3000 may be a device including a display. For example, the user's electronic device 3000 may include, but is not limited to, a smart phone, a head-mounted display (HMD) (e.g., an augmented reality (AR) / virtual reality (VR) device), a smart TV, a tablet PC, and a laptop computer, etc. The user's electronic device 3000 may be implemented as a device (such as a desktop computer) that is separated from the display and can be connected to a display device (such as a monitor). The server 2000 may send the personalized spatial map 100 to the user's electronic device 3000.

[0064] In an embodiment of the present disclosure, a virtual space generated based on a personalized space map 100 may be provided to a user via the user's electronic device 3000. Since the virtual space generated based on the personalized space map 100 implements the user's real world space as a virtual environment, objects (e.g., air purifier 102, TV 104, air conditioner 106, etc.) present in the user's space may be included in the virtual space. The user may perform various virtual space experience activities, such as changing the layout of objects in the virtual space and appreciating the virtual space. The server 2000 may send virtual space data to the user's electronic device 3000. Optionally, the user's electronic device 3000 may generate virtual space data based on the personalized space map 100.

[0065] In an embodiment of the present disclosure, the server 2000 may obtain various types of spatial maps corresponding to the space. For example, the server 2000 may obtain a public spatial map known on the Internet, etc., obtain a spatial map captured / generated by a user, or obtain a spatial map generated by another electronic device of the user (e.g., a robot cleaner, etc.). For the above-mentioned various types of spatial maps, the accuracy of the spatial map may be reduced depending on the method of obtaining the spatial map, or the spatial map may not match the spatial information about the user's real-world space depending on the time when the spatial map is generated. The server 2000 may generate a personalized spatial map 100 by obtaining various types of different spatial maps and fusing and updating the spatial maps.

[0066] In addition, although the server 2000 is described as generating the personalized space map 100 in the present disclosure, the operation of the present disclosure may be performed by other electronic devices. For example, the operation of the present disclosure may be performed by the user's electronic device 3000 including a display and by another electronic device of the user (e.g., a robot cleaner, etc.).

[0067] A specific operation of the server 2000 to generate and provide the personalized space map 100 is described in more detail in the description with reference to the following drawings.

[0068] Figure 2ais a diagram for describing a first space map used by a server according to an embodiment of the present disclosure.

[0069] In an embodiment of the present disclosure, the first spatial graph 210 may be graph data including various pieces of information related to the space. The first spatial graph 210 may include, for example, a two-dimensional (2D) / 3D spatial graph, a 2D / 3D floor plan, and a blueprint, but is not limited thereto, and the first spatial graph 210 may be graph data in various forms including 2D / 3D spatial information. One or more first spatial graphs 210 may be generated.

[0070] The first spatial graph 210 may be a pre-generated spatial graph. The first spatial graph 210 may be graph data corresponding to the user space, but is a spatial graph generated at a past time point. Therefore, the first spatial graph 210 may be a spatial graph in which objects existing in the user's real-world space, a change history of the spatial layout, etc. are not reflected.

[0071] The first spatial map 210 may be, for example, a pre-generated spatial map produced by a third party. For example, the third party may include, but is not limited to, a map production company, a real estate transaction brokerage company, an interior design company, a furniture / home appliance sales company, a personal map producer, etc. When a third party produces and distributes a spatial map so that the spatial map is publicly available, the server 2000 may obtain a public spatial map corresponding to the user space from among the public spatial maps provided by the third party as the first spatial map 210.

[0072] The first spatial map 210 may be, for example, a spatial map pre-generated by a user. The user may create a spatial map corresponding to the user space by using his or her electronic device. In detail, the user may capture an image of the space via an electronic device including a camera (e.g., a smart phone, etc.) to make a spatial map based on the captured image of the space, or by using an electronic device (e.g., a desktop computer, a laptop computer, etc.) on which a spatial mapping tool (e.g., a program, an application, etc.) may be run to make a spatial map. The spatial map made by the user may be stored in the server 2000, and the server 2000 may obtain the spatial map made by the user as the first spatial map 210.

[0073] The first spatial map 210 may be, for example, a spatial map pre-generated by an electronic device (e.g., an indoor robot, a robot cleaner, an Internet Protocol (IP) camera, etc.) operating in the user space. In detail, a spatial map generated by scanning the space using one or more sensors while the indoor robot, the robot cleaner, etc. moves through the space may be stored in the server 2000. Alternatively, a spatial map generated based on an image of the space captured by an IP camera may be stored in the server 2000. The server 2000 may obtain the spatial map generated by the electronic device operating in the user space as the first spatial map 210.

[0074] The first spatial graph 210 may include spatial information. The spatial information may be composed of graph features (hereinafter referred to as first graph features 212), which are data representing specific attributes of the space. The first graph features 212 may include, but are not limited to, spatial layout information, information related to objects in the space, spatial 3D information, and other spatial graph related information.

[0075] The space layout information may include, for example, the positions and sizes of walls, columns, aisles, doors, windows, etc. in the space, but is not limited thereto.

[0076] The information related to the object in the space may include, for example, but not limited to, the position of the object in the space, the size of the object in the space, the type of the object, identification information about the object, the direction of the object, etc.

[0077] The spatial 3D information may include, for example, but is not limited to, information on whether the spatial map is 2D / 3D, the width, length, and height of the space, height information reflected when making the spatial map, and the like.

[0078] The information related to the space map may include, for example, identification information about the space map, and update time of the space map, but is not limited thereto.

[0079] The specific value of the first graph feature 212 , ie, the attribute information about the feature, may have been included in the first spatial graph 210 or may be marked by the server 2000 .

[0080] In other words, the first spatial map 210 may be a spatial map including basic spatial information and general spatial information about the user space but not including personalized spatial information. When the server 2000 generates a personalized spatial map representing spatial information about the user's real-world space, the first spatial map 210 may be used as a basic spatial map.

[0081] Figure 2b is a diagram for describing a second space map used by a server according to an embodiment of the present disclosure.

[0082] In an embodiment of the present disclosure, the second spatial map 220 may be another graph data including various pieces of information related to the space. The second spatial map 220 may include, for example, a 2D / 3D spatial map, an occupancy grid map, etc., but is not limited thereto. The second spatial map 220 may be generated after the time when the first spatial map 210 is generated. One or more second spatial maps 220 may be generated.

[0083] In an embodiment of the present disclosure, the second spatial map 220 may be a spatial map generated based on sensor data obtained by measuring the space. Sensor data may be obtained using one or more sensors capable of measuring the space. The one or more sensors may be, for example, but not limited to, a red, green, and blue (RGB) sensor, an RGB depth (RGB-D) sensor, a time of flight (ToF) sensor, a light detection and ranging (lidar) sensor, a radio detection and ranging (radar) sensor, and the like. In addition, one or more sensors may be installed on an indoor robot, a robot cleaner, a user's smart phone, and the like.

[0084] The second spatial map 220 may be a spatial map in which objects existing in the user's real world space, a change history of the spatial layout, and the like are reflected. However, since the second spatial map 210 is generated based on sensor data obtained by measuring the space, in some cases, the spatial information based on the sensor data may include only spatial information for a local area, rather than spatial information for the entire area (global area) of the space. The global area and the local area are relative concepts, and the local area may be a part of the global area. For example, when the sensor data is obtained by measuring all areas in the space, all or part of the spatial information in the second spatial map 210 may reflect information from the sensor data. For example, when the sensor data is obtained by measuring only some areas in the space, only the spatial information corresponding to the measured area among the multiple pieces of spatial information in the second spatial map 210 may reflect information from the sensor data. In addition, the second spatial map 210 may be a spatial map including measurement noise that occurs when the sensor data is obtained. The noise includes all noise that may occur due to various reasons such as sensor measurement errors, the presence of temporary obstacles, environmental factors in the space (illuminance, light reflection, etc.), etc.

[0085] The server 2000 may obtain the second spatial map 220 generated by the electronic device including one or more sensors. Alternatively, the server 2000 may obtain sensor data from the electronic device including one or more sensors and generate the second spatial map 220.

[0086] The second spatial map 220 may include spatial information. The spatial information may be composed of a map feature (hereinafter referred to as a second map feature), which is data representing a specific attribute of the space. The second map feature may include, but is not limited to, spatial layout information, information related to objects in the space, and spatial 3D information. The elements of the second map feature may correspond to the elements of the first map feature 212, respectively, and since the second map feature differs from the first map feature 212 only in terms of feature values ​​or the presence or absence of feature values, the description provided above is omitted for the sake of brevity. The specific value of the second map feature, i.e., the attribute information about the feature, may have been included in the second spatial map 220 or marked by the server 2000.

[0087] That is, the second spatial graph 220 may be a spatial graph including personalized spatial information about the user space (e.g., objects existing in the space, current spatial layout, etc.). When the server 2000 generates a personalized spatial graph representing spatial information about the user's real-world space, multiple pieces of spatial information in the second spatial graph 220 may be updated in the personalized spatial graph by using the first spatial graph 210 as a base graph.

[0088] In addition, in reference Figure 2a In the first spatial map 210 described above, the spatial map pre-generated by the user or the spatial map pre-generated by the electronic device operating in the user space is also a spatial map generated based on the sensor data, but is different from the reference spatial map. Figure 2b A second spatial diagram 220 is described.

[0089] For example, even if a predetermined period of time has passed since the spatial map was generated, the spatial map generated based on the sensor data may be classified as the first spatial map 210. Alternatively, when the spatial map generated based on the sensor data includes a map feature that satisfies a predetermined standard (e.g., accurate layout information about the entire space, etc.) and the map feature is sufficient to be used as a basis for a personalized spatial map, the spatial map may be classified as the first spatial map 210.

[0090] Figure 2c is a diagram for describing a personalized space map generated by a server according to an embodiment of the present disclosure.

[0091] According to an embodiment of the present disclosure, the server 2000 may generate a personalized space map 230 based on the first space map 210 and the second space map 220. The personalized space map 230 may be, for example, a 2D space map or a 3D space map, but is not limited thereto.

[0092] The personalized spatial map 230 may include information about the user space. The information about the user space may consist of graph features, which are data representing specific attributes of the user space. The graph features in the personalized spatial map 230 may include, for example, but not limited to, spatial layout information, information related to objects in the space, and spatial 3D information.

[0093] In detail, the space layout information in the personalized space map 230 may indicate an updated space layout of the user space. Updating the space layout may include, but is not limited to, removing a wall, adding a wall, and the like.

[0094] The information related to the objects in the space included in the personalized space map 230 may indicate the objects existing in the user space. For example, the information related to the objects existing in the user space (such as the bed 232, the dining table 234, and the sofa 236) may include, but is not limited to, the position of each object in the space, the size of each object in the space, the type of each object, identification information about each object, the direction of each object, etc.

[0095] The spatial 3D information in the personalized spatial map 230 may include, but is not limited to, information on whether the personalized spatial map 230 is 2D / 3D, the width, length, height of the user space, height information reflected when making the personalized spatial map 230, and the like.

[0096] In other words, the personalized spatial map 230 may be a spatial map including general spatial information and personalized spatial information, wherein the general spatial information includes information related to the spatial layout of the user space, etc., and the personalized spatial information includes objects existing in the user space, changed layout, etc.

[0097] Specific operations of the server 2000 applying various graph update algorithms using the first spatial graph 210 and the second spatial graph 220 and generating and providing the personalized spatial graph 230 are described in more detail in the description with reference to the following drawings.

[0098] Figure 3 is a flowchart illustrating an operation of a server generating a personalized space map according to an embodiment of the present disclosure.

[0099] In operation S310, the server 2000 obtains a pre-generated first spatial map including a first map feature representing spatial information about the space. The server 2000 may obtain the pre-generated first spatial map from a map database stored on the server 2000, or obtain the pre-generated first spatial map from another electronic device outside the server 2000. The first spatial map may be a spatial map including basic spatial information and general spatial information about the user space but not including personalized spatial information. Figure 2aThe first spatial diagram is described, and thus the description already provided above is omitted for the sake of brevity.

[0100] In operation S320, the server 2000 obtains a second spatial map including second map features and generated based on sensor data obtained by measuring the space. The server 2000 may obtain the second spatial map from a map database stored on the server 2000, or obtain the second spatial map from another electronic device outside the server 2000. The second spatial map is generated by measuring the user space with one or more sensors, and may be a spatial map including personalized spatial information about the user space (e.g., objects present in the space, current spatial layout, etc.). As has been referred to above, Figure 2b The second spatial diagram is described, and thus the description already provided above is omitted for the sake of brevity.

[0101] In operation S330 , the server 2000 selects a graph feature for graph alignment from the first graph feature and the second graph feature.

[0102] The first spatial graph may include a first graph feature, and the second spatial graph may include a second graph feature. The server 2000 may select a common feature element from the first graph feature and the second graph feature.

[0103] The server 2000 may mark multiple pieces of feature attribute information onto features (e.g., spatial layout, objects, etc.) that may be used for graph alignment. For example, the server 2000 may mark the attribute information about the object onto the feature corresponding to the object in the first graph feature, and mark the attribute information about the object onto the feature corresponding to the object in the second graph feature. In detail, the server 2000 may mark the position of the object, the size of the object in space, the type of the object, identification information about the object, the direction of the object, etc. onto the feature corresponding to the object, but is not limited thereto. In the same manner, the server 2000 may mark multiple pieces of attribute information indicating spatial layout information, information related to objects in space, spatial 3D information, and other spatial graph-related information onto the first graph feature and the second graph feature.

[0104] Since the second spatial graph is a spatial graph generated based on sensor data, the second graph feature may include acquisition characteristics of the sensor data (e.g., acquisition time, acquisition height, etc.). Therefore, when the server 2000 selects the second graph feature, the acquisition characteristics of the sensor data in the second graph feature may be reflected. Figure 4 The operation of server 2000 selecting graph features for graph alignment and examples of graph features in a spatial graph are described in more detail in the description of FIG.

[0105] In operation S340 , the server 2000 aligns the first space graph and the second space graph based on the selected graph feature.

[0106] The server 2000 may align the first spatial graph with the second spatial graph based on the plurality of attribute information tagged to the graph features. For example, the server 2000 may align the position of object a in the first spatial graph with the position of object a in the second spatial graph, but is not limited thereto.

[0107] In addition, various known graph alignment algorithms may be used for graph alignment. For example, the server 2000 may detect a minimum bounding box including feature lines (e.g., walls) and feature points (e.g., corner points) in the spatial graph. The server 2000 may align the first spatial graph and the second spatial graph by aligning the feature lines and feature points in the first spatial graph and the second spatial graph.

[0108] In operation S350 , the server 2000 identifies a local area in the space where there is a difference between the first space map and the second space map.

[0109] The server 2000 may identify the changed local area in the second spatial map generated based on the sensor data based on the pre-generated first spatial map, and classify the type of change in the local area. For example, the server 2000 may classify the changed local area into a local area where the object has changed, a local area with a changed spatial layout, a local area containing measurement noise from the sensor, etc., but the classification is not limited thereto.

[0110] For example, when the entire area (global area) of the space is a "house", the local area may be a "room", a "bedroom", a "living room", a "kitchen", etc. The server 2000 may identify whether a "room", a "bedroom", a "living room", a "kitchen", etc. is an area where spatial information is changed (e.g., an area where a layout change, an object change, noise, etc. occurs).

[0111] In operation S360 , the server 2000 generates a personalized spatial map by determining a plurality of pieces of spatial information about the local area based on the second map feature.

[0112] When a local area having a difference between the first spatial map and the second spatial map is identified, the server 2000 may determine the spatial information based on the second map feature in the second spatial map. For example, when it is identified that there is a change in the object in the space, the server 2000 may include the object from the second spatial map in the personalized spatial map 230. For example, when it is identified that there is a change in the spatial layout within the space, the server 2000 may include the spatial layout for the second spatial map in the personalized spatial map 230. For example, when it is identified that the second spatial map includes noise, the server 2000 may remove the noise so that the noise in the second spatial map is not included in the personalized spatial map 230.

[0113] According to an embodiment of the present disclosure, the server 2000 may generate a personalized spatial map 230 including personalized spatial information by updating a plurality of pieces of spatial information about a local area based on the second map feature.

[0114] Figure 4 It is a diagram for describing the graph features in the spatial graph of the present disclosure.

[0115] In an embodiment of the present disclosure, the first spatial graph 410 may include a first graph feature 412, and the second spatial graph 420 may include a second graph feature 422. The first graph feature 412 and the second graph feature 422 may respectively include spatial layout information, information related to objects in the space, spatial 3D information, and other spatial graph related information.

[0116] The first spatial graph 410 is a pre-generated spatial graph. For example, the first spatial graph 410 may be a pre-generated 3D plan view. The specific value of the first graph feature 412 (i.e., feature attribute information) may be a value representing a spatial feature corresponding to the time when the first spatial graph 410 was pre-generated.

[0117] In an embodiment of the present disclosure, the first spatial graph 410 may include a first graph feature 412. For example, when the first spatial graph 410 is generated, information related to a layout, an object, a 3D feature, etc. in the space may be generated to be included in the first spatial graph 410. When the first spatial graph 410 includes the first graph feature 412, the server 2000 may use the first graph feature 412.

[0118] In an embodiment of the present disclosure, the server 2000 may extract and process information included in the first spatial graph 410 to mark a plurality of pieces of attribute information on the first graph feature 412 corresponding to the first spatial graph 410. For example, the server 2000 may mark information related to the layout (such as walls, columns, etc.) to the layout information included in the first graph feature 412 based on the layout of the first spatial graph 410. For example, the server 2000 may mark information such as whether the first spatial graph 410 is 3D, height information, etc. to the 3D information included in the first graph feature 412 based on the fact that the first spatial graph 410 is a spatial graph including information about the space from the floor to the ceiling (for example, 3D coordinates, etc.). Whether the first spatial graph 410 is 3D may indicate that the first spatial graph 410 is a 3D spatial graph, and the height information may indicate that the height within the space included in the first spatial graph 410 is a certain height, but is not limited thereto. The marked attribute information may be used for graph alignment.

[0119] The second spatial graph 420 is a spatial graph generated based on the sensor data. For example, the second spatial graph 420 may be a 3D plan view generated based on an image captured by a camera of a smartphone. The specific value of the second graph feature 422 (i.e., feature attribute information) may be a value representing a spatial feature corresponding to the time when the sensor data used to generate the second spatial graph 420 was obtained.

[0120] In an embodiment of the present disclosure, the server 2000 may mark a plurality of pieces of attribute information in the second graph feature 422 corresponding to the second spatial graph 420 .

[0121] In an embodiment of the present disclosure, the server 2000 may mark a plurality of pieces of attribute information in the second graph feature 422 based on metadata included in the sensor data.

[0122] For example, when the second spatial map 420 is a spatial map generated using a camera (or a depth camera), the image file captured by the camera may include camera metadata. The camera metadata may include, for example, but not limited to, date and time information, camera setting information (e.g., focal length, flash, rotation direction, shooting height, etc.), depth information, etc. Based on the camera metadata, the server 2000 may tag information related to the layout (such as walls and pillars) to the layout information included in the second map feature 422. For example, based on the camera metadata, the server 2000 may tag information such as whether the second spatial map 420 is 3D, height information, etc. to the 3D information included in the second map feature 422. Whether the second spatial map 420 is 3D may indicate that the second spatial map 420 is a 3D spatial map, and the height information may indicate that the information related to the height within the space included in the second spatial map 420 is information corresponding to a height of 50 cm or more (i.e., because the camera captures an image of the area at a height of 50 cm or more from the floor of the space, the height reflected in the map production is 50 cm or more), but is not limited thereto.

[0123] In an embodiment of the present disclosure, the server 2000 may mark multiple pieces of attribute information in the second graph feature 422 based on the analysis of the sensor data.

[0124] For example, when the second spatial graph 420 is a spatial graph generated using a camera (or a depth camera), the server 2000 may perform object recognition by using an image captured by the camera. Object recognition may be performed using an AI model capable of detecting / recognizing an object. Based on the object recognition result and / or the camera metadata, the server 2000 may tag information such as the position of the object, the height of the object, the type of the object, etc. to the object information included in the second graph feature 422. The tagged attribute information may be used for graph alignment.

[0125] In addition, despite Figure 4An example is shown in which server 2000 uses information obtainable from a camera (or depth camera) to mark attribute information on a second graph feature 422, but the present disclosure is not limited to this, and any sensor data used in the present disclosure may be used to mark attribute information on a graph feature in the same / similar manner.

[0126] Referring to the above description of the first graph feature 412 and the second graph feature 422, even if the first spatial graph 410 and the second spatial graph 420 are spatial graphs for the same space, the plurality of specific attribute information about the corresponding graph features may be different according to the method of generating the first spatial graph and the second spatial graph, the time point of generating the first spatial graph and the second spatial graph, etc. The graph features may be used by the server 2000 to align the first spatial graph 410 and the second spatial graph 420, determine the confidence between the first spatial graph 410 and the second spatial graph 420, and generate a personalized spatial graph by fusing and / or updating the first spatial graph 410 and the second spatial graph 420. A detailed description of how to use the graph features is provided in the description with reference to the following figures.

[0127] Figure 5a is a flow chart illustrating the operation of a server projecting a spatial graph according to an embodiment of the present disclosure.

[0128] Available in Figure 3 After operation S310 and operation S320 are performed Figure 5a Operation S510.

[0129] In operation S510 , based on the difference in spatial dimensions between the first spatial map and the second spatial map, the server 2000 projects a higher-dimensional spatial map of the first spatial map and the second spatial map to correspond to the spatial dimension of the lower-dimensional spatial map.

[0130] In an embodiment of the present disclosure, the server 2000 may identify the spatial dimension of the spatial graph. For example, the server 2000 may identify whether the spatial graph is 2D or 3D based on graph features included in the spatial graph. For example, the server 2000 may identify whether the spatial graph is 2D or 3D based on information (e.g., spatial coordinate information) included in the spatial graph.

[0131] In an embodiment of the present disclosure, when both the first spatial map and the second spatial map are 2D spatial maps or when both the first spatial map and the second spatial map are 3D spatial maps, the server 2000 may align the first spatial map and the second spatial map without separate dimension projection. However, when the first spatial map and the second spatial map have different spatial dimensions, the server 2000 may project a higher-dimensional spatial map to a lower dimension for comparison and updating between the first spatial map and the second spatial map.

[0132] For example, the first spatial map may be a 3D spatial map, and the second spatial map may be a 2D spatial map. The server 2000 may project the first spatial map to two dimensions to correspond to the spatial dimension (two dimensions) of the second spatial map.

[0133] For example, the first spatial map may be a 2D spatial map, and the second spatial map may be a 3D spatial map. The server 2000 may project the second spatial map to two dimensions to correspond to the spatial dimension (two dimensions) of the first spatial map.

[0134] In an embodiment of the present disclosure, the server 2000 may mark attribute information on the graph features after performing the projection for matching the dimensions of the spatial graph, such as referring to Figure 5b Further described.

[0135] Figure 5b is a diagram for describing an operation of a server projecting a spatial graph according to an embodiment of the present disclosure.

[0136] In the description Figure 5b , it is assumed that the first spatial map 510 is a 3D plan view and the second spatial map 520 is a 2D occupancy grid map. However, this is merely an example, and even when the first spatial map 510 is a 2D spatial map and the second spatial map 520 is a 3D spatial map, the same may be applied. Figure 5b Description.

[0137] In an embodiment of the present disclosure, the server 2000 may identify the dimension of the obtained spatial map. Figure 5b , since the first spatial graph 510 and the second spatial graph 520 have different dimensions, and the dimension of the first spatial graph 510 is higher than the dimension of the second spatial graph 520 , the server 2000 may project the first spatial graph 510 to a lower dimension.

[0138] In an embodiment of the present disclosure, the server 2000 may use the height information from the lower-dimensional spatial graph when projecting the higher-dimensional spatial graph to the lower dimension. For example, the server 2000 may use the height information from the second spatial graph 520 when projecting the first spatial graph 510 to two dimensions.

[0139] The second spatial map 520 may be a 2D occupancy grid map generated by a robot cleaner. The robot cleaner moves in the space and generates a 2D second spatial map 520 based on one or more sensor data. According to the field of view (FOV) of the sensor of the robot cleaner, only the area at a height of 30 cm or less from the floor of the space may be sensed. Therefore, referring to the second graph feature 524 corresponding to the second spatial map 520, the following graph feature may be included in the second graph feature 524: the graph feature indicates that the information related to the height within the space included in the second spatial map 520 is information corresponding to a height of 30 cm or less. In order to match the information level of the first spatial map 510 with the information level of the second spatial map 520, the server 2000 may only project information corresponding to a height of 30 cm or less in the space when projecting the first spatial map 510 into two dimensions. This is further described below with a specific example.

[0140] For example, when projecting the 3D object "L-shaped dining table 512" existing in the first spatial map 510 to two dimensions, the server 2000 may project only the portion of the L-shaped dining table 512 having a height of 30 cm or less, that is, information about the legs of the dining table. Therefore, the L-shaped dining table 512 in the first spatial map 510 may be projected to 2D data in the second spatial map 520, and from the perspective of the robot cleaner, the L-shaped dining table 512 has the same shape as the dining table 522 in the 2D map.

[0141] For example, when projecting the 3D object “sofa” existing in the first spatial map 510 to two dimensions, the server 2000 may project only a portion of the sofa having a height of 30 cm or less, that is, information on the legs of the sofa.

[0142] For example, the server 2000 may not project an object (eg, a ceiling-mounted air conditioner, etc.) present in the first space map 510 and located in an area having a height greater than 30 cm.

[0143] The server 2000 may mark attribute information on the graph features that can be used to align the spatial graph. For example, the server 2000 may mark the attribute information of the first graph feature 514 while projecting the 3D first spatial graph 510 into two dimensions. In the above example, when the "sofa" is projected into two dimensions, the attribute information may be marked on the area with the sofa legs, and when the "L-shaped dining table 512" is projected into two dimensions, the attribute information may be marked on the area with the legs of the dining table. In addition, since objects in areas located at a height of 30 cm or more are excluded during 2D projection, the attribute information corresponding to the excluded objects is not marked in the projected spatial graph.

[0144] In an embodiment of the present disclosure, the server 2000 may align spatial graphs with matching dimensions. For example, when the first spatial graph 510 and the second spatial graph 520 have the same dimensions, the server 2000 may directly perform graph alignment between the first spatial graph 510 and the second spatial graph 520. When the first spatial graph 510 and the second spatial graph 520 have different dimensions, the server 2000 may perform graph alignment between 2D spatial graphs with dimensions matched by dimensional projection. Figure 6 This is further described.

[0145] Figure 6 is a diagram for describing an operation of a server performing graph alignment according to an embodiment of the present disclosure.

[0146] In an embodiment of the present disclosure, the first spatial graph 610 may be aligned with the second spatial graph 620. The server 2000 may select a graph feature for graph alignment from the first graph feature of the first spatial graph 610 and the second graph feature of the second spatial graph 620. The selected graph feature may be a graph feature that exists in both the first spatial graph 610 and the second spatial graph 620.

[0147] For example, the server 2000 may select a graph feature indicating spatial layout information from among the first graph features in the first spatial graph 610 and a graph feature representing spatial layout information from among the second graph features in the second spatial graph 620. For example, the server 2000 may select a graph feature representing object related information from among the first graph features in the first spatial graph 610 and a graph feature representing object related information from among the second graph features in the second spatial graph 620.

[0148] The server 2000 may align the first spatial graph 610 and the second spatial graph 620 based on the selected graph feature.

[0149] In addition, for graph alignment, various graph alignment algorithms may be used in a complementary manner. For example, the server 2000 may detect feature lines (eg, walls) and feature points (eg, corner points) in the spatial graph, and align the spatial graph based on the detected feature lines and feature points.

[0150] The server 2000 may detect a minimum bounding box including a feature line and a feature point. The server 2000 may align the first spatial graph 610 and the second spatial graph 620 by aligning at least one of a feature line or a feature point between the first spatial graph 610 and the second spatial graph 620. In addition, the server 2000 may align the first spatial graph 610 and the second spatial graph 620 by aligning a size and a direction of a minimum bounding box between the first spatial graph 610 and the second spatial graph 620.

[0151] Figure 7ais a diagram for describing an operation of a server evaluating confidence between aligned graphs according to an embodiment of the present disclosure.

[0152] In an embodiment of the present disclosure, one or more first spatial graphs 710 may be provided. In addition, one or more second spatial graphs 720 may be provided. After aligning the first spatial graph 710 and the second spatial graph 720, the server 2000 may calculate the confidence of the first spatial graph 710 and the second spatial graph 720.

[0153] The server 2000 may calculate an error between a first graph feature in the first spatial graph 710 and a second graph feature in the second spatial graph. For example, the server 2000 may calculate an error between spatial layouts and an error between objects.

[0154] The server 2000 may calculate an error E between the graph structure in the first spatial graph 710 and the graph structure in the second spatial graph 720. m The error E between graph structures m It can indicate whether the spatial layouts do not match each other. The error E between the graph structures m It may be defined as a root mean square error (RMSE) for one or more first spatial maps 710 and one or more second spatial maps 720. In an embodiment of the present disclosure, the server 2000 may apply a higher weight to a recently generated spatial map.

[0155] The server 2000 may calculate an error E between an object in the first spatial map 710 and an object in the second spatial map 720. o The error between objects E o It may indicate whether objects in the space do not match each other. For example, the server 2000 may calculate the degree of overlap (IoU) of objects that overlap each other in the first spatial graph 710 and the second spatial graph 720. When the IoU is greater than or equal to a preset value (e.g., 0.5), the server 2000 may compare multiple pieces of feature attribute information marked on the graph feature. For example, the server 2000 may compare the types (categories) of overlapping objects in the first spatial graph 710 and the second spatial graph 720, but is not limited thereto. For example, when there is no category information about the object marked on the graph feature, the server 2000 may only compare the degree of match between the positions of the objects.

[0156] Server 2000 can use the error E between the graph structures m The error E between the o To calculate the confidence C m (C m =1 / (E m +λE o ), where λ is a parameter set according to the confidence of object detection).

[0157] The server 2000 calculates the confidence C for one or more first spatial graphs 710 and one or more second spatial graphs 720 m As a result, the spatial graph determined to have low confidence may be removed. That is, at least some of the one or more first spatial graphs 710 and the one or more second spatial graphs 720 may be removed and not used to generate the personalized spatial graph.

[0158] Figure 7b is a flowchart illustrating the operation of a server evaluating confidences between graph features in aligned graphs according to an embodiment of the present disclosure.

[0159] In an embodiment of the present disclosure, in addition to calculating the confidence between spatial graphs, the server 2000 may also calculate the confidence between graph features in the spatial graph. Figure 7b In addition, Figure 7b The operation can be replaced Figure 3 Operation S340.

[0160] In operation S710, the server 2000 calculates a confidence score representing the similarity between each first graph feature in the first spatial graph 710 and each second graph feature corresponding to each first graph feature in the second spatial graph 720. For example, the graph feature may include information related to the object. When the first spatial graph 710 and the second spatial graph 720 are aligned to overlap, the server 2000 may compare multiple pieces of information related to objects present at the same location. In detail, the server 2000 may compare the size, type, identification information, direction, etc. of the object.

[0161] In operation S720 , the server 2000 removes graph features having a confidence score less than a threshold from the first graph features and the second graph features.

[0162] A graph feature having a confidence score less than a threshold means that the data included in the corresponding graph is inaccurate. For example, fixed elements in a space (e.g., walls, built-in cabinets, etc.) are unlikely to change, while movable elements in the space (e.g., movable objects such as chairs, etc.) are likely to change. The pre-generated first spatial graph 710 is a spatial graph generated at an earlier time point than the second spatial graph 720, and some elements present in the first spatial graph 710 may not exist in the second spatial graph 720 generated at a later time point. Therefore, in order to exclude inaccurate data elements from each spatial graph, the server 2000 may use a confidence score. As a result of the server 2000 refining the first spatial graph 710 and the second spatial graph 720 based on the confidence score, various graph features may be removed. For example, graph features corresponding to movable elements may be removed from the first spatial graph 710 generated at the initial time point of creating the space.

[0163] In addition, one or more first spatial graphs 710 may be provided. In addition, one or more second spatial graphs 720 may be provided. When calculating the confidence score of the graph feature, when there are multiple spatial graphs, the server 2000 may apply weights based on the time point at which the spatial graph is updated. For example, the server 2000 may apply a higher weight to the most recently updated spatial graph so that the most recently generated graph feature can be trusted. Since the graph feature in the spatial graph generated at an earlier time point has a lower confidence, the graph feature is likely to be removed.

[0164] After executing operation S720, you can execute Figure 3 Operation S350 of a.

[0165] Figure 7c is a diagram for describing an example in which a server determines confidence for aligned graphs according to an embodiment of the present disclosure.

[0166] As described above, the server 2000 may tag a plurality of pieces of attribute information related to features to the first graph feature in the first spatial graph 710 and the second graph feature in the second spatial graph 720. For example, the server 2000 may tag information related to the update time of the spatial graph.

[0167] The pre-generated first spatial graph 710 is a spatial graph generated at an earlier time point than the second spatial graph 720. Therefore, when the update time of the first spatial graph 710 is unknown, the update time may not be marked on the first graph feature in the first spatial graph 710, or even when the update time is marked on the first graph feature in the first spatial graph 710, the update time may be earlier than the update time marked on the second graph feature in the second spatial graph 720. The server 2000 may set the confidence score based on the graph feature with the most recent update time.

[0168] In a specific example, the first graph feature in the first spatial graph 710 may include a built-in cabinet 730 as information related to the object. In addition, the second graph feature in the second spatial graph 720 may include a built-in cabinet 730 and a built-in refrigerator 740 as information related to the object. In addition, the second graph feature in the second spatial graph 720 may include information about an update time.

[0169] The server 2000 may assign a high confidence score to the built-in cabinet 730 and the built-in refrigerator 740 included in the second graph feature in the most recently updated second spatial graph 720 based on the update time. That is, the server 2000 may prevent the graph features corresponding to the built-in cabinet 730 and the built-in refrigerator 740 from being removed by assigning a higher weight to the graph features in the most recently updated spatial graph.

[0170] Figure 8ais a flowchart illustrating an operation of a server identifying a local area in a space where a difference exists between a first spatial map and a second spatial map according to an embodiment of the present disclosure.

[0171] Available in Figure 3 After performing the graph alignment in operation S340, Figure 8a In addition, Figure 8a The operation can be replaced Figure 3 Operation S350.

[0172] In operation S810 , the server 2000 compares the first space map with the second space map.

[0173] After performing the graph alignment between the first spatial graph and the second spatial graph, the server 2000 may identify a local area in the space where there is a difference between the first spatial graph and the second spatial graph. For example, the first spatial graph and the second spatial graph are spatial graphs for the entire area (global area) of the space, respectively. The entire area of ​​the space may be classified into a plurality of local areas. For example, when the entire area of ​​the space is a "house", the local area may be a "room", a "bedroom", a "living room", a "kitchen", etc., but is not limited thereto.

[0174] In operation S820, the server 2000 classifies the local areas changed in the second spatial map with respect to the first spatial map.

[0175] The local area identified as having a difference between the first spatial map and the second spatial map indicates that the spatial information has been changed at a time point after the pre-generated first spatial map is obtained. The server 2000 may classify the changed local area according to the type of change. For example, the server 2000 may classify the changed local area into at least one of a local area where an object is changed, a local area where a spatial layout is changed, or a local area containing measurement noise, but the classification is not limited thereto.

[0176] After executing operation S820, you can execute Figure 3 Operation S360 of a.

[0177] In an embodiment of the present disclosure, the server 2000 may perform operations S810 and S820 by using an AI model. Figure 8b The operation of server 2000 using the AI ​​model is further described in the description.

[0178] Figure 8b is a diagram for describing an AI model used by a server according to an embodiment of the present disclosure.

[0179] In an embodiment of the present disclosure, the server 2000 may use an AI model (hereinafter, a spatial change detection model) 800 to identify a local area in the space where there is a difference between the first spatial map and the second spatial map. The spatial change detection model 800 may be a deep neural network model that takes the first spatial map and the second spatial map as input and outputs information about the local area where there is a difference between the first spatial map and the second spatial map. The spatial change detection model 800 may be implemented using various known deep neural network architectures and algorithms, or may be implemented by variants of various known deep neural network architectures and algorithms. Figure 8b Training a spatial change detection model 800 is shown.

[0180] In an embodiment of the present disclosure, the server 2000 may train the spatial change detection model 800 based on a training data set including various spatial graphs. The training data set may include various 2D or 3D spatial graphs. In addition, spatial graphs representing the same space may be stored in pairs. For example, a plan view 810 or a synthetic plan view 812 (which is a spatial graph representing the same space) and a lidar map 814 may be stored as a pair in the training data set.

[0181] The plan view 810 corresponds to the first spatial view, and the lidar map 814 corresponds to the second spatial view. In addition, the synthetic plan view 812 is generated by applying a data augmentation algorithm to the plan view 810 to train the spatial change detection model 800, and corresponds to the first spatial view. However, Figure 8b Only an example is shown, and a data enhancement algorithm may be applied to the lidar map 814 to train the spatial change detection model 800. The map data obtained by enhancing the lidar map 814 corresponds to the second spatial map. Figure 8c Describe various data augmentation algorithms.

[0182] In an embodiment of the present disclosure, the server 2000 may train the spatial change detection model 800 by using training data including paired floor plan 810 and lidar map 814. When there is no difference between the floor plan 810 and the lidar map 814 because the spatial information about the space has not changed, the spatial change detection model 800 is trained to classify it as "no change".

[0183] In an embodiment of the present disclosure, the server 2000 may train the spatial change detection model 800 by using training data including a paired floor plan 810 and a lidar map 814. The lidar map 814 is based on sensor data, and due to glass, etc., present in the space, noise 804 may be included in the data from the lidar sensor. When noise is included in the lidar map 814, the spatial change detection model 800 is trained to classify the noise 804 as “noise in the second spatial map”.

[0184] In an embodiment of the present disclosure, the server 2000 can train the spatial change detection model 800 by using training data including paired synthetic plan views 812 and laser radar views 814. When the first spatial view (synthetic plan view 812) and the second spatial view (laser radar view 814) are generated at different time points, the spatial information about the space may have changed. For example, the object 802 may exist in the synthetic plan view 812, but not in the laser radar view 814. The spatial change detection model 800 is trained to classify the object 802 as a "removed object".

[0185] According to an embodiment of the present disclosure, by using a pair of the first spatial map and the second spatial map, the server 2000 can train the spatial change detection model 800 to classify the change of the local area in the space. The spatial change classification category 815 from the spatial change detection model 800 can be obtained by, for example, classifying the change in the space into "added object", "removed object", "added layout", "removed layout", "noise", etc., but the spatial change classification category 815 is not limited thereto.

[0186] According to an embodiment of the present disclosure, the server 2000 may identify a local area where there is a difference between the first spatial map and the second spatial map by using the trained spatial change detection model 800. The server 2000 may input the first spatial map and the second spatial map into the spatial change detection model 800. The spatial change detection model 800 classifies whether the change in the local area where there is a difference between the first spatial map and the second spatial map is a change in an object and / or a layout, a difference due to noise, etc. The server 2000 may obtain information about the local area output from the spatial change detection model 800.

[0187] Figure 8c 2 is a diagram for describing an operation of a server generating training data for an AI model according to an embodiment of the present disclosure.

[0188] Reference Figure 8c , the server 2000 trains the spatial change detection model 800 by using the training dataset 820. The training dataset 820 may include a plurality of spatial graphs, wherein a first spatial graph and a second spatial graph are stored in pairs.

[0189] In an embodiment of the present disclosure, the server 2000 may enhance the training data included in the training data set 820 for the spatial change detection model 800. The server 2000 may select at least one of the paired first spatial map and second spatial map, and enhance the training data by adding or removing objects / layouts / noise, etc.

[0190] For example, the floor plan 810 may correspond to a first spatial map in the training data set 820, and the lidar map 814 may correspond to a second spatial map in the training data set 820. The server 2000 may generate a synthetic floor plan 812 by adding an object to the floor plan 810. In addition, the server 2000 may generate a synthetic lidar map 816 by adding noise to the lidar map 814.

[0191] In an embodiment of the present disclosure, when training data is enhanced by adding objects / layouts / noise, etc. to the training data, the server 2000 can enhance the training data by extracting information related to the objects / layouts / noise, etc. from another pair of matched images.

[0192] In an embodiment of the present disclosure, when data is enhanced by adding objects / layouts / noise, etc. to training data, the server 2000 may use a generative adversarial network (GAN). GAN may include two modules, namely a generator and a discriminator. The generator may learn real data (such as objects / layouts / noise, etc.). The generator may be trained to generate false data (such as objects / layouts / noise, etc.) to approach real data (such as objects / layouts / noise, etc.), and the discriminator may be trained to discriminate whether the data output by the generator is false data or real data. GAN may be a known architecture and algorithm, or various variants for optimizing known architectures and algorithms. Server 2000 may obtain training data including objects / layouts / noise, etc., which are difficult to collect real data, by using GAN.

[0193] In addition, the enhancement of the training data by the server 2000 is not limited to "adding" objects / layouts / noises, etc. The server 2000 can enhance the training data in various ways, including removing, copying / pasting, rotating and flipping objects / layouts / noises, etc.

[0194] In an embodiment of the present disclosure, the training data obtained by the server 2000 through data enhancement may be included in the training data set 820. When the enhanced training data is stored in the training data set 820 as a pair of the first spatial map and the second spatial map, the server 2000 may use various combinations. For example, the server 2000 may store the plan view 810 and the lidar map 814 as a pair. Alternatively, the server 2000 may store the plan view 810 and the synthetic lidar map 816 as a pair. Alternatively, the server 2000 may store the synthetic plan view 812 and the lidar map 814 as a pair. Alternatively, the server 2000 may store the synthetic plan view 812 and the synthetic lidar map 815 as a pair.

[0195] Figure 9a is a diagram for describing an example in which a server generates a personalized space graph according to an embodiment of the present disclosure.

[0196] In an embodiment of the present disclosure, when the changed local area 902 is identified, the server 2000 may generate a personalized spatial map 910 by determining spatial information about the changed local area 902 based on the second map feature in the second spatial map. The base map for the personalized spatial map 910 may be the first spatial map 900, but is not limited thereto, and the second spatial map may be used as the base map.

[0197] As a result of the server 2000 classifying the changed local area 902 in the second spatial map based on the first spatial map 900, the local area 902 may be a local area where the object has changed. As a specific example, the change may be the addition of an object. This means that there is no object in the area corresponding to the local area 902 in the pre-generated first spatial map 900, but there is an object in the area corresponding to the local area 902 in the second spatial map generated later. The server 2000 may determine spatial information about the local area 902 based on the second map feature in the second spatial map. For example, the second map feature in the second spatial map may include information related to the object present in the local area 902. The server 2000 may add a virtual object to the area corresponding to the local area 902 in the personalized spatial map 910. Optionally, the server 2000 may remove or change the object in the space in the same / similar manner as described above.

[0198] In addition, the server 2000 may update the personalized spatial map 910 based on data received from the user (e.g., the user's electronic device) (e.g., a third spatial map obtained by capturing an image of the local area 902). For example, the server 2000 may receive an image 912 of the local area 902 captured by the user, and update the personalized spatial map 910 based on the received image 912. Therefore, the information about the objects located in the local area 902 in the personalized spatial map 910 may be updated. For example, the identification information (model name, serial number, etc.) about the object may be updated, but the present disclosure is not limited thereto. Fig.10 The operation of the server 2000 to additionally update the personalized space map 910 is further described in the description of .

[0199] Figure 9b is a diagram for describing an example in which a server generates a personalized space graph according to an embodiment of the present disclosure.

[0200] In an embodiment of the present disclosure, when the changed local area 922 is identified, the server 2000 may generate a personalized spatial map 930 by determining spatial information about the changed local area 922 based on the second map feature in the second spatial map. The map 920 as a base map for the personalized spatial map 930 may be a first spatial map, but is not limited thereto, and a second spatial map may be used as a base map.

[0201] As a result of the server 2000 classifying the changed local area 922 in the second spatial map based on the first spatial map, the local area 902 may be a local area with a changed spatial layout. In a specific example, the change may be the removal of a wall. This means that some of the walls present in the pre-generated first spatial map do not exist in the second spatial map due to changes in interior design (e.g., balcony expansion, etc.). The server 2000 may determine spatial information about the changed local area 922 based on the second map feature in the second spatial map. For example, the second map feature in the second spatial map may include information related to the spatial layout, and specifically, may include information related to the spatial layout extended by removing the inner wall of the local area 922. The server 2000 may remove the inner wall of the local area 922 in the personalized spatial map 930 to represent the expanded space. Optionally, the server 2000 may add virtual walls in the space in the same / similar manner as described above, and may add / remove / change other spatial layout elements.

[0202] In addition, although reference has been made to Figure 9a Describes the changes to the object and has referenced Figure 9b describes changes in the spatial layout, but they are not independent of each other and can be combined. That is, changes in the spatial layout can be reflected in Figure 9aThe personalized spatial map 910 of the object can also be reflected in the Figure 9b In the personalized space diagram 930.

[0203] Fig.9c is a diagram for describing an example in which a server generates a personalized space graph according to an embodiment of the present disclosure.

[0204] In an embodiment of the present disclosure, the server 2000 may compare the first spatial map (not shown) with the second spatial map 940. The server 2000 may identify a changed local area where there is a difference between the first spatial map and the second spatial map, and classify the changed local area.

[0205] In addition, the second spatial map 940 is based on sensor data and may include various measurement noises according to the type of sensor and the environment in the space. The server 2000 may classify the changed local area as noise. For example, the server 2000 may compare the first spatial map with the second spatial map 940, and apply an algorithm for noise detection based on the comparison result to identify the area where noise exists in the second spatial map 940. Optionally, the server 2000 may apply the algorithm for noise detection to the second spatial map 940 to identify the area where noise exists in the second spatial map 940. As an algorithm for noise detection, various known algorithms may be used. Optionally, the server 2000 may identify the area where noise exists in the second spatial map 940 by using a spatial change detection model.

[0206] When it is determined that measurement noise from the sensor data exists in the second spatial map 940 , the server 2000 may remove the noise from the second spatial map 940 to obtain a noise-free second spatial map 950 with improved map quality.

[0207] The server 2000 may identify a changed local region in the space by using the noise-free second spatial map 950. This has been described above, and therefore, will not be repeated for the sake of brevity.

[0208] Fig.10 is a flowchart for describing an operation of a server updating a local area in a personalized space map according to an embodiment of the present disclosure.

[0209] exist Figure 3 After the personalized space map is generated in operation S360, the following steps may be performed: Fig.10 operation.

[0210] In operation S1010 , the server 2000 obtains a third space map generated based on sensor data obtained by measuring one of the local areas in the space.

[0211] Each of the first spatial map and the second spatial map is a spatial map including spatial information about the entire area (global area) of the space. The entire area of ​​the space may be divided into a plurality of local areas. The server 2000 may obtain a third spatial map including spatial information about any one of the plurality of local areas. For a more specific example showing the relationship between the personalized spatial map and the third spatial map, when the personalized spatial map is a spatial map for the entire area "house" of the space, the third spatial map may be a spatial map for the local area "bedroom" which is part of the entire area.

[0212] The third spatial map may be a spatial map generated based on sensor data. The third spatial map may be, for example, a spatial map generated based on an image of a space captured by a user via an electronic device including a camera (e.g., a smart phone, etc.), or a spatial map generated by an electronic device operating in the user space (e.g., an indoor robot, a robot cleaner, an IP camera, etc.). The third spatial map may include features representing spatial information about a local area. For example, the third spatial map may include layout information about the local area, information related to an object in the local area, etc., but is not limited thereto.

[0213] In operation S1020, the server 2000 identifies a local area corresponding to the local area in the third spatial map in the personalized spatial map. The server 2000 may identify the local area corresponding to the third spatial map from the multiple local areas included in the personalized spatial map. For example, the server 2000 may identify that the third spatial map is a spatial map for the "bedroom" among the multiple local areas "room", "bedroom", "living room", and "kitchen" included in the personalized spatial map.

[0214] In an embodiment of the present disclosure, the server 2000 may identify a local area corresponding to the third spatial map in the personalized spatial map based on the user input. The server 2000 may receive a user input for selecting a specific local area in the personalized spatial map from the user's electronic device, and identify the area corresponding to the user input as the local area corresponding to the local area in the third spatial map.

[0215] In an embodiment of the present disclosure, the server 2000 may identify a local area corresponding to the third spatial map in the personalized spatial map based on the graph features in the third spatial map and the graph features in the personalized spatial map. The server 2000 may compare the graph features in the third spatial map with the graph features in the personalized spatial map, and calculate the similarity between the common graph features. For example, the server 2000 may calculate the confidence scores between the graph features, but is not limited thereto. Based on the comparison results of the graph features, the server 2000 may identify an area in the personalized spatial map that includes graph features that are identical / similar to the graph features in the third spatial map as a local area corresponding to the local area in the third spatial map.

[0216] In an embodiment of the present disclosure, the server 2000 may identify a local area corresponding to the third spatial graph in the personalized spatial graph by using a graph alignment algorithm. The server 2000 may identify an area having a graph structure (e.g., spatial layout, etc.) identical to / similar to that of the third spatial graph in the personalized spatial graph. The server 2000 may identify a local area corresponding to a local area in the third spatial graph in the personalized spatial graph based on the graph alignment result.

[0217] The above-described methods of identifying the local area corresponding to the third spatial map in the personalized spatial map are not independent of each other, and the server 2000 may use a combination of two or more of the methods.

[0218] In addition, the personalized spatial map and the third spatial map may have different spatial dimensions. When the personalized spatial map and the third spatial map have different spatial dimensions, the server 2000 may project the spatial map of the higher dimension to the lower dimension. For example, the personalized spatial map may be a 2D spatial map, and the third spatial map may be a 3D spatial map. The server 2000 may project the third spatial map to two dimensions.

[0219] In operation S1030, the server 2000 matches the objects and layout in the local area identified in the personalized spatial map with the objects and layout in the third spatial map. In detail, as in the above example, when the third spatial map is for a bedroom, the server 2000 may match the objects / layout in the local area corresponding to the bedroom in the personalized spatial map with the objects / layout in the third spatial map.

[0220] In an embodiment of the present disclosure, as a result of the server 2000 classifying whether a local area in the user space has changed based on the first spatial map and the second spatial map according to the above-mentioned embodiment of the present disclosure, there may be a local area classified as having an added object. In this case, the server 2000 adds the object (e.g., a virtual object) to the personalized spatial map. This means that "object A" has been newly placed in the user space, and "object A" is added to the personalized spatial map accordingly. In addition, "object A" may be included in a third spatial map obtained by recaptured images of the local area. In this case, the server 2000 may match "object A" in the personalized spatial map with "object A" in the third spatial map.

[0221] In an embodiment of the present disclosure, since the third spatial map is a spatial map generated by scanning only a local area, the spatial information in the third spatial map may include more accurate and detailed information than the first spatial map and / or the second spatial map for only the local area. For example, the third spatial map may include detailed information such as a model name, identification number, and color of an object, but is not limited thereto. The server 2000 may determine and update the spatial information about the local area identified in operation S1020 based on the map features in the third spatial map. In detail, the server 2000 may update detailed information about the matched "object A", such as the model name, identification number, and color of object A.

[0222] In operation S1040 , the server 2000 performs texture mapping on the personalized space map based on the matching result.

[0223] The server 2000 may map the texture included in the third spatial map to the objects and / or layout in the personalized spatial map by using a texture mapping algorithm. The texture may include surface texture and color, but is not limited thereto. For example, when the third spatial map is a spatial map for a bedroom, the server 2000 may perform texture mapping on a local area corresponding to the bedroom in the personalized spatial map. The map features in the personalized spatial map may be updated. As a result of the texture mapping performed by the server 2000, the bedroom in the personalized spatial map may have the same spatial information as the user's real-world bedroom. That is, the layout (walls, windows, etc.) and textures of objects (TV, bed, bookshelves, etc.) of the user's real-world bedroom may be reflected in the personalized spatial map.

[0224] Fig.11a is a diagram for describing an operation of a server providing a virtual space based on a personalized space graph according to an embodiment of the present disclosure.

[0225] In an embodiment of the present disclosure, the server 2000 may generate a personalized virtual space 1100 (hereinafter referred to as a virtual space) corresponding to the user's real world space based on a personalized space map. The virtual space 1100 may include a spatial layout and objects from the user's real world space. Since the virtual space 1100 generated based on the personalized space map implements the user's real world space as a virtual environment, the virtual space 1100 may include objects existing in the user's space. The user may perform various virtual space experience activities, such as changing the position of objects in the virtual space 1100 and appreciating the virtual space 1100.

[0226] In an embodiment of the present disclosure, when a user selects an object 1110 existing in the virtual space 1100, detailed information related to the object 1110 may be displayed. The detailed information related to the object 1110 may include a model name, an identification number, a color, a user manual, etc. of the object 1110. In addition, a recommendation of an object similar to the selected object 1110 (e.g., a recommendation for purchase or replacement) may be provided, but the present disclosure is not limited thereto.

[0227] In an embodiment of the present disclosure, the user may change the interior design by using the virtual space 1100. For example, after selecting an object 1110 existing in the virtual space 1100, the user may remove or move the selected object 1110 from the virtual space 1100. For example, the user may select an empty area in the virtual space 1100 and place a new object in the selected area.

[0228] In an embodiment of the present disclosure, the user may browse within the virtual space 1100. For example, when the user selects a first viewpoint 1120 present in the virtual space 1100, a 360-degree view of the virtual space 1100 rendered based on the first viewpoint 1120 may be provided. In addition, when the user selects a second viewpoint 1122 present in the virtual space 1100, a 360-degree view of the virtual space 1110 rendered based on the second viewpoint 1122 may be provided.

[0229] In an embodiment of the present disclosure, an example of a user using a virtual space 1100 may be provided via the user's electronic device 3000. That is, the server 2000 may transmit virtual space data to the user's electronic device 3000, and the virtual space 1100 may be displayed on the user's electronic device 3000 including a display. For example, the user's electronic device 3000 may include, but is not limited to, a smart phone, an HMD (e.g., an AR / VR device), a smart TV, a tablet PC, and a laptop computer, etc. The user's electronic device 3000 may be implemented as a device (e.g., a desktop computer) that is separated from a display and can be connected to a display device (e.g., a monitor).

[0230] Fig.11b is a diagram for describing an operation of a server providing a virtual space based on a personalized space graph according to an embodiment of the present disclosure.

[0231] In an embodiment of the present disclosure, the server 2000 may provide both the virtual space 1100 and the personalized space map 1102. When browsing the virtual space 1100 by using the user's electronic device 3000, the user may check where the current position is within the virtual space 1100 through the personalized space map 1102 displayed together. The personalized space map 1102 may be displayed in a corner area of ​​a screen displaying the virtual space 1100, but is not limited thereto.

[0232] exist Fig.11b In the illustration of personalized space map 1102, users are shown as being located in different virtual spaces.

[0233] For example, when the user is located in the living room 1130 in the virtual space 1100, the personalized space map 1102 may indicate that the user's current location is the living room 1130. In addition, the personalized space map 1102 may indicate in which direction the user's FOV is oriented. In addition, when the user is located in the kitchen 1140 or the bedroom 1150, the user's current location may be indicated on the personalized space map 1102. At the user's current location, a 360-degree view of the virtual space 1100 may be provided.

[0234] Fig.12 is a block diagram of a configuration of a server according to an embodiment of the present disclosure.

[0235] According to an embodiment of the present disclosure, the server 2000 may include a communication interface 2100 , a memory 2200 , and a processor 2300 .

[0236] The communication interface 2100 may include a communication circuit. The communication interface 2100 may include a communication circuit capable of performing data communication between the server 2000 and other devices by using at least one of a data communication method including, for example, a wired local area network (LAN), a wireless LAN, Wi-Fi, Bluetooth, ZigBee, Wi-Fi Direct (WFD), an infrared data association (IrDA), Bluetooth low energy (BLE), a near field communication (NFC), wireless broadband Internet (WiBro), world interoperability for microwave access (WiMAX), a shared wireless access protocol (SWAP), a wireless gigabit alliance (WiGig), and a radio frequency (RF) communication.

[0237] The communication interface 2100 may send data required to perform the operation of the server 2000 to an external electronic device and receive data required to perform the operation of the server 2000 from an external electronic device. For example, the server 2000 may receive a spatial map from an electronic device external to the server 2000 (e.g., a user's smart phone, a robot cleaner, etc.) via the communication interface 2100. The spatial map may include a first spatial map and / or a second spatial map. In addition, the server 2000 may send and receive a personalized spatial map and / or various pieces of data for displaying a virtual space generated based on the personalized spatial map to the user's electronic device 3000 via the communication interface 2100.

[0238] The memory 2200 may store instructions, data structures, and program codes readable by the processor 2300. The memory 2200 may be configured as one or more memories. In an embodiment of the present disclosure, the operations performed by the processor 2300 may be implemented by running instructions or codes of a program stored in the memory 2200.

[0239] Memory 2200 may include nonvolatile memory (such as read-only memory (ROM), for example, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM)), flash memory (for example, memory cards and solid-state drives (SSD)), and analog recording-type memory (for example, hard disk drives (HDD), magnetic tapes, and optical disks)), and volatile memory (such as random access memory (RAM), for example, dynamic RAM (DRAM) and static RAM (SRAM)).

[0240] The memory 2200 may store one or more instructions and programs that cause the server 2000 to operate to generate a personalized spatial map. For example, the spatial map processing module 2210, the AI ​​module 2220, and the personalized spatial map generation module 2230 may be stored in the memory 2200.

[0241] The processor 2300 may control all operations of the server 2000. For example, the processor 2300 may execute one or more instructions of a program stored in the memory 2200 to control all operations performed by the server 2000 to generate a personalized space map. The processor 2300 may be configured as one or more processors.

[0242] The one or more processors 2300 may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), an integrated many-core (MIC), a digital signal processor (DSP), or a neural processing unit (NPU). The one or more processors 2300 may be implemented in the form of an integrated system on chip (SoC) including one or more electronic components. The one or more processors 2300 may be implemented as separate hardware (H / W), respectively.

[0243] The processor 2300 may perform overall management and analysis tasks on the spatial map by using the spatial map processing module 2210. The processor 2300 may obtain a first spatial map and a second spatial map, and classify and store the spatial map obtained for each space as a first spatial map or a second spatial map. The processor 2300 may mark multiple feature attribute information representing spatial information onto the first spatial map and / or the second spatial map. The processor 2300 may align the first spatial map with the second spatial map. When the first spatial map and the second spatial map have different spatial dimensions, the processor 2300 may project the higher-dimensional spatial map to a lower dimension. A description related to the operation of the spatial map processing module 2210 has been provided with reference to the aforementioned figures, and therefore will not be repeated for the sake of brevity.

[0244] Processor 2300 may use AI module 2220 to perform all tasks in which AI is used. AI module 2220 may include one or more AI models implemented as data structures that simulate artificial neural networks. For example, AI module 2220 may include, but is not limited to, object recognition models, object detection models, spatial change detection models, and GANs for generating training data.

[0245] The processor 2300 may use the AI ​​module 2220 to run an AI model for identifying changes in space and process data obtained from the AI ​​model. The processor 2300 may run, for example, a spatial change detection model. When the spatial change detection model is run, the aligned first spatial map and the second spatial map may be input to the spatial change detection model, and information about a local area where there is a difference between the first spatial map and the second spatial map may be output.

[0246] The processor 2300 may train the AI ​​model by using the AI ​​module 2220. The processor 2300 may train the spatial change detection model by using, for example, training data from the first spatial map and the second spatial map included in the training data set.

[0247] A description related to the operation of the AI ​​module 2220 has been provided with reference to the aforementioned figures and thus will not be repeated for the sake of brevity.

[0248] The processor 2300 may generate and manage a personalized spatial map by using the personalized spatial map generation module 2230. The processor 2300 may generate a personalized spatial map including personalized spatial information in several ways. The processor 2300 may use, for example, a first spatial map as a base map. The processor 2300 may generate a personalized spatial map by updating the map features in the first spatial map according to the second spatial map. Alternatively, the processor 2300 may use the second spatial map as a base map. The processor 2300 may use the second spatial map as a base map, but update the map features in the second spatial map based on the comparison result between the first spatial map and the second spatial map.

[0249] Descriptions related to the operations of the personalized spatial map generation module 2230 have been provided with reference to the aforementioned figures, and thus will not be repeated for the sake of brevity.

[0250] In addition, the modules stored in the memory 2200 and executed by the processor 2300 are for convenience of description and are not necessarily limited thereto. Other modules may be added to implement the above-mentioned embodiments of the present disclosure, and the modules may be subdivided into a plurality of modules distinguished according to their detailed functions, and some of the above-mentioned modules may be combined to form a single module.

[0251] When the method according to an embodiment of the present disclosure includes multiple operations, the multiple operations may be performed by one processor or multiple processors. For example, when the method according to an embodiment of the present disclosure performs a first operation, a second operation, and a third operation, the first operation, the second operation, and the third operation may all be performed by a first processor, or the first operation and the second operation may be performed by a first processor (e.g., a general-purpose processor), and the third operation may be performed by a second processor (e.g., a dedicated AI processor). A dedicated AI processor as an example of a second processor may perform calculations for training / inference of an AI model. However, embodiments of the present disclosure are not limited thereto.

[0252] One or more processors 2300 according to the present disclosure may be implemented as a single-core processor or a multi-core processor.

[0253] When the method according to an embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one core or a plurality of cores included in the one or more processors 2300 .

[0254] Fig.13 is a block diagram of a configuration of an electronic device according to an embodiment of the present disclosure.

[0255] In an embodiment of the present disclosure, the above operations of the server 2000 may also be performed by the electronic device 3000 .

[0256] According to an embodiment of the present disclosure, the electronic device 3000 may include a communication interface 3100, a display 3200, a memory 3300, and a processor 3400. Since the communication interface 3100, the memory 3300, and the processor 3400 of the electronic device 3000 correspond to Fig.12 The communication interface 2100, the memory 2200 and the processor 2300 of the server 2000 are described, and thus the description of the components provided above is omitted.

[0257] The display 3200 may output information processed by the electronic device 3000. In addition, when the display 3200 and the touch panel form a layer structure to construct a touch screen, the display 3200 may be used as an input device as well as an output device. The display 3200 may include at least one of a liquid crystal display (LCD), a thin film transistor LCD (TFT LCD), an organic light emitting diode (OLED) display, a flexible display, a 3D display, a micro display, or an HMD. The electronic device 3000 may display a personalized space map and / or a virtual space corresponding to the personalized space map on the display 3200.

[0258] Although not in Fig.13 Although not shown in the figure, the electronic device 3000 may also include one or more cameras and / or one or more sensors. The electronic device 3000 may perform 3D scanning of the space by using one or more cameras and / or one or more sensors. For example, the one or more cameras and / or one or more sensors may include, but are not limited to, RGB-D sensors, ToF sensors, lidar sensors, radar sensors, etc.

[0259] In addition, the electronic device 3000 may further include an input / output (I / O) interface. The I / O interface may include an input interface for receiving a user's input and an output interface for outputting a signal other than an image / video signal output from the display 3200.

[0260] The input interface is used to receive input from the user. The input interface may include but is not limited to at least one of a keyboard, a dome switch, a touch pad (capacitive overlay type, resistive overlay type, infrared beam type, surface acoustic wave type, integrated strain gauge type, piezoelectric type, etc.), a scroll wheel or a scroll wheel switch.

[0261] The output interface may include a speaker. The speaker may output an audio signal received from the communication interface 3100 or an audio signal stored in the memory 3300.

[0262] The present disclosure presents a method for generating a personalized spatial map, wherein the method identifies changes in the space and updates the spatial map based on classification of the changes to generate a personalized spatial map including spatial information about a user's real-world space by using a pre-generated first spatial map and a second spatial map generated based on sensor data obtained by measuring the space.

[0263] The technical solutions to be implemented in the present disclosure are not limited to those described above, and other technical solutions not described will be clearly understood by those of ordinary skill in the art from the following description.

[0264] According to an aspect of the present disclosure, a method for providing a personalized space map performed by a server may be provided.

[0265] The method may include obtaining a pre-generated first spatial map, wherein the first spatial map includes first map features representing spatial information about the space.

[0266] The method may include obtaining a second spatial map generated based on sensor data obtained by measuring the space. The second spatial map may include a second map feature.

[0267] The method may include selecting a graph feature from the first graph feature and the second graph feature for graph alignment.

[0268] The method may include aligning the first spatial graph and the second spatial graph based on the selected graph feature.

[0269] The method may include identifying local areas in the space where differences exist between the first spatial map and the second spatial map.

[0270] The method may include generating a personalized spatial map by determining a plurality of pieces of spatial information about the local area based on the second map feature.

[0271] The graph features including the first graph feature and the second graph feature may include at least one of spatial layout information, information related to an object in a space, or spatial 3D information.

[0272] The method may include projecting a higher-dimensional spatial map of the first spatial map and the second spatial map to correspond to a spatial dimension of a lower-dimensional spatial map based on the spatial dimension of the first spatial map being different from the spatial dimension of the second spatial map.

[0273] Selecting graph features for graph alignment may include labeling the first graph feature and the second graph feature with a plurality of pieces of feature attribute information.

[0274] Aligning the first spatial graph and the second spatial graph may include aligning the first spatial graph and the second spatial graph based on the marked first graph feature and the marked second graph feature such that the first spatial graph and the second spatial graph overlap.

[0275] Aligning the first spatial map and the second spatial map may include calculating a confidence score representing a similarity between each first map feature in the first spatial map and each second map feature in the second spatial map that corresponds to each first map feature.

[0276] Aligning the first spatial graph and the second spatial graph may include removing graph features having a confidence score less than a threshold from the first graph features and the second graph features.

[0277] Identifying a local area in the space where a difference exists between the first spatial map and the second spatial map may include comparing the first spatial map and the second spatial map.

[0278] Identifying local regions in the space where there are differences between the first spatial view and the second spatial view may include classifying local regions that have changed in the second spatial view relative to the first spatial view.

[0279] Generating the personalized spatial map may include generating the personalized spatial map by determining a plurality of pieces of spatial information about the local area based on the second map feature and the classification of the local area.

[0280] The changed local region may be classified as at least one of a local region where the object is changed or a local region containing measurement noise.

[0281] Identifying local regions in the space where there are differences between the first spatial view and the second spatial view may include identifying the local regions by using a spatial change detection model.

[0282] The spatial change detection model may take the first spatial map and the second spatial map as input and output information about the local area.

[0283] The method may include obtaining a 3D spatial map of at least some of a plurality of local regions in the space.

[0284] The method may include updating spatial information about at least some local areas within the personalized spatial map based on map features in the 3D spatial map.

[0285] The method may include generating a personalized virtual space corresponding to the personalized space map based on the personalized space map.

[0286] According to one aspect of the present disclosure, a server for providing a personalized spatial map may be provided. The server may include a communication interface, at least one memory storing one or more instructions, and at least one processor configured to execute the one or more instructions stored in the memory.

[0287] At least one processor may execute one or more instructions to obtain a pre-generated first spatial map via a communication interface, wherein the first spatial map includes a first map feature representing spatial information about the space.

[0288] The at least one processor may execute one or more instructions to obtain, via the communication interface, a second spatial map generated based on sensor data obtained by measuring the space. The second spatial map may include a second map feature.

[0289] At least one processor may execute one or more instructions to select graph features for graph alignment from the first graph features and the second graph features.

[0290] At least one processor may execute one or more instructions to align the first spatial graph and the second spatial graph based on the selected graph feature.

[0291] At least one processor may execute one or more instructions to identify a local area in the space where a difference exists between the first spatial map and the second spatial map.

[0292] The at least one processor may execute one or more instructions to generate a personalized spatial map by determining a plurality of pieces of spatial information about the local area based on the second map feature.

[0293] The graph features including the first graph feature and the second graph feature may include at least one of spatial layout information, information related to an object in a space, or spatial 3D information.

[0294] At least one processor may execute one or more instructions to project a higher-dimensional spatial map of the first spatial map and the second spatial map to correspond to the spatial dimension of the lower-dimensional spatial map based on the spatial dimension of the first spatial map being different from the spatial dimension of the second spatial map.

[0295] At least one processor may execute one or more instructions to tag the first image feature and the second image feature with a plurality of pieces of feature attribute information.

[0296] At least one processor may execute one or more instructions to align the first spatial map and the second spatial map based on the marked first map feature and the marked second map feature such that the first spatial map and the second spatial map overlap.

[0297] At least one processor may execute one or more instructions to calculate a confidence score representing a similarity between each first graph feature in the first spatial graph and each second graph feature in the second spatial graph corresponding to each first graph feature.

[0298] At least one processor may execute one or more instructions to remove graph features having a confidence score less than a threshold from the first graph features and the second graph features.

[0299] At least one processor may execute one or more instructions to compare the first spatial map to the second spatial map.

[0300] At least one processor may execute one or more instructions to classify local areas that have changed in the second spatial map relative to the first spatial map.

[0301] The at least one processor may execute one or more instructions to generate a personalized spatial map by determining a plurality of pieces of spatial information about the local area based on the second map feature and the classification of the local area.

[0302] The changed local region may be classified as at least one of a local region where the object is changed or a local region containing measurement noise.

[0303] At least one processor may execute one or more instructions to identify local regions using a spatial change detection model.

[0304] The spatial change detection model may take the first spatial map and the second spatial map as input and output information about the local area.

[0305] At least one processor may execute one or more instructions to obtain a 3D spatial map of at least some of the plurality of local regions in the space.

[0306] At least one processor may execute one or more instructions to update spatial information about at least some local areas within the personalized spatial map based on map features in the 3D spatial map.

[0307] In addition, the embodiments of the present disclosure can also be implemented in the form of a recording medium including instructions that can be run by a computer (such as a program module run by a computer). A computer-readable recording medium can be any available medium that can be accessed by a computer, and includes both volatile media and non-volatile media and both removable media and non-removable media. In addition, a computer-readable recording medium may include both computer storage media and communication media. Computer storage media include both volatile media and non-volatile media and both removable media and non-removable media implemented using any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Communication media typically include computer-readable instructions, data structures, program modules or other data in a modulated data signal.

[0308] The computer-readable storage medium may be provided in the form of a non-transitory storage medium. In this regard, the term "non-transitory storage medium" merely means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), and the term does not distinguish whether data is semi-permanently stored in the storage medium or temporarily stored in the storage medium. For example, a "non-transitory storage medium" may include a buffer for temporarily storing data.

[0309] According to an embodiment of the present disclosure, when a computer program product is provided, the method according to various embodiments of the present disclosure may be included in the computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a computer-readable storage medium (e.g., a compact disc ROM (CD-ROM)), or distributed online (e.g., downloaded or uploaded) via an application store, or distributed directly between two user devices (e.g., smart phones). For online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be at least temporarily stored or temporarily generated in a computer-readable storage medium, such as a memory of a manufacturer's server, a server of an application store, or a relay server.

[0310] The above description of the present disclosure is provided for illustration, and it will be understood by those skilled in the art that the form and details may be changed at will without departing from the technical ideas or key features of the present disclosure. Therefore, the above embodiments of the present disclosure and all aspects thereof are merely illustrative and non-restrictive. For example, each component defined as an integrated component may be implemented in a distributed manner, and similarly, components defined as separate components may be implemented in an integrated form.

[0311] The scope of the disclosure is defined not by the detailed description thereof but by the following claims, and all changes or modifications within the meaning and scope of the appended claims and their equivalents are to be construed as being included in the scope of the disclosure.

Claims

1. A method for providing a personalized spatial map performed by a server (2000), the method comprising: Obtaining a first spatial graph including first graph features representing spatial information about a space (S310); Obtaining a second spatial map based on sensor data obtained by measuring the space, wherein the second spatial map includes a second map feature (S320); identifying a third graph feature for graph alignment from the first graph feature and the second graph feature (S330); aligning the first spatial graph and the second spatial graph based on the identified third graph feature (S340); identifying a local area in the space where there is at least one difference between the first spatial map and the second spatial map (S350); and The personalized spatial map is generated by identifying a plurality of pieces of spatial information about the local area based on the second map features ( S360 ).

2. The method according to claim 1, wherein: The graph features including the first graph feature and the second graph feature include at least one of spatial layout information, information related to an object in the space, or spatial three-dimensional (3D) information.

3. The method according to any one of claims 1 and 2, further comprising: Based on the fact that a spatial dimension of the first spatial map is different from a spatial dimension of the second spatial map, a spatial map with a higher dimension in the first spatial map and the second spatial map is projected to correspond to a spatial dimension of a spatial map with a lower dimension.

4. The method according to any one of claims 1 to 3, in, The third graph features identified for graph alignment include: The first image feature and the second image feature are marked with a plurality of feature attribute information, and Wherein, aligning the first spatial graph with the second spatial graph comprises: The first spatial map and the second spatial map are aligned based on the marked first map features and the marked second map features, so that the first spatial map and the second spatial map overlap.

5. The method according to any one of claims 1 to 4, in, Aligning the first spatial pattern with the second spatial pattern comprises: identifying a confidence score representing a similarity between each first graph feature in the first spatial graph and each second graph feature in the second spatial graph corresponding to the each first graph feature; and A fourth image feature having a confidence score less than a predetermined threshold is removed from the first image feature and the second image feature.

6. The method according to any one of claims 1 to 5, in, Identifying a local region of the space having at least one difference between the first spatial map and the second spatial map comprises: comparing the first spatial map to the second spatial map; and classifying local areas that have changed in the second spatial map relative to the first spatial map, and Wherein, generating the personalized space map includes: The personalized spatial map is generated by identifying a plurality of pieces of spatial information about the local area based on the second map features and the classified local area.

7. The method according to claim 6, wherein: The changed local region is classified as at least one of a local region where the object is changed or a local region containing measurement noise.

8. A server (2000) for providing a personalized spatial map, the server comprising: Communication interface (2100); at least one memory (2200) storing one or more instructions; as well as At least one processor (2300) is configured to execute the one or more instructions stored in the memory (2200) to perform the following operations: obtaining, via the communication interface (2100), a first spatial graph including first graph features representing spatial information about the space, obtaining, via the communication interface (2100), a second spatial map based on sensor data obtained by measuring the space, wherein the second spatial map comprises a second map feature, identifying a third image feature for image alignment from the first image feature and the second image feature, aligning the first spatial graph and the second spatial graph based on the identified third graph feature, identifying a local region in the space where there is at least one difference between the first spatial map and the second spatial map, and The personalized spatial map is generated by identifying a plurality of pieces of spatial information about the local area based on the second map features.

9. The server according to claim 8, wherein: The graph features including the first graph feature and the second graph feature include at least one of spatial layout information, information related to an object in the space, or spatial three-dimensional (3D) information.

10. The server according to any one of claims 8 and 9, wherein: The at least one processor is further configured to execute the one or more instructions to perform the following operations: Based on the fact that a spatial dimension of the first spatial map is different from a spatial dimension of the second spatial map, a spatial map with a higher dimension in the first spatial map and the second spatial map is projected to correspond to a spatial dimension of a spatial map with a lower dimension.

11. The server according to any one of claims 8 to 10, in, The at least one processor is further configured to execute the one or more instructions to perform the following operations: Marking the first graph feature and the second graph feature with multiple pieces of feature attribute information; The first spatial map and the second spatial map are aligned based on the marked first map features and the marked second map features, so that the first spatial map and the second spatial map overlap.

12. The server according to any one of claims 8 to 11, wherein: The at least one processor is further configured to execute the one or more instructions to perform the following operations: identifying a confidence score representing a similarity between each first graph feature in the first spatial graph and each second graph feature in the second spatial graph corresponding to the each first graph feature, and A fourth image feature having a confidence score less than a predetermined threshold is removed from the first image features and the second image features.

13. A server according to any one of claims 8 to 12, in, The at least one processor is further configured to execute the one or more instructions to perform the following operations: comparing the first spatial map with the second spatial map, classifying local areas that have changed in the second spatial map relative to the first spatial map, and The personalized spatial map is generated by identifying a plurality of pieces of spatial information about the local area based on the second map feature and the classified local area.

14. The server according to claim 13, wherein: The changed local region is classified as at least one of a local region where the object is changed or a local region containing measurement noise.

15. A computer-readable recording medium having recorded thereon a program for executing the method according to any one of claims 1 to 7 on a computer.

Images