Augmented reality apparatus for providing augmented reality service for controlling object in real space and method of operating same

Abstract

An augmented reality apparatus for providing an augmented reality service for controlling an object in a real world space, and an operation method performed by the augmented reality apparatus are provided. An augmented reality device according to an embodiment of the present disclosure may identify a wall and a floor from a space image obtained by photographing a real world space using a camera, generating a three-dimensional (3D) model of the real-world space by expanding the identified wall and floor and performing 3D repair on areas of the expanded wall and floor hidden by the object using the spatial image, performing two-dimensional (2D) segmentation to segment the object selected by the user input from the spatial image, and performing 3D segmentation to segment the object on the spatial image from the real-world space based on the 3D model or the 3D position information of the object.

Description

Technical Field

[0001] The present disclosure relates to an augmented reality device for providing an augmented reality service for controlling an object in a real world space, and an operation method performed by the augmented reality device. Specifically, the present disclosure discloses an augmented reality device for providing an augmented reality service for controlling an object in a real world space and a virtual object using image processing and augmented reality technology by using a deep neural network, and an operation method of the augmented reality device. Background Art

[0002] Augmented reality is a technology that displays a virtual image together with a real-world object by superimposing the virtual image on the real world or the physical environment space of the real-world object. Augmented reality devices (e.g., smart glasses) using augmented reality technology are effectively used in daily life, for example, for information search, direction guidance, and camera shooting. In particular, smart glasses as an example of an augmented reality device are worn as a fashion item and are mainly used for outdoor activities.

[0003] Because augmented reality technology is a technology that displays virtual objects in real-world space, it is necessary to recognize the real-world space in real time and utilize the information of the recognized real-world objects. For example, when a user wants to replace the furniture in the indoor living room, the augmented reality service in the prior art uses a method of clumsily overlapping virtual furniture (virtual objects) on real furniture (real-world objects). The augmented reality technology in the prior art uses a method of deleting the area of ​​the furniture to be replaced and arranging virtual furniture (virtual objects). However, when multiple pieces of furniture are attached together, it is difficult to selectively delete only specific pieces of furniture, and because the patching is performed using a simple interpolation method, when the background is not completely flat but curved between walls or between walls and floors, the generated virtual object will be different from the real-world space. In particular, since the augmented reality device should continuously perform computationally intensive operations such as segmentation and patching using a deep neural network while running an augmented reality application to provide an augmented reality service, the processing time may be long, and the heat generation and power consumption of the device may increase. Summary of the invention

[0004] Technical Solution

[0005] One aspect of the present disclosure provides an augmented reality device for providing an augmented reality service for controlling an object in a real world space. The augmented reality device according to an embodiment of the present disclosure may include a camera, an inertial measurement unit (IMU) sensor including an accelerometer and a gyroscope sensor, a memory storing at least one instruction, and at least one processor configured to execute at least one instruction. At least one processor may obtain a spatial image by photographing the real world space using a camera. At least one processor may identify a plane including a wall and a floor from the obtained spatial image. At least one processor may generate a three-dimensional (3D) model of the real world space by extending the wall and the floor along the identified plane and performing 3D patching on the area of ​​the extended wall and the floor hidden by the object using the spatial image. At least one processor may perform two-dimensional (2D) segmentation, which segments an object selected by a user input from the spatial image. At least one processor may perform 3D segmentation, which segments an object on the spatial image from the real world space based on a 3D model or 3D position information of the object.

[0006] Another aspect of the present disclosure provides a method for providing an augmented reality service for controlling an object in a real-world space, performed by an augmented reality device. The method may include identifying a plane including a wall and a floor from a spatial image obtained by photographing the real-world space using a camera. The method may include generating a three-dimensional (3D) model of the real-world space by extending the walls and the floor along the identified plane, and performing 3D repair on the area of ​​the extended wall and the floor that is hidden by the object by using the spatial image. The method may include performing two-dimensional (2D) segmentation, which segments an object selected by user input from the spatial image. The method may include performing 3D segmentation, which segments an object on the spatial image from the real-world space based on a 3D model or 3D position information of the object.

[0007] Another aspect of the present disclosure provides a computer program product including a computer-readable storage medium. The storage medium may include instructions executed by an augmented reality device to recognize a plane including a wall and a floor from a spatial image obtained by photographing a real-world space using a camera; generate a three-dimensional (3D) model of the real-world space by extending the wall and the floor along the recognized plane and performing 3D repair on an area of ​​the extended wall and the floor hidden by an object by using the spatial image; perform two-dimensional (2D) segmentation that segments an object selected by a user input from the spatial image; and perform 3D segmentation that segments an object on the spatial image from the real-world space based on a 3D model or 3D position information of the object. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The present disclosure can be easily understood from the following detailed description taken in conjunction with the accompanying drawings, in which the reference numerals represent structural elements.

[0009] Figure 1 is a conceptual diagram for describing an operation of an augmented reality device of the present disclosure providing an augmented reality service for deleting, moving, or adding an object in a real world space.

[0010] Figure 2 is a diagram illustrating an operation performed by an augmented reality device to control an object in a real world space according to an embodiment of the present disclosure.

[0011] Figure 3 is a block diagram illustrating elements of an augmented reality device according to an embodiment of the present disclosure.

[0012] Figure 4 is a flowchart illustrating an operation method performed by an augmented reality device according to an embodiment of the present disclosure.

[0013] Figure 5 is a flowchart illustrating a method for generating a three-dimensional (3D) model of a real world space performed by an augmented reality device according to an embodiment of the present disclosure.

[0014] Figure 6 is a diagram illustrating an operation performed by an augmented reality device to obtain a 3D model shape related to a shape of a plane in a real world space according to an embodiment of the present disclosure.

[0015] Fig. 7A is a diagram illustrating an operation performed by an augmented reality device to distinguish planes of a wall and a floor on a space image according to an embodiment of the present disclosure.

[0016] Figure 7B is a diagram illustrating an operation performed by an augmented reality device to distinguish planes of walls and windows on a space image according to an embodiment of the present disclosure.

[0017] Figure 7C is a diagram illustrating an operation performed by an augmented reality device for distinguishing planes of a wall and a floor having different patterns on a space image according to an embodiment of the present disclosure.

[0018] Figure 8 is a diagram illustrating an operation for performing 3D inpainting performed by an augmented reality device according to an embodiment of the present disclosure.

[0019] Fig. 9 is a diagram illustrating an operation performed by an augmented reality device for performing two-dimensional (2D) segmentation from a spatial image according to an embodiment of the present disclosure.

[0020] Fig.10is a flowchart illustrating a method of performing 3D segmentation based on whether a 3D model of an object is stored by an augmented reality device according to an embodiment of the present disclosure.

[0021] Fig.11 is a diagram illustrating an operation performed by an augmented reality device for performing 3D segmentation by using a pre-stored 3D model of an object according to an embodiment of the present disclosure.

[0022] Fig.12 is a diagram illustrating an operation performed by an augmented reality device for performing 3D segmentation when a 3D model of an object is not stored according to an embodiment of the present disclosure.

[0023] Fig.13 is a flowchart illustrating a method by which an augmented reality device arranges a 3D model in a real world space and performs rendering by using the 3D model according to an embodiment of the present disclosure.

[0024] Fig.14 is a flowchart illustrating a method by which an augmented reality device additionally performs 3D segmentation and updates a segmentation result according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] Considering the functions in the present disclosure, the terms used herein are those common terms currently widely used in the art, but these terms may change according to the intention of ordinary technicians in the art, precedents, or new technologies in the art. In addition, some of the terms used herein may be arbitrarily selected by the applicant, and in this case, these terms are defined in detail below. Therefore, the specific terms used herein should be defined based on their unique meanings and the entire context of the present disclosure.

[0026] Unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include the plural forms as well. Terms (including technical or scientific terms) used herein may have the same meanings as commonly understood by one of ordinary skill in the art described in the present disclosure.

[0027] When a part "includes" an element, unless otherwise described, another element may be further included, rather than excluding the presence of another element. Also, the term "... unit" or "... module" refers to a unit that performs at least one function or operation, and may be implemented as hardware or software or as a combination of hardware and software.

[0028] The expression "configured (or set) to" used in the present disclosure may be replaced with, for example, "suitable for", "having the ability to", "designed to", "adapted to", "manufactured to", or "capable of", depending on the circumstances. The term "configured (or set) to" does not always mean "specially designed to" by hardware only. Alternatively, in some cases, the expression "a system configured to..." may mean that the system "can" operate together with another device or component. For example, a "processor configured (or set) to perform A, B, and C" may be a dedicated processor (e.g., an embedded processor) for performing the corresponding operations or a general-purpose processor (e.g., a central processing unit (CPU) or an application processor) that can perform the corresponding operations by executing at least one software program stored in a memory.

[0029] Furthermore, in the present disclosure, it should be understood that, unless otherwise specified, when elements are “connected” or “coupled” to each other, the elements may be directly connected or coupled to each other, but may alternatively be connected or coupled to each other with intervening elements therebetween.

[0030] In the present disclosure, “augmented reality” means showing a virtual image in a physical environment space of the real world or showing a real world object together with a virtual image.

[0031] In the present disclosure, an “augmented reality device” may be a device capable of expressing augmented reality, and may be, for example, any one of a mobile device, a smart phone, a laptop computer, a desktop computer, a tablet PC, an e-book terminal, a digital broadcast terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation device, an MP3 player, or a camera. However, the present disclosure is not limited thereto, and in an embodiment of the present disclosure, the augmented reality device may be implemented as augmented reality glasses in the form of glasses that a user typically wears on his / her face, a head-mounted display (HMD) device mounted on the head, an augmented reality helmet, and the like.

[0032] In the present disclosure, "real world space" refers to a space in the real world that a user sees via an augmented reality device. In an embodiment of the present disclosure, the real world space may refer to an indoor space. A real world object may be located in the real world space.

[0033] In the present disclosure, a "virtual object" is an image generated via an optical engine, and may include both still images and videos. A virtual object may be shown together with a real scene, and may be a virtual image representing information about a real-world object in a real scene or information about the operation of an augmented reality device or a control menu. In an embodiment of the present disclosure, a "virtual object" may include a user interface (UI) provided via an application or program executed by an augmented reality device. For example, a virtual object may be a UI including a bounding box representing a recognized area such as a wall, floor, or window recognized from a spatial image.

[0034] A general augmented reality device includes an optical engine for generating a virtual object, the virtual object including light generated from a light source; and a waveguide formed of a transparent material to guide the virtual object generated by the optical engine to the user's eyes and also view the real world scene. As described above, because the augmented reality device must be able to observe not only the virtual object but also the real world scene, an optical element for changing the optical path with straightness is basically required so as to guide the light generated from the optical engine to the user's eyes via the waveguide. In this case, the optical path may be changed via reflection from a mirror or the like, or may be changed via diffraction of a diffraction element, such as a diffractive optical element (DOE) or a holographic optical element (HOE), but the present disclosure is not limited thereto.

[0035] In this disclosure, “inpainting” refers to an image processing technique that restores hidden, lost, or distorted portions of an image.

[0036] In the present disclosure, “segmentation” refers to an image processing technique that classifies objects in an image into classes or categories, distinguishes the objects from other objects or background images in the image based on the classification results, and segments the objects.

[0037] The embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings so that those skilled in the art can perform the embodiments of the present disclosure without difficulty. However, the present disclosure can be embodied in many different forms and is not limited to the embodiments of the present disclosure set forth herein.

[0038] Hereinafter, embodiments of the present disclosure are described in detail with reference to the accompanying drawings.

[0039] Figure 1 1 is a conceptual diagram for describing an operation of the augmented reality device 100 of the present disclosure providing an augmented reality service for deleting, moving, or adding objects (eg, 11 , 12 , and 13 ) in a real world space 10 .

[0040] The augmented reality device 100 is a device for providing an augmented reality service, and may be, for example, any one of a mobile device, a smart phone, a laptop computer, a desktop computer, a tablet PC, an e-book terminal, a digital broadcast terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation device, an MP3 player, or a camera. Figure 1 In an embodiment of the present disclosure, the augmented reality device 100 may be a smart phone. However, the present disclosure is not limited thereto, and in an embodiment of the present disclosure, the augmented reality device 100 may be implemented as augmented reality glasses in the form of glasses that a user generally wears on his / her face, a head mounted display (HMD) device mounted on the head, an augmented reality helmet, etc.

[0041] refer to Figure 1 , the augmented reality device 100 may run the augmented reality service (operation ①). In an embodiment of the present disclosure, the augmented reality device 100 may run the augmented reality application to provide the user with the augmented reality service. In response to the augmented reality application being run, the augmented reality device 100 may use the camera 110 (see Figure 3 ) captures the real world space 10 to obtain a spatial image 20. The augmented reality device 100 can obtain a spatial image 20 including a plurality of image frames by capturing the real world space 10 in real time.

[0042] The augmented reality device 100 can display the spatial image 20 on the display unit, and can receive a user input (operation ②) for selecting a specific object on the spatial image 20. The spatial image 20 may include object images (e.g., 21, 22, and 23) of objects (e.g., 11, 12, and 13) located in the real-world space 10. For example, the first object image 21 may be an image of a lighting fixture as the first object 11, the second object image 22 may be an image of a television as the second object 12, and the third object image 23 may be an image of a table as the third object 13. In an embodiment of the present disclosure, the augmented reality device 100 may include a touch screen, and may receive a user's touch input for selecting any one object image from a plurality of object images (e.g., 21, 22, and 23) included in the spatial image 20 displayed on the touch screen. Figure 1In an embodiment of the invention, the augmented reality device 100 may receive a touch input of a user selecting a second object image 22 on the spatial image 20. The augmented reality device 100 may select the second object image 22 based on the touch input of the user. However, the present disclosure is not limited thereto. For example, in a case where the augmented reality device 100 is implemented as augmented reality glasses, an object may be selected based on a gaze point where the gaze directions of the user's two eyes converge. In addition, in another example, in a case where the augmented reality device 100 is implemented as an HMD device, an object may be selected based on a user input received via an external controller.

[0043] The augmented reality device 100 may delete the object selected by the user input (operation ③). Figure 1 In the embodiment of the augmented reality device 100, the second object image 22 selected by the user input can be deleted from the space image 20. In the area where the second object image 22 is deleted, the area where the second object ( Figure 1 In an embodiment of the present disclosure, the augmented reality device 100 may identify the planes of the walls and the floor in the real world space 10 by using a plane detection technology, and may generate a 3D model of the real world space 10 by extending the planes of the identified walls and the floor and performing 3D patching on the area hidden by the multiple objects (e.g., 11, 12, and 13) in the extended plane area of ​​the walls and the floor. The augmented reality device 100 may segment the second object image 22 selected by the user input from the spatial image 20 by performing 3D segmentation. The augmented reality device 100 may display the area hidden by the second object image 22 segmented in the spatial image 20 by arranging the 3D model of the real world space 10 on the planes of the walls and the floor in the identified real world space 10 and performing rendering using the 3D model. A specific 3D model generation method and a specific 3D segmentation method will be described in detail in the following description of the accompanying drawings.

[0044] The augmented reality device 100 can move and add an object image selected by the user input on the space image 20 (operation ④). Figure 1 In the embodiment of the present disclosure, the augmented reality device 100 may move the position of the first object image 21 related to the lighting lamp from the left to the right, and may add a new object image 30 of the sofa (an object not in the real world space 10). In the embodiment of the present disclosure, the augmented reality device 100 may move the first object image 21 or add the new object image 30 by performing 3D segmentation using a pre-stored 3D model of the object or using at least one of the 3D position coordinate values ​​of the edge, feature point, or pixel of the object recognized from the space image 20.

[0045] Will refer to Figure 2A specific method in which the augmented reality device 100 controls an object, such as deleting, moving, or adding an object on the space image 20 of the real world space 10, is described in detail.

[0046] Figure 2 is a diagram illustrating an operation performed by an augmented reality device to control an object in a real world space according to an embodiment of the present disclosure.

[0047] refer to Figure 2 , augmented reality device 100 (see Figure 1 ) can recognize a space from a space image 200 of a real-world space, and can recognize the planes of walls 202, 203, and 204 and a floor 201 in the real-world space (operation ①). The augmented reality device 100 can start an augmented reality service by running an augmented reality application, and by using the camera 110 (see Figure 3 ) captures the real world space to obtain the space image 200. In an embodiment of the present disclosure, the augmented reality device 100 can identify the planes of the walls 202, 203, and 204 and the floor 201 from the space image 200 by using a plane detection technology provided by the augmented reality application.

[0048] The augmented reality device 100 can perform 3D patching on the wall and the floor (operation ②). In an embodiment of the present disclosure, the augmented reality device 100 can extract three arbitrary points from each of the identified planes of the walls 202, 203 and 204 and the floor 201, and can derive the plane equations of each of the walls 202, 203 and 204 and the floor 201. The augmented reality device 100 can obtain a 3D model shape including virtual walls 212, 213 and 214 and a virtual floor 211 by extending the planes of the walls 202, 203 and 204 and the floor 201 based on the derived plane equations. The augmented reality device 100 can obtain depth value information of each wall, floor and object in the real world space obtained from the camera 110, can obtain color information from the space image 200, and can distinguish the areas of the virtual walls 212, 213 and 214, the virtual floor 211 and the object based on the depth value information and the color information. The augmented reality device 100 can perform repair on the area hidden by the object of each of the virtual walls 212, 213 and 214 and the virtual floor 211 by using the images of the walls 202, 203 and 204 of the real world space, except for the parts of the virtual walls 212, 213 and 214 and the virtual floor 211 hidden by the object in the 3D model shape.

[0049] In the present disclosure, “painting” refers to an image processing technique that restores a hidden, lost, or distorted portion of an image. In the present disclosure, “3D inpainting” may refer to an image processing technique that restores an area of ​​a 3D image that is hidden by a 3D object. The augmented reality device 100 may generate a 3D model including virtual walls 212, 213, and 214 and a virtual floor 211 by performing inpainting. In an embodiment of the present disclosure, the augmented reality device 100 may store the generated 3D model in the memory 140 (see Figure 3 ) in a storage space (e.g., a 3D model database 148 (see Figure 3 ))middle.

[0050] The augmented reality device 100 may perform 2D segmentation on the object selected by the user input (operation ③). Figure 2 In an embodiment of the present disclosure, the augmented reality device 100 may receive a user input that selects an object to be controlled on a spatial image 200 displayed via a running augmented reality application. In an embodiment of the present disclosure, the augmented reality device 100 may display the spatial image 200 via a touch screen, and may receive a user input that selects any one object 220 from a plurality of objects included in the spatial image 200. The augmented reality device 100 may select the object 220 based on the user input, and may perform 2D segmentation on the object 220 based on 2D image information (e.g., 2D position coordinate information) of the selected object 220, and plane information of the walls 202, 203, and 204 and the floor 201, depth value information, and object distinction information obtained in operations ① and ②. In an embodiment of the present disclosure, the augmented reality device 100 may segment the object 220 selected by the user input by using a deep neural network model that is pre-trained to classify a plurality of objects into labels, classes, or categories. In the present disclosure, “segmentation” refers to an image processing technique of classifying an object in an image, distinguishing the object from other objects or background images in the image based on the classification result, and segmenting the object.

[0051] The augmented reality device 100 may perform 3D segmentation on the selected object 220 (operation ④). The augmented reality device 100 may perform 3D segmentation, which is to segment the object 220 from the spatial image 200 by using information obtained through 2D segmentation. In an embodiment of the present disclosure, the 3D segmentation may be performed differently depending on whether the 3D model of the object 220 is stored. For example, in the case where the 3D model of the object 220 is pre-stored in the memory 140 (see Figure 3), the augmented reality device may arrange the 3D model of the object 220 to overlap with the 2D segmentation contour on the spatial image by adjusting the direction of the 3D model stored in the memory 140, and may obtain the 3D position coordinate value information of the object 220 from the arranged 3D model. The augmented reality device 100 may perform 3D segmentation of segmenting the object 220 from the spatial image 200 based on the obtained 3D position coordinate value.

[0052] For example, when the 3D model of the object 220 is not stored in the memory 140 (see Figure 3 ), the augmented reality device 100 can perform 3D vertex modeling based on at least one of the edges 222, feature points 224 and 3D position coordinate values ​​of the pixels of the object 220 identified from the spatial image 200, and can perform 3D segmentation of the object 220 from the spatial image 200 via 3D vertex modeling.

[0053] The augmented reality device 100 may perform at least one of deleting, moving, and replacing the object 220 (operation ⑤). The augmented reality device 100 may render an area hidden by the object 220 by using a 3D model. Via rendering, the augmented reality device 100 may display an area segmented via 3D segmentation of the object 220. The augmented reality device 100 may perform a deletion operation of deleting the object 220 from the spatial image 200, a replacement operation of replacing the object 220 with a virtual image of another object, or a movement operation of arranging the object 220 in another area by using 2D segmentation information and 3D segmentation information of the object 220. Therefore, the augmented reality device 100 may control the object 220 in the real world space in augmented reality.

[0054] Prior art augmented reality services use virtual objects that are awkwardly overlaid on objects in real-world space (e.g. Figure 111, 12 and 13). For example, the augmented reality technology in the prior art uses a method of deleting the area of ​​the furniture to be replaced and arranging virtual furniture (virtual objects). However, when multiple pieces of furniture are attached together, it is difficult to selectively delete only specific pieces of furniture, and because the patching is performed using a simple interpolation method, when the background is not completely flat but curved between walls or between walls and floors, the generated virtual object will be different from the real world space. In particular, multiple image frames of the real world space are obtained in real time while running an augmented reality application to provide an augmented reality service. When the object is recognized and segmented by using a deep neural network model for each of the multiple image frames obtained in real time, the amount of calculation increases and the processing time is long. When the amount of calculation increases, real-time object recognition and segmentation will not be performed normally and will be delayed, thereby reducing the satisfaction and user convenience of the augmented reality service. In addition, when object recognition and segmentation are performed for each image frame, the heat generation and power consumption of the device increase. Due to the characteristics of the augmented reality device designed to have a small form factor for portability, heat generation and power consumption may greatly affect the use time of the device.

[0055] The purpose of the present disclosure is to provide an augmented reality device 100 and an operating method of the augmented reality device 100, wherein the augmented reality device is used to provide an augmented reality service that freely controls not only virtual objects in a real world space but also real world objects. According to an embodiment of the present disclosure, an augmented reality device 100 is provided, which is used to freely control not only virtual objects in a real world space but also real world objects. Figure 1 of 20 or Figure 2 200) identifies planes of walls and floors, generates a 3D model of the real world space by extending the identified planes and performing 3D patching, segments the object via 2D and 3D segmentation, and renders an area hidden by the object from among the areas of the walls and floors by using the 3D model.

[0056] When multiple image frames of the real world space are obtained in real time, because according to Figure 1 and Figure 2 The augmented reality device 100 of the embodiment of the present disclosure renders the wall and floor by using the 3D model only when there is a control input by the user (such as deleting, moving, or replacing an object), so the amount of calculation can be reduced compared to the augmented reality technology of the related art that performs separate object recognition and segmentation for each of a plurality of image frames. Therefore, the augmented reality device 100 of the present disclosure provides a technical effect of reducing the computing power and suppressing the heat generation of the device. In addition, because the augmented reality device 100 according to the embodiment of the present disclosure stores the 3D model in the memory 140 (see Figure 3 ) in a storage space (e.g., a 3D model database 148 (see Figure 3)), and rendering is performed by loading a 3D model only when there is control through user input (such as deleting, moving, or replacing an object), so the processing time can be reduced and a real-time augmented reality service can be provided.

[0057] Figure 3 is a block diagram illustrating elements of the augmented reality device 100 according to an embodiment of the present disclosure.

[0058] refer to Figure 3 , the augmented reality device 100 may include a camera 110, an inertial measurement unit (IMU) sensor 120, a processor 130, a memory 140, and a display unit 150. The camera 110, the IMU sensor 120, the processor 130, the memory 140, and the display unit 150 may be electrically and / or physically connected to each other. Figure 3 Only necessary elements for describing the operation of the augmented reality device 100 are shown in FIG. 1 , but the elements included in the augmented reality device 100 are not limited to Figure 3 In an embodiment of the present disclosure, the augmented reality device 100 may further include a communication interface for performing data communication with an external device or a server. In an embodiment of the present disclosure, the augmented reality device 100 may be implemented as a portable device, and in this case, the augmented reality device 100 may further include a battery for powering the camera 110, the IMU sensor 120, the processor 130, and the display unit 150.

[0059] The camera 110 is configured to obtain a spatial image of the real-world space by photographing the real-world space. In an embodiment of the present disclosure, the camera 110 may include a lens module, an image sensor, and an image processing module. The camera 110 may obtain a still image or video of the real-world space by using an image sensor (e.g., CMOS or CCD). The video may include a plurality of image frames obtained in real time by photographing the real-world space via the camera 110. The image processing module may send a still image including a single image frame or video data including a plurality of image frames obtained via the image sensor to the processor 130. In an embodiment of the present disclosure, the image processing module may process the obtained still image or video, may extract the required information, and may send the extracted information to the processor 130.

[0060] The inertial measurement unit (IMU) sensor 120 is a sensor configured to measure the movement speed, direction, angle, and gravity acceleration of the augmented reality device 100. The IMU sensor 120 may include an accelerometer 122 and a gyroscope sensor 124. Figure 3 , but the IMU sensor 120 may also include a magnetometer.

[0061] The accelerometer 122 is a sensor configured to measure acceleration according to a change in movement when a dynamic force such as acceleration, vibration, or shock is generated in the augmented reality device 100. In an embodiment of the present disclosure, the accelerometer 122 may be configured as a three-axis accelerometer that measures acceleration in a row direction, a horizontal direction, and a height direction.

[0062] The gyroscope sensor 124 is a sensor configured to measure an angular velocity, which is an amount of rotational change of the augmented reality device 100. In an embodiment of the present disclosure, the gyroscope sensor 124 may include a three-axis gyroscope sensor that measures roll, pitch, and yaw angular velocities.

[0063] In an embodiment of the present disclosure, the IMU sensor 120 can measure acceleration and angular velocity by using the accelerometer 122 and the gyroscope sensor 124, and can detect the direction of gravity based on the measured acceleration and angular velocity. The detected "direction of gravity" can be the same as the direction of the normal vector of the floor surface of the real world space. The IMU sensor 120 can provide information about the direction of gravity to the processor 130.

[0064] The processor 130 may execute one or more instructions of a program stored in the memory 140. The processor 130 may include hardware components for performing arithmetic, logic, and input / output operations and image processing. The processor 130 may include, for example, but not limited to, at least one of a central processing unit (CPU), a microprocessor, a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), and a field programmable gate array (FPGA).

[0065] In an embodiment of the present disclosure, the processor 130 may include an artificial intelligence processor that performs artificial intelligence (AI) learning. The AI ​​processor may be manufactured as a dedicated hardware chip for AI, or may be manufactured as part of an existing general-purpose processor (e.g., a CPU or an application processor) or a dedicated graphics processor (e.g., a GPU) and installed on the augmented reality device 100.

[0066] Although processor 130 Figure 3 In the embodiment of the present disclosure, the processor 130 may include a single element, but the present disclosure is not limited thereto. In the embodiment of the present disclosure, the processor 130 may include a plurality of elements.

[0067] For example, the memory 140 may include at least one type of storage medium such as a flash memory type, a hard disk type, a multimedia card micro, a card-type memory (e.g., an SD or XD memory), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), or an optical disk.

[0068] The memory 140 may store instructions related to the operation of the augmented reality device 100 to provide an augmented reality service for controlling an object in a real world space by performing 3D patching and 3D segmentation. In an embodiment of the present disclosure, at least one of instructions, algorithms, data structures, program codes, or applications readable by the processor 130 may be stored in the memory 140. The instructions, algorithms, data structures, and program codes stored in the memory 140 may be implemented in a programming language or scripting language such as C, C++, Java, or assembly language.

[0069] Instructions, algorithms, data structures, or program codes related to the patching module 142, the segmentation module 144, and the rendering module 146 may be stored in the memory 140. The "module" included in the memory 140 refers to a unit that processes a function or operation performed by the processor 130, and may be implemented as software such as instructions, algorithms, data structures, or program codes. In an embodiment of the present disclosure, a 3D model database 148 as a storage space may be included in the memory 140.

[0070] In the following embodiments, the processor 130 may be implemented by executing instructions or program codes stored in the memory 140 .

[0071] The processor 130 may run an augmented reality application, and when the augmented reality application is run, the processor 130 may obtain a spatial image of a real world space from the camera 110. In an embodiment of the present disclosure, the processor 130 may obtain a spatial image including a plurality of image frames obtained by the camera 110 in real time.

[0072] The processor 130 may identify a plane including a wall and a floor from the spatial image. In an embodiment of the present disclosure, the processor 130 may detect a horizontal plane and a vertical plane from the spatial image by using a plane detection technology, and may identify a wall and a floor in the real world space from the detected horizontal plane and vertical plane. However, the present disclosure is not limited thereto, and the processor 130 may identify a plane including a window or a door as well as a wall and a floor from the spatial image. In an embodiment of the present disclosure, the processor 130 may determine whether the identified multiple planes are the same plane via position information, and may integrate the multiple planes into one plane if the multiple planes are the same plane.

[0073] The patching module 142 includes instructions or program codes related to the functions and / or operations of extending the wall and floor along the plane identified from the spatial image, and performs 3D patching on the area of ​​the extended wall and floor hidden by the object by using the spatial image. The processor 130 can run the instructions or program codes of the patching module 142 to obtain the 3D model shape of the wall and floor, and perform patching on the 3D model shape by using the information of the spatial image. In an embodiment of the present disclosure, the processor 130 can derive a plane equation from each of the planes including the identified walls and floors. The processor 130 can extract three points with a high confidence index from each of the identified walls and floors, and can define a plane based on the extracted three points. In an embodiment of the present disclosure, the processor 130 can identify the direction of the normal vector of the floor surface based on the information about the gravity direction obtained from the IMU sensor 120, and can determine whether 90° is formed between the normal vector of the floor surface and the normal vector of the wall surface. When the angle between the normal vector of the floor surface and the normal vector of the wall surface differs from 90° by ±3° to ±5°, the processor 130 may perform plane re-recognition.

[0074] The processor 130 may obtain the 3D model shape of the virtual wall and the virtual floor by extending the wall and the floor based on the derived plane equation. In an embodiment of the present disclosure, the processor 130 may extract the intersection line where the extended wall and the floor intersect each other, and may distinguish the planes of the wall and the floor based on the extracted intersection line. The processor 130 may obtain the vertex coordinates of each of the distinguished planes, and may generate a 3D model shape including the virtual wall and the virtual floor based on the obtained vertex coordinates.

[0075] The processor 130 can identify objects located on the walls and floors in the real world space from the spatial image. The processor 130 can obtain depth information and color information of the real world space from the spatial image, and can identify the object based on at least one of the obtained depth information and color information. In an embodiment of the present disclosure, the processor 130 can identify the 3D position coordinate values ​​of the walls and floors via a plane detection technology, and when a depth value other than the depth value of the z-axis coordinate information including the 3D position coordinate values ​​of the identified walls and floors is obtained, the processor 130 can determine that the object exists. In the case where it is difficult to identify the object by using the depth information, the processor 130 can identify the object by using the color information obtained from the spatial image. The processor 130 can distinguish the identified object from the identified walls and floors.

[0076] The processor 130 can patch the area hidden by the identified object in the 3D model shape including the virtual wall and the virtual floor. In an embodiment of the present disclosure, the processor 130 can patch the area of ​​the wall and the floor hidden by the object by using the information of the spatial image. In the present disclosure, "patching" refers to an image processing technology that restores the hidden, lost or distorted parts of the image. In an embodiment of the present disclosure, the processor 130 can patch the area hidden by the object by using the images of the wall and floor parts of the spatial image corresponding to the wall and the floor, respectively. However, the present disclosure is not limited to this, and in an embodiment of the present disclosure, the processor 130 can patch the area on the wall and the floor hidden by the object by combining the images of the wall and floor parts of the spatial image with the virtual image.

[0077] The processor 130 may generate a 3D model of the real world space by applying the texture of the patched image to the 3D model shape including the virtual wall and the virtual floor. In an embodiment of the present disclosure, the processor 130 may generate a 3D model of the real world space via image processing that applies the texture of the patched image to the 3D model shape.

[0078] In an embodiment of the present disclosure, the processor 130 may store the generated 3D model in the 3D model database 148 in the memory 140. Figures 5 to 8 A specific embodiment in which the processor 130 generates a 3D model of a real world space by performing inpainting is described in detail.

[0079] The augmented reality device 100 may further include an input interface for receiving a user input for selecting a specific object on the spatial image. For example, the input interface may include a keyboard, a mouse, a touch screen, or a voice input device (e.g., a microphone), and may include any other input device that is obvious to a person of ordinary skill in the art. In an embodiment of the present disclosure, the display unit 150 may include a touch screen including a touch panel, and the touch screen may receive a user's touch input, which selects a specific object on the spatial image displayed on the display unit 150. The processor 130 may select an object based on the touch input received from the user. However, the present disclosure is not limited thereto, and in an embodiment of the present disclosure, in the case where the augmented reality device 100 is implemented as augmented reality glasses, the input interface may include a gaze tracking sensor that detects a gaze point where the gaze directions of the user's two eyes converge. In this case, the processor 130 may select an object at the location of the gaze point detected by the gaze tracking sensor. In an embodiment of the present disclosure, when the augmented reality device 100 is implemented as an HMD device, the processor 130 may select an object based on a user input received via an external controller.

[0080] The segmentation module 144 includes instructions or program codes related to functions and / or operations of classifying objects in the spatial image into classes or categories, distinguishing the objects from other objects or background images in the image according to the classification results, and segmenting the objects from the spatial image. The processor 130 may run the instructions or program codes of the segmentation module 144 to identify the object selected by the user input and perform segmentation of the identified object from the spatial image. In an embodiment of the present disclosure, the processor 130 may obtain image information, 2D position coordinate information, and depth value information of the object selected by the user input from the spatial image. The processor 130 may perform 2D segmentation on the object based on at least one of the image information, 2D position coordinate information, and depth information of the object, the plane information obtained via plane detection, and the object distinguishing information. In the present disclosure, "2D segmentation" refers to an image processing technology that distinguishes an object from other objects or background images in an image and segments the 2D contour of the object from the image. In an embodiment of the present disclosure, 2D segmentation may include not only segmenting the 2D contour of the object, but also classifying the object into classes or categories and distinguishing the object from other objects or background images in the image according to the classification results.

[0081] In an embodiment of the present disclosure, the processor 130 may segment an object selected by a user input by using a deep neural network pre-trained to classify multiple objects into labels, classes or categories. The deep neural network model may be an AI model that includes model parameters pre-trained by applying tens of thousands to hundreds of millions of images as input data and applying the label values ​​of the objects included in the images as ground truth. The deep neural network model may include, for example, at least one of a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), and a deep Q network. The deep neural network model may be stored in the memory 140, but the present disclosure is not limited thereto. In an embodiment of the present disclosure, the deep neural network model may be stored in an external server, and the augmented reality device 100 may send the image data of the spatial image to the server, and may receive information about the classification result of the object as an inference result from the deep neural network model of the server.

[0082] Will refer to Fig. 9 A specific embodiment in which the processor 130 segments an object selected by user input via 2D segmentation using a pre-trained deep neural network model is described in detail.

[0083] The processor 130 may run the instructions or program code of the segmentation module 144 to perform 3D segmentation, which segments objects on the spatial image from the real world space. The 3D segmentation may be performed differently depending on whether the 3D model of the object selected by the user input is stored in the 3D model database 148. In an embodiment of the present disclosure, when the 3D model of the object selected by the user input is pre-stored in the 3D model database 148, the processor 130 may load and obtain the pre-stored 3D model from the 3D model database 148, and may arrange the 3D model on the spatial image by rotating the direction of the obtained 3D model in the z-axis direction to overlap with the 2D segmentation contour on the spatial image. The processor 130 may obtain the 3D position coordinate value of the object from the arranged 3D model. The processor 130 may perform 3D segmentation to segment the object from the spatial image based on the obtained 3D position coordinate value.

[0084] In an embodiment of the present disclosure, when the 3D model of the object selected by the user input is not stored in the 3D model database 148, the processor 130 can obtain the 3D position coordinate values ​​of the edges, feature points and pixels of the object from the spatial image, and can perform 3D vertex modeling based on at least one of the obtained 3D position coordinate values ​​of the edges, feature points and pixels of the object. The processor 130 can perform 3D segmentation, which segments the object from the spatial image via 3D vertex modeling.

[0085] Will refer to Figures 10 to 12 A specific embodiment in which the processor 130 performs 3D segmentation differently according to whether a 3D model of an object selected by a user input is stored is described in detail.

[0086] The rendering module 146 includes instructions or program codes related to the functions and / or operations of rendering the area where the object is segmented via 3D segmentation by using the 3D model. The processor 130 can run the instructions or program codes of the rendering module 146 to arrange the 3D models of the wall and the floor at the position of the wall and the floor in the actual real world space, and when the object selected by the user input is segmented, the area deleted by the 3D segmentation is rendered. In an embodiment of the present disclosure, the processor 130 can perform rendering after deleting the original image of the area where the 3D segmentation is performed. However, the present disclosure is not limited thereto, and in an embodiment of the present disclosure, the processor 130 can simply render the 3D models of the wall and the floor, and the 3D model can be arranged on the object in the real world space without deleting the area where the 3D segmentation is performed. In this case, the processor 130 can set the depth test value so that the real world object is rendered on the 3D model by adjusting the depth value between the 3D model and the object.

[0087] The 3D model database 148 is a storage space in the memory 140, in which the 3D models of the shapes of the walls and floors of the real world space generated by the processor 130 are stored. In an embodiment of the present disclosure, the 3D model database 148 can store 3D models of objects. The 3D models of objects stored in the 3D model database 148 can be, for example, 3D models of objects such as furniture (such as a sofa, dining table, table, chair TV or lighting) in an indoor space.

[0088] The 3D model database 148 may include a nonvolatile memory. A nonvolatile memory refers to a storage medium that stores and retains information even when power is not supplied and can use the stored information again when power is supplied. The nonvolatile memory may include, for example, at least one of a flash memory, a hard disk, a solid state drive (SSD), a micro multimedia card, a card-type memory (e.g., an SD or XD memory), a read-only memory (ROM), a magnetic memory, a magnetic disk, and an optical disk.

[0089] Although the 3D model database 148 is included Figure 3 In the memory 140 in the present disclosure, the present disclosure is not limited thereto. In an embodiment of the present disclosure, the 3D model database 148 may be configured as a database separated from the memory 140. In an embodiment of the present disclosure, the 3D model database 148 may be an element of an external third device or an external server, rather than an element of the augmented reality device 100, and may be connected to the augmented reality device 100 via a wired / wireless communication network.

[0090] The display unit 150 is configured to display a spatial image captured via the camera 110 and a 3D model of a real-world space. The display unit 150 may include, for example, at least one of a liquid crystal display, a thin-film transistor liquid crystal display, an organic light-emitting diode, a flexible display, a 3D display, and an electrophoretic display. In an embodiment of the present disclosure, in the case where the augmented reality device 100 is configured as augmented reality glasses, the display unit 150 may also include an optical engine for projecting a virtual image. The optical engine may include a projector configured to generate light for a virtual image and including an image panel, an illumination optical system, and a projection optical system. The optical engine may be located, for example, on a frame or a temple of the augmented reality glasses.

[0091] Figure 4 is a flowchart illustrating an operating method of the augmented reality device 100 according to an embodiment of the present disclosure.

[0092] refer to Figure 4, in operation S410, the augmented reality device 100 identifies a plane including a wall and a floor from a spatial image obtained by photographing a real-world space. In an embodiment of the present disclosure, the augmented reality device 100 may detect a horizontal plane and a vertical plane from a spatial image by using a plane detection technology, and may identify a wall and a floor in the real-world space from the detected horizontal plane and vertical plane. However, the present disclosure is not limited thereto, and the augmented reality device 100 may identify a plane including a window or a door as well as a wall and a floor from a spatial image. In an embodiment of the present disclosure, the augmented reality device 100 may determine whether the identified multiple planes are the same plane via position information, and may integrate the multiple planes into one plane when the multiple planes are the same plane.

[0093] In operation S420, the augmented reality device 100 generates a 3D model of the real world space by extending the wall and floor along the identified plane and performing 3D patching on the area of ​​the extended wall and floor hidden by the object using the spatial image. In an embodiment of the present disclosure, the augmented reality device 100 can derive the plane equation of each of the identified walls and floors, and can obtain the 3D model shape of the virtual wall and the virtual floor by extending the wall and floor based on the derived plane equation. The augmented reality device 100 can recognize objects on the wall and the floor located in the real world space from the spatial image. In an embodiment of the present disclosure, the augmented reality device 100 can obtain depth information and color information of the real world space from the spatial image, and can recognize the object based on at least one of the obtained depth information and color information. The augmented reality device 100 can patch the area hidden by the identified object in the 3D model shape including the virtual wall and the virtual floor. In the present disclosure, "patching" refers to an image processing technique that restores the hidden, lost or distorted part of the image. In an embodiment of the present disclosure, the augmented reality device 100 can repair the area of ​​the virtual wall and the virtual floor that is hidden by the object by using the information of the spatial image. Figures 5 to 8 A specific embodiment in which the augmented reality device 100 generates a 3D model of a real world space via 3D patching is described in detail.

[0094] In operation S430, the augmented reality device 100 performs 2D segmentation, which segments the object selected by the user input from the spatial image. In an embodiment of the present disclosure, the augmented reality device 100 may receive a user input selecting a specific object on the spatial image. For example, the augmented reality device 100 may include a touch screen, which includes a touch panel, and may receive a user's touch input selecting a specific object on the spatial image displayed via the touch screen. The augmented reality device 100 may identify the object selected by the user input, and may perform segmentation of the object identified from the spatial image segmentation. In an embodiment of the present disclosure, the augmented reality device 100 may obtain image information, 2D position coordinate information, and depth value information of the object selected by the user input from the spatial image. The augmented reality device 100 may perform 2D segmentation on the object based on at least one of the image information, 2D position coordinate information, and depth information of the object, the plane information obtained via plane detection, and the object distinction information.

[0095] In an embodiment of the present disclosure, the augmented reality device 100 may segment an object by using a deep neural network pre-trained to classify objects into labels, classes, or categories.

[0096] In operation S440, the augmented reality device 100 performs 3D segmentation, which segments the object on the spatial image from the real world space based on the 3D model or 3D position information of the selected object. In an embodiment of the present disclosure, when the 3D model of the object selected by the user input is stored in the storage space in the augmented reality device 100, the augmented reality device 100 can load and obtain the pre-stored 3D model, and can perform 3D segmentation, which segments the object from the spatial image by using the obtained 3D model. In an embodiment of the present disclosure, when the 3D model of the object selected by the user input is not pre-stored, the augmented reality device 100 can perform 3D segmentation, which segments the object from the spatial image based on at least one of the 3D position coordinate values ​​of the edge, feature points and pixels of the object obtained from the spatial image.

[0097] Figure 5 is a flowchart illustrating a method in which the augmented reality device 100 generates a 3D model of a real world space according to an embodiment of the present disclosure.

[0098] Figure 5 Operations S510 to S550 are Figure 4 The detailed operation of operation S420 can be performed in Figure 4 After operation S410 is performed Figure 5 In the operation S510. Figure 5 After operation S550, you can perform Figure 4 Operation S430.

[0099] In the following, reference will be made to Figures 6 to 8 The embodiment of the present invention describes operations S510 to S550.

[0100] In operation S510, the augmented reality device 100 derives a plane equation from each of the planes including the identified walls and floors. In an embodiment, the augmented reality device 100 may run an application to start augmented reality, and may detect horizontal and vertical planes from a spatial image of the real world space via a plane detection technique of an augmented reality session. The augmented reality device 100 may identify walls and floors in the real world space from the detected horizontal and vertical planes. In this case, the augmented reality session may include determining whether the real world space is an environment suitable for performing augmented reality space recognition and notifying that the real world space is stably recognized. In an embodiment of the present disclosure, the augmented reality device 100 may determine whether the identified multiple planes are the same plane via position information, and may integrate the multiple planes into one plane when it is determined that the multiple planes are the same plane.

[0101] Figure 6 FIG. 1 is a diagram showing that the augmented reality device 100 according to an embodiment of the present disclosure obtains a plane (eg, P 1 To P 3 ) is a diagram of an operation of a 3D model shape 610 associated with a shape. Figure 6 Together with Figure 5 In operation S510, the augmented reality device 100 may recognize a plurality of planes (eg, P 1 To P 3 ) extracts the 3D position coordinate values ​​of the three points. In the embodiment of the present disclosure, the processor 130 (see Figure 3 ) can be selected from multiple planes (e.g., P 1 To P 3 ) obtains depth value information of a plurality of points, and three points with high confidence values ​​may be selected. Here, the “confidence value” is calculated by the augmented reality session and may be based on the depth information of the IMU sensor 120 (see FIG. 1 ) when the orientation (or field of view (FOV)) of the augmented reality device 100 changes. Figure 3 ) is calculated based on the sensor noise of the feature point measured. For example, as the sensor noise decreases, the confidence value of the feature point can increase. In an embodiment of the present disclosure, when the confidence value of the feature point is calculated based on the sensor noise of the feature point measured from multiple points (for example, P 1 To P 3 ) is the same, the processor 130 may arbitrarily select three points from a plurality of points having the same confidence value. Figure 6In an embodiment, the processor 130 may be configured to receive a signal from the first plane P 1 Three points 611, 612 and 613 with high confidence values ​​are extracted and can be obtained from the second plane P 2 Extract three points 621, 622, and 623. Although not shown, the processor 130 may extract three points 621, 622, and 623 from the third plane P 3 Extract three points.

[0102] The processor 130 may be based on the information from multiple planes (eg, P 1 To P 3 ) to derive the plane equation. In the embodiment of the present disclosure, in multiple planes (for example, P 1 To P 3 ) when the angle of the point where the planes intersect each other is equal to or less than a threshold, the processor 130 may determine that the planes are a single plane and may integrate the planes into one plane equation. The processor 130 may use the IMU sensor 120 (see Figure 3 ) to obtain the second plane P of the wall 2 and the third plane P 3 The processor 130 can identify the normal vector of the wall obtained and the first plane P of the floor. 1 The normal vectors form a 90° angle. 1 The normal vector is the second plane P of the wall 2 and the third plane P 3 When the angle formed by the normal vectors differs from 90° by ±3° to ±5°, the processor 130 may perform plane detection again.

[0103] Return to reference Figure 5 In operation S520, the augmented reality device 100 obtains a 3D model shape of a virtual wall and a virtual floor by extending the wall and the floor based on a plane equation. Figure 6 , the processor 130 of the augmented reality device 100 may expand the plurality of planes (eg, P 1 , P 2 and P 3 ) to generate a virtual plane (e.g., P 1 '、P 2 ' and P 3 '). For example, the first plane P of the floor 1 Can be extended to a virtual floor P 1 ', and is the second plane P of the wall 2 and the third plane P 3 Can be extended to virtual wall P 2 ' and P 3In an embodiment of the present disclosure, the processor 130 may extract an extended virtual plane (eg, P 1 '、P 2 ' and P 3 ') intersecting lines l 1 , l 2 and l 3 , and can be based on the extracted intersection line l 1 , l 2 and l 3 To distinguish the virtual floor P 1 ' and virtual wall P 2 ' and P 3 '. The processor 130 may obtain the distinguished virtual plane (eg, P 1 '、P 2 ' and P 3 ') vertex coordinates V 1 To V 9 , and can be based on the vertex coordinates V 1 To V 9 Get including virtual floor P 1 ' and virtual wall P 2 ' and P 3 '3D model shape 610.

[0104] The processor 130 may store the 3D model shape in a storage space in the augmented reality device 100. In an embodiment of the present disclosure, the processor 130 may store the 3D model shape 610 and the position information in the memory 140 (see Figure 3 ) in the 3D model database 148 (see Figure 3 )middle.

[0105] Return to reference Figure 5 In operation S530, the augmented reality device 100 recognizes objects on the wall and the floor located in the real world space from the space image. In an embodiment of the present disclosure, the augmented reality device 100 can distinguish each wall, floor, and object by using depth information and color information based on plane detection information of the wall and the floor.

[0106] Fig. 7A 2 is a diagram illustrating an operation of the augmented reality device 100 to distinguish the planes of the wall and the floor on the space image 700a according to an embodiment of the present disclosure. Figure 5 Operation S530 together with Fig. 7A, the processor 130 of the augmented reality device 100 can identify multiple planes (e.g., 711, 721, and 722) by using a plane detection technology, and can distinguish the floor surface 711 and the wall surfaces 721 and 722 from the multiple planes (e.g., 711, 721, and 722) based on the color information obtained from the spatial image 700a.

[0107] Figure 7B 2 is a diagram illustrating an operation of the augmented reality device 100 to distinguish the planes of the wall and the window on the space image 700b according to an embodiment of the present disclosure. Figure 5 Operation S530 together with Figure 7B , the processor 130 of the augmented reality device 100 can identify multiple planes (e.g., 721, 722, 731, and 732) from the spatial image 700b by using a plane detection technology, and can distinguish between wall surfaces 721 and 722 and windows 731 and 732 based on depth information of the real world space and color information of the spatial image 700b.

[0108] Figure 7C 2 is a diagram illustrating an operation of the augmented reality device 100 to distinguish planes of walls and floors having different patterns on a space image 700c according to an embodiment of the present disclosure. Figure 5 Operation S530 together with Figure 7C , the processor 130 of the augmented reality device 100 can identify a plurality of planes (e.g., 711, 721, and 723) from the spatial image 700c by using a plane detection technology, and can identify the floor surface 711 and the wall surfaces 721 and 723 based on the depth information of the real world space. The processor 130 can obtain color information of the spatial image 700c, and can distinguish the wall surfaces 721 and 723 having different patterns based on the obtained color information.

[0109] The processor 130 of the augmented reality device 100 can obtain the 3D position coordinate value of the real world space via the augmented reality session, and in the case where the depth value including the z-axis coordinate information is not recognized as a wall and a floor, it can be determined that the depth value corresponds to an object. In an embodiment of the present disclosure, in the case where it is difficult to recognize an object by using depth information, the processor 130 can recognize the object by using color information obtained from the space image.

[0110] Figure 8 2 is a diagram showing an operation of performing 3D inpainting by the augmented reality device 100 according to an embodiment of the present disclosure. Figure 5 Operation S530 together with Figure 8, the processor 130 of the augmented reality device 100 can obtain a spatial image 800 of the real world space via the augmented reality session, and can identify a wall surface 810 and an object 820 from the spatial image 800. The processor 130 can obtain 3D position coordinate values ​​of the real world space, and can identify a region having a depth value (which is z-axis coordinate information) different from that of the wall surface 810 as the object 820. In an embodiment of the present disclosure, the processor 130 can obtain color information of the spatial image 800, and can identify a region having a color significantly different from that of the wall surface 810 as the object 820. Figure 8 In the embodiment of FIG. 8 , the processor 130 may distinguish between the wall surface 810 and the object 820 (eg, a chair) based on the depth information of the real world space and the color information of the spatial image 800 .

[0111] Return to reference Figure 5 In operation S540, the augmented reality device 100 repairs the area of ​​the wall and the floor hidden by the object by using the information of the spatial image. The augmented reality device 100 can recognize the area hidden by the object on the wall and the floor of the spatial image. Figure 8 , the processor 130 of the augmented reality device 100 can identify the area 830 hidden by the object from the spatial image 800. The processor 130 can repair the area 830 hidden by the object in the area of ​​the wall and the floor by using the color information of the spatial image 800. In the present disclosure, "repairing" refers to an image processing technique that restores a hidden, lost or distorted portion in an image.

[0112] The processor 130 may inpaint the hidden area 830 in the 3D model shape by using the images of the wall and floor portions. In an embodiment of the present disclosure, the processor 130 may perform inpainting separately on each of the wall and the floor. In the case where the window is identified separately from the wall and the floor, the processor 130 may perform inpainting separately on each of the wall, the floor, and the window. Figure 8 In the embodiment of FIG. 8 , the processor 130 may repair the area 830 hidden by the object by using color information of the wall surface 810 of the spatial image 800 , and may obtain a repaired image 840 .

[0113] The processor 130 may inpaint an area corresponding to the area 830 hidden by the object on the virtual wall and virtual floor of the 3D model shape by combining the images of the wall and floor parts with the virtual image.

[0114] Return to reference Figure 5 In operation S550, the augmented reality device 100 generates a 3D model of the real world space by applying the texture of the patched image to the 3D model shapes of the virtual wall and the virtual floor. Figure 8 , the processor 130 of the augmented reality device 100 may perform image processing to apply the patched image 840 as a texture to the generated 3D model shape. For example, the processor 130 may generate a 3D model of the real world space by applying the patched image 840 to the 3D model shape obtained in operation S520.

[0115] In an embodiment of the present disclosure, the augmented reality device 100 may store the 3D model generated by patching in a storage device (eg, Figure 3 The 3D model database 148 in the memory 140 (see Figure 3 ))middle.

[0116] Fig. 9 is a diagram illustrating an operation of the augmented reality device 100 performing 2D segmentation from a spatial image 900 according to an embodiment of the present disclosure.

[0117] refer to Fig. 9 , the augmented reality device 100 can run an augmented reality application to initiate an augmented reality session, and can use the camera 110 (see Figure 3 ) captures the real world space 10 to obtain a space image 920. The augmented reality device 100 can obtain 3D position information of the real world space 10, and can recognize walls, floors, and objects by using a plane detection technology. In this case, the augmented reality session may include determining whether the real world space is an environment suitable for performing augmented reality space recognition and notifying that the real world space is stably recognized.

[0118] The augmented reality device 100 may obtain 3D position information of the real world space 10 via the augmented reality session, and may obtain a depth map 910 including depth information, which is a z-axis coordinate value among the 3D position coordinate values.

[0119] The augmented reality device 100 can identify the planes of the wall and the floor (eg, P 1 , P 2 and P 3 ), and can be identified by extending the plane (e.g. P 1 , P 2 and P 3 ) to obtain the 3D model shape 930. The specific method and reference of the augmented reality device 100 to obtain the 3D model shape 930 Figure 5 and Figure 6 The methods described are the same, and therefore, their repeated descriptions will be omitted.

[0120] The augmented reality device 100 may receive a user input for selecting a specific object on the spatial image displayed via the display unit 150. In an embodiment of the present disclosure, the display unit 150 may include a touch screen, and the augmented reality device 100 may receive a user's touch input for selecting a specific object on the spatial image 920 displayed via the touch screen. Fig. 9 In the embodiment of FIG. 9 , the augmented reality device 100 may receive a touch input of a user selecting a table in the spatial image 920 .

[0121] In response to receiving a user input selecting an object, the processor 130 (see Figure 3 ) can obtain image information of the object selected by the user input, depth information (depth map data in the form of a 2D image, which indicates the depth values ​​of pixels constituting the object), 2D position coordinate value information (x-axis and y-axis coordinate values) on the spatial image 920 of the selected object, and 2D position coordinate value information (x-axis and y-axis coordinate values) and 3D position coordinate value information (x-axis, y-axis, and z-axis coordinate values) of the main feature points (AR feature points). The processor 130 can identify the planes of the walls and floors (e.g., P 1 , P 2 and P 3 ), can identify objects such as furniture or household appliances in the real world space based on depth information, and can identify the positional relationship between the identified objects and the walls and floors.

[0122] The augmented reality device 100 can perform 2D segmentation on the object selected by the user input by using the depth information of the depth map 910, the spatial image 920, the 2D position coordinates of the object selected by the user, and the wall, floor, and wall distinction information obtained from the 3D model shape 930. Here, '2D segmentation' refers to an image processing technology that distinguishes an object from other objects or background images in an image and segments the 2D outline of the object from the image. In an embodiment of the present disclosure, 2D segmentation may include not only segmenting the 2D outline of the object, but also classifying the object into classes or categories, and distinguishing the object from other objects or background images in the image based on the classification result.

[0123] In an embodiment of the present disclosure, the processor 130 of the augmented reality device 100 may input at least one of the depth information of the depth map 910, the image information of the spatial image 920, the 2D position coordinate information of the object, and the object distinction information to the pre-trained deep neural network model 940, and may perform 2D segmentation on the object by performing inference using the deep neural network model 940. The deep neural network model 940 may be an AI model including model parameters pre-trained by applying tens of thousands to hundreds of millions of images as input data and applying the label value of the object included in the image as the ground truth. In an embodiment of the present disclosure, the deep neural network model 940 may be an AI model pre-trained for each furniture, each home appliance, or for each category of furniture such as a chair or a sofa. The deep neural network model 940 may include at least one of a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), and a deep Q network.

[0124] The processor 130 may obtain a segmented image 950 in which a 2D contour of an object 952 is segmented from other objects or a background image via inference using a deep neural network model 940 .

[0125] Fig.10 is a flowchart illustrating a method in which the augmented reality device 100 performs 3D segmentation based on whether a 3D model of an object is stored according to an embodiment of the present disclosure.

[0126] Fig.10 Operations S1010 to S1070 are Figure 4 The detailed operation of operation S440 can be performed in Figure 4 After operation S430 is performed Fig.10 Operation S1010.

[0127] In operation S1010, the augmented reality device 100 determines whether there is a pre-stored 3D model of the object. The processor 130 (see Figure 3 ) can determine whether the 3D model of the object selected by the user input is stored in the memory 140 (see Figure 3 ) in the storage space in the . The operation may be performed differently depending on whether a 3D model of the object is stored in advance. Fig.10 3D segmentation method. Fig.11 An embodiment of pre-storing a 3D model of an object is described, and reference will be made to Fig.12 Embodiments are described in which a 3D model of an object is not stored.

[0128] When the 3D model of the object is stored in advance (operation S1020 ), the augmented reality apparatus 100 arranges the 3D model to overlap the 2D segmentation contour on the space image by adjusting the direction of the 3D model. Fig.11 2 is a diagram illustrating an operation of the augmented reality device 100 performing 3D segmentation by using a pre-stored 3D model of an object according to an embodiment of the present disclosure. Fig.10 Operation S1020 together with Fig.11 , the 3D models 1101 to 1103 of the plurality of objects may be stored in the 3D model database 148. The processor 130 (see Figure 3 ) The 3D model 1101 of the first object selected by the user input may be identified from among the 3D models 1101 to 1103 of the plurality of objects pre-stored in the 3D model database 148, and the identified 3D model 1101 of the first object may be loaded from the 3D model database 148. The processor 130 may rotate the loaded 3D model 1101 of the first object along the Z axis until the 3D model 1101 has the same shape as the 2D contour generated in the spatial image 1100 according to the 2D segmentation result. In this case, the resolution of the rotation angle may be freely determined according to the required accuracy. The processor 130 may arrange the 3D model 1101 of the first object whose orientation is adjusted according to the rotation result on the 2D contour on the spatial image 1100.

[0129] Return to reference Fig.10 In operation S1030, the augmented reality device 100 obtains the 3D position coordinate value of the object from the arranged 3D model. Fig.11 , the 3D model 1101 of the first object may be arranged to fit the 2D contour on the space image 1100 , and 3D position coordinate values ​​of a plurality of feature points constituting the first object may be obtained according to the arrangement result.

[0130] refer to Fig.10 In operation S1040, the augmented reality device 100 performs 3D segmentation, which segments an object from the spatial image based on the obtained 3D position coordinate value. Fig.11 , the processor 130 of the augmented reality device 100 can obtain a segmented image 1120, in which the first object is segmented from the image of the real world space based on the 3D position coordinate values ​​of a plurality of feature points constituting the first object.

[0131] Return to reference Fig.10 , when the 3D model of the object is not stored (operation S1050 ), the augmented reality apparatus 100 obtains 3D position coordinate values ​​of edges, feature points, and pixels of the object recognized from the spatial image. Fig.122 is a diagram illustrating an operation of the augmented reality device 100 performing 3D segmentation when a 3D model of an object is not stored according to an embodiment of the present disclosure. Fig.10 Operation S1050 together with Fig.12 , the processor 130 of the augmented reality device 100 can obtain the 3D position coordinate values ​​of the edges, feature points and pixels constituting the first object 1210 selected by the user input in the spatial image 1200.

[0132] Return to reference Fig.10 In operation S1060, the augmented reality device 100 performs 3D vertex modeling based on at least one of the 3D position coordinate values ​​of the edge, feature point, and pixel of the object. Fig.12 , the processor 130 of the augmented reality device 100 can input the 3D position coordinate values ​​of the edges, feature points, and pixels of the object into a pre-trained deep learning neural network, and can perform 3D vertex modeling by using a deep neural network model. The deep neural network model can be an AI model pre-trained for each furniture, each home appliance, or each category of furniture such as a chair or a sofa. The deep neural network model may include at least one of a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recursive deep neural network (BRDNN), and a deep Q network. However, the deep neural network model of the present disclosure is not limited to the above examples.

[0133] Return to reference Fig.10 In operation S1070, the augmented reality device 100 performs 3D segmentation, which segments an object from a spatial image via 3D vertex modeling. Fig.12 The processor 130 of the augmented reality device 100 may obtain the 3D position coordinate value of the first object 1220 via 3D vertex modeling. The processor 130 may obtain a segmented image 1230 in which the first object 1220 is segmented from the spatial image 1200 based on the 3D position coordinate values ​​of a plurality of pixels constituting the first object 1220.

[0134] Fig.13 is a flowchart illustrating a method by which the augmented reality device 100 arranges a 3D model in a real world space and performs rendering by using the 3D model according to an embodiment of the present disclosure.

[0135] Can be executed Figure 4 After operation S440 is performed Fig.13 Operation S1310.

[0136] In operation S1310, the augmented reality device 100 arranges the generated 3D model at the positions of the wall and the floor in the real world space.

[0137] In operation S1320, the augmented reality device 100 renders an area in which an object is segmented via 3D segmentation by using a 3D model. When the object selected by the user input is segmented via 3D segmentation, the augmented reality device 100 can render the area deleted due to the 3D segmentation by using the 3D model. In an embodiment of the present disclosure, the augmented reality device 100 may perform rendering after deleting the original image of the area where the 3D segmentation is performed. However, the present disclosure is not limited thereto, and in an embodiment of the present disclosure, the augmented reality device 100 may simply render a 3D model of a wall and a floor, and may arrange the 3D model on an object in a real-world space without deleting the area where the 3D segmentation is performed. In this case, the augmented reality device 100 may set a depth test value so that the real-world object is rendered on the 3D model by adjusting the depth value between the 3D model and the object.

[0138] Fig.14 is a flowchart illustrating a method by which the augmented reality device 100 additionally performs 3D segmentation and updates the segmentation result according to an embodiment of the present disclosure.

[0139] Can be executed Figure 4 After operation S440 is performed Fig.14 Operation S1410.

[0140] In operation S1410, the augmented reality device 100 tracks the 3D segmentation result in the augmented reality space. In an embodiment of the present disclosure, in the case where the orientation or field of view (FOV) of the augmented reality device 100 is changed by the manipulation of the user, the augmented reality device 100 can track the 3D segmentation result in the augmented reality space by using the IMU sensor 120 (see Figure 3 ) to track the region segmented via 3D segmentation in the augmented reality space. Via the tracking operation, real-time 3D segmentation of the object is possible.

[0141] In operation S1420, the augmented reality device 100 additionally obtains a spatial image of the real world space. The augmented reality device 100 may run an augmented reality application to obtain a spatial image of the real world space via the camera 110 (see Figure 3 ) photographs the real world space and additionally obtains a space image. In an embodiment of the present disclosure, the augmented reality device 100 may periodically obtain a space image at a preset time interval.

[0142] In operation S1430, the augmented reality apparatus 100 extracts a contour of the object by performing 2D segmentation on the additionally obtained spatial image. The operation of the augmented reality apparatus 100 to extract a contour of the object by performing 2D segmentation is similar to the operation of the reference image. Fig. 9 The operations described are the same, and therefore, repeated descriptions thereof will be omitted.

[0143] In operation S1440 , the augmented reality apparatus 110 measures similarity by comparing a contour of a region in which the object is segmented via the 3D segmentation with a contour extracted via the 2D segmentation.

[0144] In operation S1450, the augmented reality device 100 compares the measured similarity with a preset threshold value α.

[0145] In a case where the measured similarity is less than the threshold value α (operation S1460), the augmented reality device 100 additionally performs 3D segmentation. In a case where the similarity is less than the threshold value α, the augmented reality device 100 may recognize that an error has occurred in the 3D segmentation result due to a change in orientation or FOV.

[0146] In operation S1470, the augmented reality apparatus 100 updates the segmentation result through the additionally performed 3D segmentation.

[0147] In a case where the measured similarity exceeds the preset threshold value α, the augmented reality device 100 may return to operation S1420 and may repeatedly perform an operation of additionally obtaining a spatial image.

[0148] The present disclosure provides an augmented reality device 100 for providing an augmented reality service for controlling an object in a real world space. The augmented reality device 100 according to an embodiment of the present disclosure may include a camera 110 (see Figure 3 ), including an accelerometer 122 (see Figure 3 ) and the gyro sensor 124 (see Figure 3 ) of the IMU sensor 120 (see Figure 3 ), a memory 140 storing at least one instruction (see Figure 3 ), and at least one processor 130 configured to execute at least one instruction (see Figure 3 ). The at least one processor 130 may obtain a spatial image by photographing the real world space using the camera 110. The at least one processor 130 may identify a plane including a wall and a floor from the obtained spatial image. The at least one processor 130 may generate a 3D model of the real world space by extending the wall and the floor along the identified plane and performing 3D repair on an area of ​​the extended wall and the floor hidden by the object using the spatial image. The at least one processor 130 may perform 2D segmentation that segments an object selected by a user input from the spatial image. The at least one processor 130 may perform 3D segmentation that segments an object on the spatial image from the real world space based on a 3D model or 3D position information of the object.

[0149] In an embodiment of the present disclosure, the at least one processor 130 may arrange the generated 3D model at the position of the wall and the floor in the real world space, and may render the area in which the object is segmented via the 3D segmentation by using the 3D model.

[0150] In an embodiment of the present disclosure, at least one processor 130 may derive a plane equation from each of the planes including the identified walls and floors. At least one processor 130 may obtain a 3D model shape of a virtual wall and a virtual floor by extending the wall and the floor based on the derived plane equation. At least one processor 130 may recognize objects on the wall and the floor located in the real world space from the spatial image. At least one processor 130 may patch areas of the wall and the floor that are hidden by the identified objects by using information of the spatial image. At least one processor 130 may generate a 3D model of the real world space by applying the texture of the patched image to the 3D model shape of the virtual wall and the virtual floor.

[0151] In an embodiment of the present disclosure, at least one processor 130 may extract an intersection line where the extended wall and floor intersect each other, and may distinguish the planes of the wall and floor based on the extracted intersection line. At least one processor 130 may obtain vertex coordinates of each of the distinguished planes, and may generate a 3D model shape of a virtual wall and a virtual floor based on the obtained vertex coordinates.

[0152] In an embodiment of the present disclosure, at least one processor 130 may recognize an object based on at least one of depth information and color information of a real world space obtained from an air image. At least one processor 130 may distinguish the recognized wall and floor from the recognized object.

[0153] In an embodiment of the present disclosure, the at least one processor 130 may store the generated 3D model in a storage space in the memory 140 .

[0154] In an embodiment of the present disclosure, at least one processor 130 may determine whether a 3D model of the selected object is pre-stored in the memory 140, and in the case of determining that the 3D model of the selected object is stored, the at least one processor 130 may arrange the 3D model by adjusting the direction of the pre-stored 3D model to overlap with the 2D segmentation contour on the spatial image. The at least one processor 130 may obtain the 3D position coordinate value of the object from the arranged 3D model. The at least one processor 130 may perform 3D segmentation, which segments the object from the spatial image based on the obtained 3D position coordinate value.

[0155] In an embodiment of the present disclosure, at least one processor 130 may determine whether a 3D model of the selected object is stored in the memory 140, and may obtain 3D position coordinate values ​​of edges, feature points, and pixels of the object identified from the spatial image if it is determined that the 3D model of the selected object is not stored. At least one processor 130 may perform 3D vertex modeling based on at least one of the obtained 3D position coordinate values ​​of the edges, feature points, and pixels of the object. At least one processor 130 may perform 3D segmentation, which segments the object from the spatial image via 3D vertex modeling.

[0156] In an embodiment of the present disclosure, at least one processor 130 may additionally obtain a spatial image when the orientation or field of view of the augmented reality device 100 changes, and may extract the contour of the object by performing 2D segmentation on the additionally obtained spatial image. The at least one processor 130 may measure similarity by comparing the 2D contour of the area in which the object is segmented via 3D segmentation with the extracted contour. The at least one processor 130 may determine whether to additionally perform 3D segmentation by comparing the similarity with a preset threshold.

[0157] In an embodiment of the present disclosure, in a case where the similarity is less than a threshold value, the at least one processor 130 may additionally perform 3D segmentation, and may update the segmentation result via the additionally performed 3D segmentation.

[0158] The present disclosure provides a method for providing an augmented reality for controlling an object in a real-world space, which is performed by an augmented reality device 100. The method may include an operation S410 of identifying a plane including a wall and a floor from a spatial image obtained by photographing the real-world space using a camera 110. The method may include an operation S420 of generating a 3D model of the real-world space by extending the wall and the floor along the identified plane and performing 3D patching on the area of ​​the extended wall and the floor hidden by the object using the spatial image. The method may include an operation S430 of performing 2D segmentation, which segments an object selected by a user input from the spatial image. The method may include an operation S440 of performing 3D segmentation, which segments an object on the spatial image from the real-world space based on a 3D model or 3D position information of the object.

[0159] In an embodiment of the present disclosure, the method may further include operation S1310 of arranging the generated 3D model at the positions of the wall and the floor in the real world space and operation S1320 of rendering the area in which the object is segmented via the 3D segmentation by using the 3D model.

[0160] In an embodiment of the present disclosure, the operation S420 of generating a 3D model of the real world space may include an operation S510 of deriving a plane equation from each of the planes including the identified walls and floors. The operation S420 of generating the 3D model of the real world space may include an operation S520 of obtaining a 3D model shape of a virtual wall and a virtual floor by extending the wall and the floor based on the derived plane equation. The operation S420 of generating the 3D model of the real world space may include an operation S530 of recognizing objects located on the walls and the floor in the real world space from a spatial image. The operation S420 of generating the 3D model of the real world space may include an operation S540 of patching an area of ​​the wall and the floor hidden by the identified object by using information of the spatial image. The operation S420 of generating the 3D model of the real world space may include an operation S550 of generating the 3D model of the real world space by applying the texture of the patched image to the 3D model shape of the virtual wall and the virtual floor.

[0161] In an embodiment of the present disclosure, operation S520 of generating a 3D model shape of a virtual wall and a virtual floor may include extracting intersection lines where the extended wall and the floor intersect each other, distinguishing planes of the wall and the floor based on the extracted intersection lines, obtaining vertex coordinates of each of the distinguished planes, and generating a 3D model shape of the virtual wall and the virtual floor based on the obtained vertex coordinates.

[0162] In an embodiment of the present disclosure, operation S530 of recognizing objects on the walls and floor located in the real-world space may include recognizing the objects based on at least one of depth information and color information of the real-world space obtained from the spatial image and distinguishing the recognized walls and floor from the recognized objects.

[0163] The operation S440 of performing 3D segmentation may include an operation S1010 of determining whether the 3D model of the selected object is pre-stored, and an operation S1020 of arranging the 3D model to overlap the 2D segmentation contour on the spatial image by adjusting the direction of the pre-stored 3D model if it is determined that the 3D model of the selected object is pre-stored. The operation S440 of performing 3D segmentation may include an operation S1030 of obtaining a 3D position coordinate value of the object from the arranged 3D model. The operation S440 of performing 3D segmentation may include an operation S1040 of performing 3D segmentation that segments the object from the spatial image based on the obtained 3D position coordinate value.

[0164] In an embodiment of the present disclosure, the operation S440 of performing 3D segmentation may include an operation S1010 of determining whether a 3D model of the selected object is pre-stored, and an operation S1050 of obtaining 3D position coordinate values ​​of edges, feature points, and pixels of the object recognized from the spatial image if it is determined that the 3D model of the selected object is not stored. The operation S440 of performing 3D segmentation may include an operation S1060 of performing 3D vertex modeling based on at least one of the obtained 3D position coordinate values ​​of the edges, feature points, and pixels of the object. The operation S440 of performing 3D segmentation may include an operation S1070 of performing 3D segmentation, which segments the object from the spatial image via 3D vertex modeling.

[0165] In an embodiment of the present disclosure, the method may further include an operation S1420 of additionally obtaining a spatial image and extracting a contour of an object by performing 2D segmentation on the additionally obtained spatial image when the orientation or field of view (FOV) of the augmented reality device 100 is changed. The method may further include an operation S1430 of measuring similarity by comparing the 2D contour of the region in which the object is segmented via 3D segmentation with the extracted contour. The method may further include an operation S1440 of determining whether to additionally perform 3D segmentation by comparing the similarity with a preset threshold.

[0166] In an embodiment of the present disclosure, the method may further include: when the similarity is less than a threshold, additionally performing 3D segmentation and updating the segmentation result via the additionally performed 3D segmentation operation S1460. The updating of the segmentation result may be performed periodically at preset intervals.

[0167] The present disclosure provides a computer program product including a computer-readable storage medium. The storage medium may include instructions that can be read by an augmented reality device 100 to enable the augmented reality device 100 to recognize a plane including a wall and a floor from a spatial image obtained by photographing a real-world space using a camera 110; generate a 3D model of the real-world space by extending the wall and the floor along the recognized plane and performing 3D repair on the area of ​​the extended wall and the floor hidden by the object using the spatial image; perform 2D segmentation that segments an object selected by a user input from the spatial image; and perform 3D segmentation that segments an object on the spatial image from the real-world space based on the 3D model or 3D position information of the object.

[0168] The program executed by the augmented reality device 100 described in the present disclosure may be implemented as a hardware component, a software component, and / or a combination of hardware and software components. The program may be executed by any system capable of executing computer-readable instructions.

[0169] Software may include computer programs, codes, instructions, or a combination of one or more thereof, and may configure a processing device to operate as desired or instruct the processing device independently or collectively.

[0170] Software may be implemented as a computer program including instructions stored in a computer-readable storage medium. Examples of computer-readable recording media include magnetic storage media (e.g., read-only memory (ROM), random access memory (RAM), floppy disks, or hard disks) and optical recording media (e.g., compact disk ROM (CD-ROM) or digital versatile disk (DVD)). Computer-readable recording media may be distributed in computer systems connected in a network so that computer-readable code is stored and executed in a distributed manner. The medium may be computer-readable, may be stored in a memory, and may be executed by a processor.

[0171] The computer-readable storage medium may be provided in the form of a non-transitory storage medium. In this case, "non-transitory" means that the storage medium does not include a signal and is tangible, but does not distinguish whether the data is stored in the storage medium semi-permanently or temporarily. For example, a "non-transitory storage medium" may include a buffer that temporarily stores data.

[0172] Furthermore, the program according to the embodiment of the present disclosure may be provided in a computer program product. The computer program product can be traded between a seller and a buyer as a product.

[0173] The computer program product may include a software program and a computer-readable storage medium storing the software program. For example, the computer program product may include a software program-type product (e.g., a downloadable application) electronically distributed through an electronic market (e.g., Samsung Galaxy Store) or a manufacturer of the augmented reality device 100. For electronic distribution, at least a portion of the software program may be stored in a storage medium or may be temporarily generated. In this case, the storage medium may be a storage medium of a server of the manufacturer of the augmented reality device 100, a server of an electronic market, or a relay server that temporarily stores the software program.

[0174] The computer program product may include a storage medium of the server or a storage medium of the augmented reality device 100 in a system including the server and / or the augmented reality device 100. Alternatively, when there is a third device (e.g., a wearable device) communicatively connected to the augmented reality device 100, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include a software program transmitted from the augmented reality device 100 to the third device, or from the third device to the electronic device.

[0175] In this case, the augmented reality device 100 or one of the third devices may execute a computer program product to perform the method according to the disclosed embodiment. Alternatively, at least one of the augmented reality device 100 and the third device may execute a computer program product to perform the method according to the disclosed embodiment in a distributed manner.

[0176] For example, the augmented reality device 100 may execute a program stored in the memory 140 (see Figure 3 ) to control another electronic device (e.g., a wearable device) communicatively connected to the augmented reality device 100 to perform the method according to the disclosed embodiment.

[0177] In another example, the third device may execute the computer program product to control an electronic device communicably connected to the third device to perform a method according to the disclosed embodiments.

[0178] In the case where the third device executes the computer program product, the third device may download the computer program product from the augmented reality device 100 and may execute the downloaded computer program product. Alternatively, the third device may execute the computer program product provided in a preloaded state to perform the method according to the disclosed embodiment.

[0179] Although the embodiments have been described through the limited embodiments and drawings as described above, various modifications and changes may be made by those skilled in the art based on the above description. For example, the described techniques may be performed in a different order than the described methods, and / or the described elements (such as computer systems and modules) may be combined or integrated in a different form than the described methods, or may be replaced or substituted by other elements or equivalents to achieve appropriate results.

Claims

1. An augmented reality device (100), comprising: Camera (110); an inertial measurement unit (IMU) sensor (120) comprising an accelerometer (122) and a gyroscope sensor (124); a memory (140) storing at least one instruction; as well as At least one processor (130) is configured to execute the at least one instruction to: By using the camera (110) to photograph the real world space to obtain a space image, identifying a plane including a wall and a floor from the acquired spatial image, generating a three-dimensional (3D) model of the real world space by extending the wall and floor along the identified plane and performing 3D inpainting on areas of the extended wall and floor that are hidden by the object by using the spatial image, performing two-dimensional (2D) segmentation that segments an object selected by a user input from the spatial image, and 3D segmentation is performed, the 3D segmentation segmenting the object on the spatial image from the real world space based on the 3D model or 3D position information of the object.

2. The augmented reality device (100) according to claim 1, wherein: The at least one processor (130) is further configured to execute the at least one instruction to: placing the generated 3D model at the location of the wall and floor in real world space, and The region in which the object is segmented via 3D segmentation is rendered by using the 3D model.

3. The augmented reality device (100) according to claim 1 or 2, wherein: The at least one processor (130) is further configured to execute the at least one instruction to: deriving a plane equation from each of the planes including the identified walls and floors, By extending the wall and the floor based on the derived plane equation, a 3D model shape of a virtual wall and a virtual floor is obtained, recognizing objects located on the wall and the floor in the real world space from the spatial image, repairing the areas of the wall and floor hidden by the recognized object by using information of the spatial image, and A 3D model of the real world space is generated by applying the texture of the patched image to the 3D model shapes of the virtual walls and virtual floor.

4. The augmented reality device (100) according to claim 3, wherein: The at least one processor (130) is further configured to execute the at least one instruction to: Extract the intersection lines where the extended walls and floors intersect each other, distinguishing the planes of the wall and the floor based on the extracted intersection lines, obtaining the vertex coordinates of each of the distinguished planes, and The 3D model shapes of the virtual wall and the virtual floor are generated based on the obtained vertex coordinates.

5. The augmented reality device (100) according to claim 3, wherein: The at least one processor (130) is further configured to execute the at least one instruction to: recognizing the object based on at least one of depth information and color information of the real world space obtained from the spatial image, and The identified walls and floors are distinguished from the recognized objects.

6. The augmented reality device (100) according to any one of claims 1 to 5, wherein: The at least one processor (130) is further configured to execute the at least one instruction to: determining whether a 3D model of the selected object is pre-stored in the memory (140), and In case it is determined that the 3D model of the selected object is stored, arranging the 3D model by adjusting the orientation of the pre-stored 3D model to overlap with the 2D segmentation contour on the aerial image, obtaining the 3D position coordinate values ​​of the object from the 3D model of the arrangement, and A 3D segmentation is performed, the 3D segmentation segmenting the object from the aerial image based on the obtained 3D position coordinate values.

7. The augmented reality device (100) according to any one of claims 1 to 6, wherein: The at least one processor (130) is further configured to execute the at least one instruction to: determining whether a 3D model of the selected object is stored in the memory (140), In the case where it is determined that the 3D model of the selected object is not stored, obtaining 3D position coordinate values ​​of edges, feature points and pixels of the object recognized from the aerial image, performing 3D vertex modeling based on at least one of the obtained 3D position coordinate values ​​of the edge, feature point, and pixel of the object, and 3D segmentation is performed which segments the object from the aerial image via 3D vertex modeling.

8. A method performed by an augmented reality device (100) for providing an augmented reality service for controlling an object in a real world space, the method comprising: recognizing a plane including a wall and a floor from a space image obtained by photographing a real world space using a camera (110) (S410); generating a three-dimensional (3D) model of a real world space by extending the wall and floor along the identified plane and performing 3D inpainting on a region of the extended wall and floor hidden by the object by using the space image (S420); performing two-dimensional (2D) segmentation that segments an object selected by a user input from the spatial image ( S430 ); as well as A 3D segmentation is performed, the 3D segmentation segmenting the object on the spatial image from the real world space based on the 3D model or 3D position information of the object (S440).

9. The method according to claim 8, further comprising: Arranging the generated 3D model at the position of the wall and the floor in the real world space (S1310); as well as A region in which the object is segmented via 3D segmentation is rendered by using the 3D model ( S1320 ).

10. The method according to claim 8 or 9, wherein: The generation of the 3D model of the real world space (S420) includes: deriving a plane equation from each of the planes including the identified walls and floors (S510); Obtaining 3D model shapes of virtual walls and virtual floors by extending the walls and floors based on the derived plane equations (S520); recognizing objects located on the wall and the floor in the real world space from the space image (S530); repairing the area of ​​the wall and the floor hidden by the recognized object by using information of the spatial image (S540); and A 3D model of the real world space is generated by applying the texture of the patched image to the 3D model shapes of the virtual wall and the virtual floor ( S550 ).

11. The method according to claim 10, wherein: Generating the 3D model shapes of the virtual wall and the virtual floor (S520) includes: extracting intersection lines where the extended walls and floors intersect each other; distinguishing the planes of the wall and the floor based on the extracted intersection lines; obtaining the vertex coordinates of each of the distinguished planes, and The 3D model shapes of the virtual wall and the virtual floor are generated based on the obtained vertex coordinates.

12. The method according to claim 10, wherein: Recognizing objects located on the wall and the floor in the real world space (S530) includes: recognizing the object based on at least one of depth information and color information of a real-world space obtained from the spatial image; and The identified walls and floors are distinguished from the recognized objects.

13. The method according to any one of claims 8 to 12, wherein: Performing 3D segmentation (S440) includes: Determine whether the 3D model of the selected object is pre-stored (S1010); In a case where it is determined that the 3D model of the selected object is stored, arranging the 3D model by adjusting a direction of the pre-stored 3D model to overlap with the 2D segmentation contour on the aerial image (S1020); obtaining a 3D position coordinate value of the object from the arranged 3D model (S1030); and A 3D segmentation is performed, the 3D segmentation segmenting the object from the spatial image based on the obtained 3D position coordinate value (S1040).

14. The method according to any one of claims 8 to 13, wherein: Performing 3D segmentation involves: Determine whether the 3D model of the selected object is pre-stored (S1010); In a case where it is determined that the 3D model of the selected object is not stored, obtaining 3D position coordinate values ​​of edges, feature points, and pixels of the object recognized from the aerial image (S1050); performing 3D vertex modeling based on at least one of the obtained 3D position coordinate values ​​of the edge, feature point, and pixel of the object (S1060); and 3D segmentation is performed, which segments the object from the aerial image via 3D vertex modeling (S1070).

15. A computer program product comprising a computer-readable storage medium, in, The computer-readable storage medium includes instructions executed by an augmented reality device (100) to: recognizing a plane including a wall and a floor from a space image obtained by photographing a real-world space using a camera (110); generating a three-dimensional (3D) model of a real world space by extending the wall and floor along the identified plane and performing 3D inpainting on areas of the extended wall and floor that are hidden by the object by using the spatial image; performing two-dimensional (2D) segmentation that segments an object selected by a user input from the spatial image; as well as 3D segmentation is performed, the 3D segmentation segmenting the object on the spatial image from the real world space based on the 3D model or 3D position information of the object.

Images