Augmented reality apparatus for acquiring depth information and method of operating same

Abstract

Provided are an augmented reality device and a method of operating the same, the device and method correcting a depth value based on a direction of gravity measured by an IMU sensor in order to acquire a depth map with high precision without additional hardware modules. An augmented reality device according to one embodiment of the present disclosure may acquire a depth map from an image acquired using a camera, acquire a normal vector of at least one pixel from the depth map, correct a direction of the normal vector of the at least one pixel based on a direction of gravity measured by an inertial measurement unit (IMU) sensor, and determine a direction of gravity based on the corrected direction. And corrects depth values in the depth map based on the direction of the modified normal vector.

Description

Technical Field

[0001] The present disclosure relates to an augmented reality device for acquiring depth information, and more particularly, to an augmented reality device for modifying a depth value of each pixel in a depth map based on a gravity direction and an operating method thereof. Background Art

[0002] Augmented reality is a technology that can overlay virtual images on the physical environment space of the real world or real-world objects and can display virtual images together with real-world objects. Augmented reality devices (e.g., smart glasses) using augmented reality technology are effectively used in daily life, for example, for information retrieval, direction guidance, and camera shooting. As an example of an AR device, smart glasses can be worn as a fashion item and can also be used for outdoor activities.

[0003] Recently, in order to give users an immersive feeling, devices including depth sensors have been widely used, which acquire depth information indicating a sense of space of objects included in a real three-dimensional space. Examples of depth information acquisition technology using existing depth sensors according to the related art may include, for example, a structured light (SL) method, a stereoscopic vision method, a time-of-flight (ToF) method, and the like. Among the depth information acquisition technologies, the stereoscopic vision method and the ToF method are camera-based depth estimation methods, and the accuracy of the depth value decreases as it moves away from the camera. In the structured light method or the ToF method having a relatively high-precision depth value about the distance, an additional hardware module (such as an illuminator) is required, and the hardware module requires power consumption and additional costs.

[0004] In addition, most AR applications run by AR devices require constant depth information, so power consumption increases. Due to the characteristics of AR devices (i.e., AR devices are portable devices with compact size specifications), heat generation and power consumption greatly affect the usable time of the device. Summary of the invention

[0005] An augmented reality device for modifying a depth value based on a gravity direction is provided.

[0006] According to one aspect of the present disclosure, an augmented reality device is provided, comprising: a camera; an inertial measurement unit (IMU) sensor configured to measure a gravity direction; a memory configured to store at least one instruction; and at least one processor configured to execute the at least one instruction to: obtain a depth map from an image obtained using the camera, obtain a normal vector of at least one pixel included in the depth map, modify a direction of the normal vector of at least one pixel based on the gravity direction measured by the IMU sensor, and adjust a depth value of at least one pixel based on the modified direction of the normal vector.

[0007] At least one processor may also be configured to obtain a normal vector by converting at least one pixel into a three-dimensional (3D) coordinate value based on a normal vector of at least one pixel included in the depth map and a depth value of at least one pixel, and obtaining a vector cross product of the 3D coordinate values ​​of multiple adjacent pixels adjacent to the at least one pixel in any one direction of an upper direction, a lower direction, a left direction, and / or a right direction.

[0008] The camera may include a left-eye lens configured to obtain a left-eye image and a right-eye lens configured to obtain a right-eye image, wherein at least one processor may also be configured to: apply the left-eye image and the right-eye image as input to an artificial intelligence (AI) model, and obtain a depth map by obtaining disparity based on similarity of intensity values ​​between pixels in the left-eye image and pixels in the right-eye image using the AI ​​model.

[0009] At least one processor may also be configured to obtain a loss of a depth map obtained by the AI ​​model based on a depth value of a pixel on a plane defined by a modified normal vector and depth values ​​of multiple adjacent pixels adjacent to the pixel, and adjust the depth value of at least one pixel by performing training that can apply the obtained loss to the AI ​​model.

[0010] At least one processor may also be configured to define a plane to include a pixel having a modified normal vector and a plurality of neighboring pixels adjacent to the pixel, obtain depth values ​​of the plurality of neighboring pixels based on a plurality of points where the defined plane intersects a ray vector of the camera, obtain a difference between the depth value of the pixel and each of the depth values ​​of the plurality of neighboring pixels, and obtain a loss by performing a weighted sum of the difference values ​​obtained for each of the plurality of neighboring pixels in which a weight is applied.

[0011] The weight may include a first weight determined based on a distance between each of a plurality of neighboring pixels in the depth map and the camera, and a second weight determined based on a difference in intensity values ​​between a pixel of the depth map and each of the plurality of neighboring pixels.

[0012] At least one processor may also be configured to obtain an adjusted depth map by performing inference providing the left-eye image and the right-eye image to the trained AI model.

[0013] The camera may include a left-eye lens configured to obtain a left-eye image and a right-eye lens configured to obtain a right-eye image, wherein the at least one processor may also be configured to: modify the direction of the normal vector in the left-eye image and the right-eye image according to the direction of gravity or a direction perpendicular to the direction of gravity, perform a plane assumption along the direction of the modified normal vector or a direction perpendicular to the direction of the modified normal vector, obtain a depth value of at least one pixel by performing a plane scan along a plane defined by the plane assumption, and modify the depth value of at least one pixel by using the obtained depth value.

[0014] The camera may include a time-of-flight (ToF) camera, wherein the at least one processor may be further configured to obtain the depth map by using the ToF camera.

[0015] At least one processor may also be configured to define a plane for each pixel of at least one pixel based on the modified normal vector, identify a plane region of the plane defined in the depth map based on a region segmented according to color information of the RGB image, and adjust the depth value based on depth values ​​of neighboring pixels in the identified plane region.

[0016] According to one aspect of the present disclosure, a method for adjusting a depth value by an augmented reality device is provided, including: obtaining a depth map from an image obtained using a camera; obtaining a normal vector of at least one pixel included in the depth map; modifying a direction of the normal vector of at least one pixel based on a gravity direction measured by an Inertial Measurement Unit (IMU) sensor; and adjusting the depth value of at least one pixel based on the modified direction of the normal vector.

[0017] Obtaining a depth map may include: providing a left-eye image obtained by using a left-eye lens and a right-eye image obtained by using a right-eye lens to an artificial intelligence (AI) model; and obtaining a depth map by obtaining disparity according to similarity of intensity values ​​between pixels of the left-eye image and pixels of the right-eye image using the AI ​​model.

[0018] Adjusting the depth value of each of at least one pixel may include: obtaining a loss of a depth map obtained by an AI model based on the depth value of the pixel on a plane defined by a modified normal vector and depth values ​​of multiple neighboring pixels adjacent to the pixel; and adjusting the depth value of at least one pixel by performing training in which the obtained loss is applied to the AI ​​model.

[0019] Identifying a loss may include: defining a plane to include a pixel having a modified normal vector and a plurality of neighboring pixels adjacent to the pixel; obtaining depth values ​​of the plurality of neighboring pixels based on a plurality of points at which the defined plane intersects a ray vector of a camera; obtaining a difference between the depth value of the pixel and each of the depth values ​​of the plurality of neighboring pixels; and obtaining the loss by performing a weighted sum of the difference values ​​obtained for each of the plurality of neighboring pixels by applying a weight to the weighted sum.

[0020] The weight may include a first weight determined based on a distance between each of a plurality of neighboring pixels in the depth map and the camera, and a second weight determined based on a difference in intensity values ​​between a pixel of the depth map and each of the plurality of neighboring pixels.

[0021] The method may also include obtaining an adjusted depth map by performing inference providing the left-eye image and the right-eye image to the trained AI model.

[0022] Modifying the direction of the normal vector of each pixel may include: modifying the direction of the normal vector of a left eye image obtained by using a left eye lens and a right eye image obtained by using a right eye lens according to the direction of gravity or a direction perpendicular to the direction of gravity, wherein adjusting the depth value of at least one pixel may include: performing a plane assumption along the direction of the modified normal vector and a direction perpendicular to the direction of the modified normal vector; obtaining a depth value by performing a plane scan along a plane defined by the plane assumption; and adjusting the depth value of at least one pixel by using the obtained depth value.

[0023] Obtaining the depth map may include obtaining the depth map by using a time-of-flight camera.

[0024] Modifying the depth value of at least one pixel may include: defining a plane for each pixel of the at least one pixel based on a modified normal vector; identifying a plane area of ​​the plane defined in the depth map based on an area segmented according to color information of an RGB image; and adjusting the depth value of at least one pixel based on depth values ​​of neighboring pixels in the identified plane area.

[0025] According to one aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, including instructions executed by an augmented reality device to perform a method including the following steps: obtaining a depth map from an image obtained by a camera; obtaining a normal vector of at least one pixel included in the depth map; modifying the direction of the normal vector of at least one pixel based on the gravity direction measured by an inertial measurement unit sensor; and adjusting the depth value of at least one pixel based on the modified direction of the normal vector. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following description in conjunction with the accompanying drawings, in which: Figure 1a is a conceptual diagram for illustrating a method of adjusting a depth value performed by an augmented reality device according to an embodiment of the present disclosure; Figure 1b is a conceptual diagram illustrating a method of adjusting a depth value performed by an augmented reality device according to a gravity direction measured by an IMU sensor according to an embodiment of the present disclosure; Figure 2 is a block diagram showing an augmented reality device according to an embodiment of the present disclosure; Figure 3 is a flowchart illustrating a method of operating an augmented reality device according to an embodiment of the present disclosure; Figure 4 is a flowchart illustrating a method for obtaining a normal vector of each pixel performed by an augmented reality device according to an embodiment of the present disclosure; Figure 5 is a diagram illustrating an operation of obtaining a normal vector of each pixel from a depth map by an augmented reality device according to an embodiment of the present disclosure; Figure 6a is a diagram illustrating an operation of modifying a direction of a normal vector according to a gravity direction, performed by an augmented reality device according to an embodiment of the present disclosure; Figure 6b is a diagram illustrating an operation of modifying the direction of a normal vector in a direction perpendicular to the direction of gravity, performed by an augmented reality device according to an embodiment of the present disclosure; Figure 7 is a diagram illustrating a training operation of obtaining a depth map by using an artificial intelligence model in an augmented reality device according to an embodiment of the present disclosure; Figure 8 is a flowchart illustrating a method of adjusting a depth value of each pixel in a depth map performed by an augmented reality device according to an embodiment of the present disclosure; Fig. 9 is a flowchart illustrating a method of calculating loss performed by an augmented reality device according to an embodiment of the present disclosure; Fig.10 is a diagram illustrating an operation of calculating a loss performed by an augmented reality device according to an embodiment of the present disclosure; Fig.11 is a diagram illustrating an operation of obtaining a depth map by using an artificial intelligence model, performed by an augmented reality device according to an embodiment of the present disclosure; Fig.12 is a flowchart illustrating a method of adjusting a depth value of each pixel in a depth map performed by an augmented reality device according to an embodiment of the present disclosure; Fig.13is a diagram illustrating an operation of adjusting a depth value of each pixel in a depth map by a plane scanning method performed by an augmented reality device according to an embodiment of the present disclosure; Fig.14 is a flowchart illustrating a method of adjusting a depth value of each pixel in a depth map performed by an augmented reality device according to an embodiment of the present disclosure; Fig.15 is a diagram illustrating an operation of adjusting a depth value of each pixel in a depth map obtained by a time-of-flight (ToF) method, performed by an augmented reality device according to an embodiment of the present disclosure; Fig.16 2 is a diagram illustrating a space model reconstructed from a depth map obtained by a general method and a space model reconstructed from a depth map obtained by an augmented reality device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] Throughout the disclosure, the expression "at least one of a, b, or c" means only a, only b, only c, a and b, a and c, b and c, or all of a, b, and c.

[0028] All terms used herein, including descriptive terms or technical terms, should be interpreted as having obvious meanings to those of ordinary skill in the art. However, these terms may have different meanings according to the intentions of those of ordinary skill in the art, precedents, or the emergence of new technologies. In addition, some terms may be selected arbitrarily by the applicant. In this case, the meaning of the selected terms will be described in the detailed description. Therefore, the terms used herein should be defined together with the description of the entire specification based on the meaning of the terms.

[0029] As used herein, unless the context clearly indicates otherwise, the singular form also includes the plural form. Unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as commonly understood by ordinary technicians in the field of this disclosure.

[0030] It will also be understood that when the term "comprising" is used in this specification, it specifies the presence of the components, but does not exclude the presence or addition of one or more components. In addition, the terms "... unit", "... module" used herein specify a unit for processing at least one function or operation, and can be implemented in hardware or software or a combination of hardware and software.

[0031] Depending on the specific circumstances, the expression "configured to" used in the present disclosure may be replaced with, for example, "suitable for", "capable of doing", "designed to", "adapted to", "manufactured to", or "capable of". The term "configured to" may not necessarily mean "specially designed to" in hardware. Alternatively, in some cases, the expression "the system is configured to" may mean that the system is "capable of" together with other devices or components. For example, the phrase "a processor adapted to (or configured to) perform A, B, and C" may mean a dedicated processor (e.g., an embedded processor) that is only used to perform the corresponding operations or a general-purpose processor (e.g., a central processing unit (CPU) or an application processor (AP)) that can perform the corresponding operations by running one or more software programs stored in a storage device.

[0032] Furthermore, in this specification, when an element is referred to as being “connected to” another element, it should be understood that, unless otherwise specified, the element may be directly connected to the other element, but may be connected to or connected via another intervening element.

[0033] Hereinafter, one or more embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can perform the present disclosure. However, the present disclosure can be implemented in various ways and is not limited to the one or more embodiments described herein.

[0034] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0035] Figure 1a is a conceptual diagram for illustrating a method of adjusting a depth value performed by an augmented reality device according to an embodiment of the present disclosure.

[0036] Reference Figure 1a , the augmented reality device may estimate the depth value 20 of the actual bottom surface 10 by using the image obtained by using the camera 110. The estimated depth value 20 may be different from the depth value of the actual bottom surface 10 according to the positional relationship, such as the physical distance or height difference between the actual bottom surface 10 and the camera 110. For example, when the camera 110 includes a left-eye lens and a right-eye lens, and the augmented reality device estimates the depth value 20 by a stereoscopic vision method using a left-eye image obtained from the left-eye lens and a right-eye image obtained from the right-eye lens, a parallax error, i.e., a distance between corresponding points of the left-eye image and the right-eye image, may increase as the distance from the camera 110 increases. For example, when the augmented reality estimates the depth value 20 by a time-of-flight (ToF) method, the accuracy of the estimated depth value 20 decreases as the distance from the camera 110 increases.

[0037] exist Figure 1aIn the embodiment, the estimated depth value 20 is similar to the depth value of the actual bottom surface 10 in the area close to the camera 110, but as the distance from the camera 110 increases, the depth value may be obtained in an upwardly inclined shape, which is different from the depth value of the actual bottom surface 10.

[0038] The augmented reality device may obtain information about the gravity direction G based on a measurement value measured by an inertial measurement unit (IMU) sensor 120. In an embodiment of the present disclosure, the IMU sensor 120 includes a gyroscope sensor (gyroscope), and the augmented reality device may obtain information about the gravity direction G by using the gyroscope included in the IMU sensor 120. The augmented reality device may obtain an adjusted depth value 30 by adjusting the estimated depth value 20 based on the gravity direction G. Figure 1a In the illustrated embodiment, the adjusted depth value 30 may have the same flat slope as the actual bottom surface 10 and may have the same depth value as the actual bottom surface 10. Figure 1b A detailed method in which the augmented reality device adjusts the depth value based on the information about the gravity direction G obtained through the IMU sensor 120 is described.

[0039] Figure 1b is a conceptual diagram illustrating a method of adjusting a depth value according to a gravity direction measured by an IMU sensor by an augmented reality device according to the present disclosure.

[0040] Reference Figure 1b The augmented reality device may obtain a depth value of a pixel based on an image obtained by the camera 110. 1 to p n In an embodiment of the present disclosure, the camera 110 includes a left eye lens and a right eye lens, and the augmented reality device may obtain a depth image by a stereoscopic vision method using a left eye image obtained from a left eye image and a right eye image obtained from a right eye lens. In an embodiment of the present disclosure, the camera 110 includes a ToF camera, and the augmented reality device may obtain a depth map by using the ToF camera. However, the embodiments of the present disclosure are not limited thereto, and in an embodiment of the present disclosure, the augmented reality device may obtain a depth map by a structured light method.

[0041] The depth value of the pixel included in the depth map may be different from the depth value of the actual bottom surface 10 based on the distance from the camera 110. Figure 1b In the embodiment shown, in the plurality of pixels p 1 to p n The first pixel p relatively close to the camera 110 1 The depth value of the second pixel p 2The depth value of the pixel p is the same as or similar to the depth value of the actual bottom surface 10, but is far away from the camera 110. n-1 The depth value and p n The depth value of φ may be different from the depth value of the actual bottom surface 10 .

[0042] The augmented reality device obtains the normal vector N of the pixel 1 To N n (Operation ②). In an embodiment of the present disclosure, the augmented reality device may be based on a plurality of pixels p included in the depth map. 1 to p n direction vector and depth value, multiple pixels p 1 to p n The augmented reality device can obtain the plurality of pixels p by calculating the vector cross product of the 3D coordinate values ​​of the adjacent pixels arranged at positions adjacent to any one pixel in the up-down and left-right directions. 1 to p n The normal vector N 1 To N n .

[0043] The augmented reality device modifies the normal vector N based on the direction of gravity 1 To N n (Operation ③). The augmented reality device includes an IMU sensor 120 (see Figure 1a ), and may obtain information about the gravity direction G based on a measurement value measured by using a gyroscope of the IMU sensor 120. The augmented reality device may modify the normal vector N of the pixel according to the gravity direction G or a direction perpendicular to the gravity direction G. 1 To N n As a result of the modification, the augmented reality device can obtain a modified normal vector N 1 ' to N n '.

[0044] The augmented reality device is based on the modified normal vector N 1 ' to N n ' to adjust the depth value of each pixel (operation ④). The augmented reality device adjusts the depth value of each pixel according to the modified normal vector N 1 ' to N n ' to define the plane of each pixel, and the depth value of the pixel can be adjusted based on the plane of each pixel. In an embodiment of the present disclosure, the augmented reality device obtains a depth map by using an artificial intelligence (AI) model, based on the modified normal vector N 1 ' to N n'The depth value of the pixel on the plane defined by the algorithm and the depth value of the adjacent pixel are used to calculate the loss, and the depth value of each pixel is adjusted by performing training, wherein the calculated loss is applied to the AI ​​model. In an embodiment of the present disclosure, the augmented reality device can be based on the modified normal vector N 1 ' to N n ' direction to perform plane scanning on the plane of each pixel defined to adjust the depth value of each pixel. In the embodiment of the present disclosure, the depth value of each pixel can be adjusted based on the color information of the RGB image from the modified normal vector N 1 ' to N n ' A plane region is identified in the defined plane, and the depth value of each pixel can be adjusted based on the depth values ​​of neighboring pixels in the identified plane region.

[0045] The augmented reality device may obtain a plurality of pixels p having adjusted depth values. 1 'To p n '. Multiple pixels p 1 'To p n Each of ' may have a depth value equal to or similar to the depth of the actual bottom surface 10.

[0046] An image-based depth value acquisition method (e.g., a structured light method, a stereoscopic vision method, or a ToF method) using an image acquired by the camera 110 has a problem in that the accuracy of the depth value decreases as the distance from the camera 110 increases. In the case where the structured light method or the ToF method has a relatively high-precision depth value with respect to the distance, an additional hardware module (such as an illuminator) is required, and the hardware module causes power consumption and additional costs. In addition, because most augmented reality applications always require depth information, there is a problem of increasing the power consumption of the augmented reality device. Due to the characteristics of the augmented reality device being a portable device with a compact size specification, heat generation and power consumption can greatly affect the usable time of the device.

[0047] The present disclosure provides an augmented reality device and an operating method thereof, wherein the augmented reality device can be used to Figure 1a ) to obtain the gravity direction G information to obtain a depth value with high precision without unnecessary power consumption of additional hardware modules (e.g., light emitting unit modules).

[0048] exist Figure 1a and Figure 1b In the illustrated embodiment, the augmented reality device modifies the normal vector N of each pixel based on the gravity direction G information obtained by using the IMU sensor 120. 1 To N n direction, and based on the modified normal vector N 1 ' to N n' to adjust the depth value of each pixel so as to obtain a depth map with high precision and reduce power consumption. In addition, an augmented reality device generally includes an IMU sensor 120 as a basic element, and the augmented reality device according to an embodiment of the present disclosure can achieve lower power consumption while maintaining a compact size specification, thereby providing a technical effect of increasing portability and device usage time.

[0049] Figure 2 is a block diagram illustrating an augmented reality device 100 according to an embodiment of the present disclosure.

[0050] The augmented reality device 100 may be an augmented reality glasses formed in the form of glasses worn on the user's face. The augmented reality device 100 may provide virtual image content displayed on the waveguide and real world objects within the field of view (FOV) by running applications. The augmented reality device 100 may provide the user with virtual image content displayed on each application by running, for example, movie applications, music applications, photo applications, gallery applications, web browser applications, e-book applications, game applications, augmented reality applications, social network service (SNS) applications, communication applications, object detection applications, etc.

[0051] However, the embodiments of the present disclosure are not limited thereto, and the augmented reality device 100 may be implemented as a head mounted display device (HMD) worn on the user's head, an augmented reality helmet, etc. However, the augmented reality device 100 according to the present disclosure is not limited to the above examples. In the embodiments of the present disclosure, the augmented reality device 100 may be implemented in the form of various devices, such as a mobile device, a smart phone, a laptop computer, a tablet computer, an e-book terminal, a digital broadcast terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation device, an MP3 player, an Internet Protocol television (IPTV), a digital television (DTV), a wearable device, etc.

[0052] Reference Figure 2 , the augmented reality device 100 may include a camera 100, an IMU sensor 120, a processor 130, and a memory 140. The camera 100, the IMU sensor 120, the processor 130, and the memory 140 may be electrically or physically connected to each other. The elements included in the augmented reality device 100 are not limited to Figure 2 In an embodiment of the present disclosure, the augmented reality device 100 may further include an eye tracking sensor and / or a display engine.

[0053] The camera 110 obtains an image about an object by capturing an image of the object in a real word space. In an embodiment of the present disclosure, the camera 110 may include a lens module, an image sensor, and an image processing module. The camera 110 may obtain a still image or a video acquired by an image sensor (e.g., a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD)). The image processing module processes the still image or the video acquired by the image sensor to extract necessary information, and transmits the extracted information to the processor 130.

[0054] In an embodiment of the present disclosure, the camera 110 may include a stereo camera including a left eye lens and a right eye lens, and a 3D image of an object is obtained by using the two lenses. However, the embodiment of the present disclosure is not limited thereto, and the camera 110 may include a single lens or three or more lenses.

[0055] In an embodiment of the present disclosure, the camera 110 may include a ToF camera that irradiates light to an object, detects light reflected from the object, and obtains a depth value of the object based on a flight time that is a difference between a time point at which the reflected light is detected and a time point at which the light is irradiated. When implemented as a ToF camera, the camera 110 may obtain an RGB image 1500 (see FIG. 1 ). Fig.15 ) and depth map image 1510 (see Fig.15 ).

[0056] The IMU sensor 120 is a sensor configured to measure the speed, direction, angle, and gravity acceleration of the augmented reality device 100 through a combination of an accelerometer, a gyroscope, and a magnetometer. In an embodiment of the present disclosure, the IMU sensor 120 may include a three-axis accelerometer that measures acceleration in the longitudinal, lateral, and height directions, and a three-axis angular velocity meter that measures roll, pitch, and yaw angular velocities. In an embodiment of the present disclosure, the IMU sensor 120 measures the angular velocity of the gyroscope and can sense the direction of gravity based on the measured angular velocity. The IMU sensor 120 can provide information about the direction of gravity to the processor 130.

[0057] The processor 130 may execute one or more instructions of a program stored in the memory 140. The processor 130 may include hardware elements that perform calculations, logic, input / output operations, and signal processing. The processor 130 may include, for example, at least one hardware element of a central processing unit (CPU), a microprocessor, a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), and a field programmable gate array (FPGA).

[0058] In an embodiment of the present disclosure, the processor 130 may include an AI processor that performs AI learning. The AI ​​processor may be manufactured in the form of a hardware chip dedicated to AI, or may be manufactured as part of an existing general-purpose processor (e.g., a CPU or an application processor) or a graphics-specific processor (e.g., a GPU) and loaded onto the augmented reality device 100.

[0059] Figure 2 The processor 130 is shown as one element, but the embodiments of the present disclosure are not limited thereto. In the embodiments of the present disclosure, the processor 130 may include one or more elements.

[0060] The memory 140 may include at least one type of storage medium such as a flash memory type, a hard disk type, a multimedia card micro, a card type memory (e.g., an SD memory or an XD memory, etc.), a random access memory (RAM), a static RAM (SRAM), a read-only memory (ROM), an electrically erasable and programmable ROM (EEPROM), a programmable (PROM), a magnetic memory, a magnetic disk, and an optical disk.

[0061] The memory 140 may store instructions related to the function or operation of the augmented reality device 100 to obtain the depth value information of the object. In an embodiment of the present disclosure, the memory 140 may store at least one of instructions, algorithms, data structures, program codes, or applications that can be read by the processor 130. The instructions, algorithms, data structures, and program codes stored in the memory 140 may be implemented in, for example, a programming language or a scripting language (such as C, C++, Java, assembly, etc.).

[0062] The memory 140 may store instructions, algorithms, data structures, or program codes regarding the depth map acquisition module 142, the normal vector modification module 144, and the depth value adjustment module 146. The “module” included in the memory 140 refers to a unit for processing a function or operation performed by the processor 130, and the module may be implemented with software (such as instructions, algorithms, data structures, or program codes).

[0063] In the following embodiments, the processor 130 may be implemented by executing instructions or program codes stored in the memory 140 .

[0064] The depth map acquisition module 142 includes instructions or program codes related to functions and / or operations of obtaining a depth map about an object based on an image obtained by using the camera 110. In an embodiment of the present disclosure, the depth map acquisition module 142 may be configured to obtain a depth map by using an AI model. The processor 130 may obtain a depth map by running instructions or program codes related to the depth map acquisition module 142. In an embodiment of the present disclosure, the camera 110 may include a stereo camera including a left eye lens and a right eye lens. In this case, the processor 130 may input a left eye image obtained by using the left eye lens and a right eye image obtained by using the right eye lens into the AI ​​model, and may obtain a depth map by using the AI ​​model to calculate the disparity according to the similarity between the pixel intensity values ​​of the left eye image and the pixel intensity values ​​of the right eye image. The AI ​​model may be implemented as a deep neural network model. The deep neural network model may be a known artificial intelligence model, such as a convolutional neural network (CNN) model, a recurrent neural network (RNN) model, a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q network. The deep neural network model used to obtain the depth map may include, but is not limited to, DispNet, for example. In an embodiment of the present disclosure, the processor 130 may obtain a depth map by calculating a disparity between a left-eye image and a right-eye image using a plane scanning method.

[0065] In an embodiment of the present disclosure, the camera 110 may include a ToF camera, and the processor 130 may calculate the flight time (i.e., the time difference between the time point at which the reflected light obtained by using the ToF camera is detected and the time point at which the clock signal is input), and calculate the distance (i.e., the depth value) between the position where the reflected light is detected and the augmented reality device 100 by an operation between the flight time and the speed of light. The processor 130 may obtain information about the position where the irradiated light and the reflected light are detected, and may obtain a depth map by mapping the calculated depth value to the obtained position information.

[0066] The normal vector modification module 144 may include instructions or programs related to functions and / or operations of obtaining a normal vector of each of a plurality of pixels included in the depth map and modifying the normal vector according to the gravity direction obtained by the IMU sensor 120. The processor 130 may obtain a normal vector of at least one pixel and modify the direction of the obtained normal vector by running instructions or program codes related to the normal vector modification module 144. In an embodiment of the present disclosure, the processor 130 may obtain a normal vector of each pixel by converting each of the plurality of pixels into a 3D coordinate value based on the direction vectors of the plurality of pixels included in the depth map and the depth values ​​of the plurality of pixels, and calculating a vector cross product of the 3D coordinate values ​​of pixels adjacent to each of the converted pixels in the up, down, left, and right directions. The processor 130 may obtain information about the direction of gravity based on the measurement value measured by the gyroscope in the IMU sensor 120, and modify the direction of the normal vector of each pixel based on the direction of gravity. In an embodiment of the present disclosure, the processor 130 may modify the direction of the normal vector of each pixel according to the direction perpendicular to the direction of gravity and the direction of gravity. For example, for a pixel having a normal vector close to a vertical direction, the processor 130 may modify the direction of the normal vector to a direction parallel to the direction of gravity. For example, for a pixel having a normal vector almost parallel to the ground (e.g., a pixel indicating an object (such as a wall, a pillar, etc.)), the processor 130 may modify the direction of the normal vector to a direction perpendicular to the direction of gravity.

[0067] Will refer to Figure 4 , Figure 5 , Figure 6a and Figure 6b A detailed example in which the processor 130 obtains a normal vector of each pixel and modifies the direction of the obtained normal vector is described in detail.

[0068] The depth value adjustment module 146 includes instructions or program codes related to the function and / or operation of adjusting the depth value of at least one pixel in the depth map based on the direction of the modified normal vector. The processor 130 can adjust the depth value of at least one pixel by running the instructions or program codes related to the depth value adjustment module 146. In an embodiment of the present disclosure, the processor 130 can adjust the depth value of at least one pixel in the depth map by calculating the loss of the depth map output from the training process of the AI ​​model and applying the calculated loss to the training of the AI ​​model. The processor 130 can calculate the loss based on the depth value of the pixel on the plane defined by the modified normal vector and the depth value of the adjacent pixel in the position. In an embodiment of the present disclosure, the processor 130 can obtain the depth value of the adjacent pixel based on the multiple points where the defined plane and the ray vector of the camera 110 intersect, calculate the difference between the depth value of the pixel and the depth value of the adjacent pixel, and calculate the loss by weighted sum, which applies the weight to the difference calculated for each of the adjacent pixels.

[0069] In an embodiment of the present disclosure, the weights applied to the weighted sum calculation may include a first weight determined based on the distance between the position of the adjacent pixel in the depth map and the position of the camera 110, and a second weight determined based on the difference between the intensity value of the pixel in the depth map and the intensity value of the adjacent pixel. Figures 7 to 10 A detailed example in which the processor 130 calculates a loss according to a positional relationship between a plane defined by a modified normal vector and the camera 110, and adjusts a depth value of at least one pixel by training an AI model using the calculated loss is described in detail.

[0070] In an embodiment of the present disclosure, the processor 130 may obtain a depth value of at least one pixel by assuming a plane (plane assumption) for each pixel along a direction of the modified normal vector or a direction perpendicular to the direction of the modified normal vector and performing a plane scan according to the assumed plane. The processor 130 may adjust the depth value of at least one pixel in the depth map by using the obtained depth value. Fig.12 and Fig.13 A detailed example of the processor 130 adjusting the depth value of at least one pixel by performing a plane scan along a plane defined based on the modified normal vector is described in detail.

[0071] In an embodiment of the present disclosure, the processor 130 may obtain an RGB image and a depth map image by using a ToF camera. The processor 130 may define the plane of each pixel in the depth map image based on the modified normal vector, and identify the plane area of ​​the depth map image based on the area segmented according to the color information of the RGB image. The processor 130 may adjust the depth value of the pixel to be adjusted in the identified plane area based on the depth value of the adjacent pixels in the same plane area. Here, the "pixel to be adjusted" refers to the pixel whose depth value needs to be adjusted, as well as the pixel whose depth value difference with the pixel that has no depth value in the plane area (the pixel that has not obtained the depth value) or the adjacent pixel in the plane area exceeds a preset critical value. This will be referred to later. Fig.14 and Fig.15 A detailed example of describing in detail that the processor 130 recognizes a plane area from a depth map image obtained using a ToF camera according to an RGB image, and adjusts a depth value of at least one pixel in the recognized plane area.

[0072] Figure 3 is a flowchart illustrating a method of operating the augmented reality device 100 according to an embodiment of the present disclosure.

[0073] In operation S310, the augmented reality device 100 obtains a depth map from an image obtained by a camera. In an embodiment of the present disclosure, the camera may be a stereo camera including a left-eye image and a right-eye image, and the augmented reality device 100 may obtain a depth map by inputting a left-eye image obtained by a left-eye camera and a right-eye image obtained by a right-eye camera into an AI model, and calculating a disparity based on a similarity between a pixel intensity value of the left-eye image and a pixel intensity value of the right-eye image using the AI ​​model. The AI ​​model may be implemented as a deep neural network model. The deep neural network model for obtaining a depth map may include, but is not limited to, for example, DispNet.

[0074] In an embodiment of the present disclosure, the augmented reality device 100 may obtain a depth map by calculating the disparity between the left-eye image and the right-eye image using a plane scanning method.

[0075] In an embodiment of the present disclosure, the camera includes a ToF camera, and the augmented reality device 100 may obtain a depth map based on the flight time of light obtained by using the ToF camera.

[0076] In operation S320, the augmented reality device 100 obtains a normal vector of at least one pixel included in the depth map. In an embodiment of the present disclosure, the augmented reality device 100 may convert a plurality of pixels into three-dimensional coordinate values ​​based on a direction vector and a depth value of each of the plurality of pixels included in the depth map. The augmented reality device 100 may obtain a normal vector of each pixel by using the converted 3D coordinate values.

[0077] In operation S330, the augmented reality device 100 modifies the direction of the normal vector of at least one pixel based on the gravity direction measured by the IMU sensor. When the pixel has a normal vector close to the vertical direction, the augmented reality device 100 may modify the direction of the normal vector to be parallel to the gravity direction. When the pixel has a normal vector close to the horizontal direction relative to the ground, the augmented reality device 100 may modify the direction of the normal vector to be perpendicular to the gravity direction.

[0078] In operation S340, the augmented reality device 100 adjusts the depth value of at least one pixel in the depth map based on the direction of the modified normal vector. When the depth map is obtained by using the AI ​​model in operation S310, the augmented reality device 100 may adjust the depth value of at least one pixel included in the depth map by calculating the loss of the depth map obtained during the training process of the AI ​​model and performing the application of the calculated loss to the training of the AI ​​model. The augmented reality device 100 may calculate the loss based on the depth value of the pixel on the plane defined by the modified normal vector and the depth value of the adjacent neighboring pixels at adjacent positions. In an embodiment of the present disclosure, the augmented reality device 100 may obtain the depth value of the adjacent pixel based on a plurality of points where the defined plane and the ray vector of the camera intersect, calculate the difference between the depth value of the pixel and the depth value of the adjacent pixel, and calculate the loss by a weighted sum, which applies a weight to the difference calculated based on each of the adjacent pixels. In an embodiment of the present disclosure, the augmented reality device 100 can obtain the loss by using a weighted sum of a first weight determined based on the distance between the position of an adjacent pixel in a depth map and the position of a camera and a second weight determined based on the difference between the intensity value of a pixel in the depth map and the intensity value of an adjacent pixel.

[0079] In an embodiment of the present disclosure, the augmented reality device 100 may perform a plane hypothesis in which a plane of each pixel is defined according to a direction of a modified normal vector or a direction perpendicular to the direction of the modified normal vector. The augmented reality device 100 may obtain a depth value of at least one pixel by performing a plane scan along a plane defined by the plane hypothesis. The augmented reality device 100 may adjust a depth value of at least one pixel included in a depth map by using the obtained depth value.

[0080] When the depth map is obtained by using the ToF camera in operation S310, the augmented reality device 100 may obtain an RGB image by using the ToF camera. In an embodiment of the present disclosure, the augmented reality device 100 may define a plane of each pixel in the depth map based on the modified normal vector, and may identify a plane area in the depth map based on an area segmented according to color information of the RGB image. The augmented reality device 100 may adjust the depth value of the pixel to be modified in the identified plane area based on the depth values ​​of adjacent pixels in the same plane.

[0081] Figure 4 is a flowchart illustrating a method of obtaining a normal vector of each pixel performed by the augmented reality device 100 according to an embodiment of the present disclosure.

[0082] exist Figure 4 Operation S410 and operation S420 shown in FIG. Figure 3 The details of operation S320 are shown in FIG. Figure 3 After the operation S310 is performed Figure 4 In the operation S410 shown in FIG. Figure 4 After operation S420, the Figure 3 Operation S330.

[0083] Figure 5 is a diagram illustrating an operation of obtaining a normal vector of each pixel from a depth map by the augmented reality device 100 according to an embodiment of the present disclosure.

[0084] In the following, reference will be made to Figure 4 and Figure 5 The operation of the augmented reality device 100 is described.

[0085] Reference Figure 4 In operation S410, the augmented reality device 100 may convert the pixel into a 3D coordinate value based on the direction vector of the pixel and the depth value of the pixel included in the depth map. Figure 5 , the processor 130 of the augmented reality device 100 (see Figure 2 ) The direction vector and depth value D of the first pixel 501 can be obtained from the depth map 500 i,jand the direction vectors and depth values ​​D of the plurality of adjacent pixels 502 to 505 i-1,j , D i+1,j , D i,j-1 , D i,j+1 .exist Figure 5 In the embodiment of FIG. 5 , the first pixel 501 represents a pixel in the i-th column and the j-th row among the plurality of pixels included in the depth map 500 and may have a depth value D i,j The plurality of adjacent pixels 502 to 505 are pixels arranged adjacent to the first pixel 501 in the up-down direction and the left-right direction, and may have depth values ​​D i-1,j , D i+1,j , D i,j-1 and D i,j+1 For example, the second pixel 502 is arranged in the i-1th column and the jth row and has a depth value D i-1,j The third pixel 503 is arranged in the i+1th column and the jth row and may have a depth value D i+1,j The fourth pixel 504 and the fifth pixel 505 may have depth values ​​D i,j-1 and D i,j+1 .

[0086] The processor 130 may calculate the direction vector and the depth value D of the first pixel 501 based on the direction vector and the depth value D of the first pixel 501. i,j and the direction vectors and depth values ​​D of the plurality of adjacent pixels 502 to 505 i-1,j , D i+1,j , D i,j-1 , D i,j+1 , converting the first pixel 501 and the plurality of adjacent pixels 502 to 505 into 3D coordinate values ​​P in the 3D space 510 i,j , P i-1,j , P i+1,j , P i,j-1 , P i,j+1 In an embodiment of the present disclosure, the processor 130 may convert the first pixel 501 and the plurality of adjacent pixels 502 to 505 into 3D coordinate values ​​on the 3D space 510 through the following Equation 1.

[0087] Equation 1

[0088] exist Figure 5 In the embodiment shown, the processor 130 can convert the first pixel 501 into P i,j , and convert multiple adjacent pixels 502 to 505 into P i-1,j , P i+1,j , P i,j-1 , P i,j+1 .

[0089] Reference Figure 4 In operation S420, the augmented reality device 100 may obtain a normal vector for each pixel by calculating a vector cross product of 3D coordinate values ​​of a plurality of adjacent pixels adjacent to the pixel 501 in the up-down direction and the left-right direction. Figure 5 , the processor 130 of the augmented reality device 100 may be based on the 3D coordinate value P of the first pixel i,j To calculate the 3D coordinate values ​​P of multiple adjacent pixels i-1,j , P i+1,j , P i,j-1 , P i,j+1 In an embodiment of the present disclosure, the processor 130 may obtain the normal vector N about the first pixel by performing the following operation of Equation 2: i,j .

[0090] Equation 2

[0091] exist Figure 5 In the embodiment of the present disclosure shown in FIG. 2, the processor 130 can calculate the 3D coordinate values ​​(P i-1,j ,P i+1,j ) and the 3D coordinate values ​​(P i,j-1 , P i,j+1 ) and calculate the vector cross product of these differences to obtain the normal vector N of the first pixel i,j . Normal vector N i,j It may be a vector defining a plane including the first pixel to the fifth pixel.

[0092] Figure 6a FIG. 1 is a diagram showing a method of modifying a normal vector N according to a gravity direction G performed by the augmented reality device 100 according to an embodiment of the present disclosure. i,j Illustration of the operation in the direction of.

[0093] Reference Figure 6a , the augmented reality device 100 may include an IMU sensor 120 , a processor 130 , and a normal vector modification module 144 . Figure 6a Only basic elements are shown to describe the augmented reality device 100 for modifying the normal vector N i,j The elements included in the augmented reality device 100 are not limited to Figure 6a .

[0094] The IMU sensor 120 obtains information about the gravity direction G based on a measurement value measured by using a gyroscope, and may provide the information about the gravity direction G to the processor 130. The processor 130 may modify the normal vector N based on the information about the gravity direction G obtained from the IMU sensor 120 by executing instructions or program codes of the normal vector modification module 144. i,j In an embodiment of the present disclosure, the processor 130 may determine the normal vector N i,j When the direction of the normal vector is close to the vertical direction or the horizontal direction, the processor 130 may calculate the gravity normal vector -N according to the gravity normal vector parallel to the gravity direction G. G To modify the normal vector N i,j direction. Figure 6a In the embodiment shown, since the normal vector N i,j The direction of the normal vector N is close to the vertical direction, so the processor 130 can i,j The direction is changed to be parallel to the gravity normal vector - N G As a result of the modification, the processor 130 may obtain a modified normal vector N' i,j .

[0095] The processor 130 may use the modified normal vector N' i,j The direction is modified by the 3D coordinate value P of the first pixel i,j In the embodiment of the present disclosure, the processor 130 may use the modified normal vector N' i,j and the normal vector N before modification i,j The angular difference between them modifies the orientation of the plane just as much.

[0096] Figure 6b is a diagram showing the process of converting the normal vector N to the normal vector N by the augmented reality device 100 according to an embodiment of the present disclosure. i,j An illustration of an operation in which the direction of is modified to be perpendicular to the gravity direction G.

[0097] Figure 6b Examples and Figure 6a In addition to the normal vector N of the first pixel i,j The direction and modified normal vector N' i,j The directions are the same except for the directions, so redundant descriptions are omitted.

[0098] The processor 130 may modify the normal vector N based on the information about the gravity direction G obtained from the IMU sensor 120 by executing the instructions or program code of the normal vector modification module 144. i,j In an embodiment of the present disclosure, the processor 130 may determine the normal vector N i,j Is the direction close to the vertical direction or the horizontal direction.i,j Pixels whose directions are close to the horizontal direction may be pixels indicating, for example, walls, pillars, etc., or pixels indicating objects arranged in the vertical direction. i,j When approaching the horizontal direction, the processor 130 may modify the normal vector N according to the direction perpendicular to the gravity direction G. i,j direction. Figure 6b In the example, since the normal vector N i,j The direction of the normal vector N is close to the horizontal direction, the processor 130 may modify the normal vector N in the direction perpendicular to the gravity direction G. i,j As a result of the modification, the processor 130 may obtain a modified normal vector N' i,j .

[0099] The processor 130 may use the modified normal vector N' i,j The direction is modified by the 3D coordinate value P of the first pixel i,j In the embodiment of the present disclosure, the processor 130 may use the modified normal vector N' i,j and the normal vector N before modification i,j The angular difference between them modifies the orientation of the plane just as much.

[0100] Figure 7 is a diagram illustrating a training operation of the augmented reality device 100 to obtain a depth map 730 by using the artificial intelligence model 700 according to an embodiment of the present disclosure.

[0101] Reference Figure 7 , the augmented reality device 100 may include a stereo camera including a left eye lens and a right eye lens. The augmented reality device 100 may input a left eye image 710L obtained by using the left eye camera and a right eye image 710R obtained by using the right eye camera to the AI ​​model 700. The augmented reality device 100 may obtain a depth map 730 by calculating a disparity based on a similarity between a pixel intensity value in the left eye image 710L and a pixel intensity value in the right eye image 710R using the AI ​​model 700.

[0102] In an embodiment of the present disclosure, the AI ​​model 700 may be implemented as a deep neural network model. The deep neural network model may be a known artificial intelligence model, such as a convolutional neural network (CNN) model, a recurrent neural network (RNN) model, a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q network. For example, the deep neural network model used to obtain the depth map 730 may include, but is not limited to, DispNet. For example, when the AI ​​model 700 is implemented as a CNN (such as DispNet), the AI ​​model 700 may extract a feature map by convolving each of the left eye image 710L and the right eye image 710R. During the process of extracting the feature map from the left eye image 710L and the right eye image 710R, respectively, the weights may have the same value and may be shared. The AI ​​model 700 may be trained to output a depth map 730 by performing hierarchical refinement, wherein the difference between the feature maps extracted from the left eye image 710L and the right eye image 710R, respectively, is accumulated as a 3D cost volume, and the accumulated 3D cost volume is up-convolved. In an embodiment of the present disclosure, the IMU sensor measurement value 720 is applied as an input to the AI ​​model 700, and the IMU sensor measurement value 720 may be input to the 3D cost volume. There may be an error between the depth map 730 output through the training process and the true value of the disparity based on the similarity between the pixel intensity values ​​of the left eye image 710L and the pixel intensity values ​​of the right eye image 710R, and the augmented reality device 100 can train the AI ​​model 700 by using the loss 740 to modify the error of the depth map 730.

[0103] The augmented reality device 100 obtains information about the direction of gravity based on the IMU sensor measurement value 720, and can calculate the normal vector N′ based on the modified normal vector N′ according to the direction of gravity or a direction perpendicular to the direction of gravity. i,j (See Figure 6a and Figure 6b ) is used to calculate the loss 740. Figures 8 to 10 A detailed example of the augmented reality device 100 calculating the loss 740 is described in detail.

[0104] Figure 8 is a flowchart illustrating a method of adjusting a depth value of each pixel in a depth map performed by the augmented reality device 100 according to an embodiment of the present disclosure.

[0105] Figure 8 Operation S810 and operation S820 are Figure 3 The details of operation S340 can be found in the following table. Figure 3 After operation S330 is performed Figure 8 Operation S810 shown in FIG.

[0106] In operation S810, the augmented reality device 100 may be based on the modified normal vector N′ i,j (See Figure 6a and Figure 6b ) is used to calculate the loss 740 using the depth values ​​of the pixel and multiple adjacent pixels on the plane defined by Figure 7 ). The following will refer to Fig. 9 and Fig.10A detailed example of the augmented reality device 100 calculating the loss 740 is described in detail.

[0107] Fig. 9 7 is a flowchart illustrating a method of calculating a loss 740 performed by the augmented reality device 100 according to an embodiment of the present disclosure.

[0108] Fig. 9 Operations S910 to S940 are Figure 8 The detailed process of operation S810 is shown in FIG. Fig. 9 Operation S940 is then performed Figure 8 Operation S820.

[0109] Fig.10 740 is a diagram showing the operation of the augmented reality device 100 for calculating the loss 740 according to an embodiment of the present disclosure. Fig. 9 and Fig.10 To describe Figure 8 Operation S810.

[0110] Reference Fig. 9 In operation S910, the augmented reality device 100 defines a plane including a pixel having a modified normal vector and a plurality of adjacent pixels that are adjacent to the pixel in position. Fig.10 , the processor 130 of the augmented reality device 100 (see Figure 2 ) can be defined including a modified normal vector N' i,j The 3D coordinate value P of the first pixel i,j and 3D coordinate values ​​P of a plurality of adjacent pixels adjacent to the first pixel in the up-down direction and the left-right direction i-1,j , P i+1,j , P i,j-1 , P i,j+1 The 3D coordinate value P of the first pixel i,j and the 3D coordinate values ​​P of multiple adjacent pixels i-1,j , P i+1,j , P i,j-1 , P i,j+1 Can have different depth values.

[0111] exist Fig. 9 In operation S920, the augmented reality device 100 obtains depth values ​​of a plurality of adjacent pixels based on a plurality of points where the defined plane and the ray vector of the camera intersect. Fig.10 , the processor 130 may identify the plane defined in operation S910 and the ray vector of the camera 110 ( ) intersects a plurality of points. The ray vector ( ). The processor 130 may obtain the depth values ​​D' of the multiple identification points. i,j , D' i-1,j , D' i+1,j , D' i,j-1 , D' i,j+1 .exist Fig.10 In the embodiment of the present invention, the first depth value D' i,j It can be equal to the 3D coordinate value P of the first pixel i,j The second depth value D' i-1,j may be equal to the 3D coordinate value P of the second pixel i-1,j However, with a second depth value D' i-1,j The position of the second point may be located at the 3D coordinate value P of the second pixel i-1,j Similarly, the third depth value D' i+1,j It can be equal to the 3D coordinate value P of the third pixel i+1,j The depth value of the third point can be the same as the 3D coordinate value P of the third pixel. i+1,j The location is different.

[0112] exist Fig. 9 In operation S930, the augmented reality device 100 calculates the difference between the depth value of the pixel and the depth values ​​of a plurality of adjacent pixels. Fig.10 , the processor 130 may calculate the depth value D' at the first pixel based on the following equation 3 i,j and the depth values ​​D' of multiple adjacent pixels i-1,j , D' i+1,j , D' i,j-1 , D' i,j+1 The difference between pq .

[0113] Equation 3

[0114] In the above equation 3, D' p Represented as having a modified normal vector N' i,j The depth value D' of the first pixel of the reference pixel i,j , D' q Denotes the depth value D' of each of the plurality of adjacent pixels located adjacent to the first pixel i-1,j , D' i+1,j , D' i,j-1 , D' i,j+1 According to the above equation 3, the processor 130 can calculate D' i,j -D' i-1, j , D' i,j -D' i+1, j , D'i,j -D' i, j-1 and D' i,j -D' i, j+1 To get d pq .

[0115] exist Fig. 9 In operation S940, the augmented reality device 100 obtains the loss by weighted sum operation, wherein the weight is applied to the calculated difference. Fig.10 , the processor 130 can calculate the first weight w q D and the second weight w pq C The difference d pq Perform weighted sum operation to obtain loss Loss g In the embodiment of the present disclosure, the loss Loss g It can be calculated by the following equation 4.

[0116] Equation 4

[0117] In the above equation 4, based on the depth map 730 (see Figure 7 ) and the distance between the position of the camera 110 to determine the first weight w q D In an embodiment of the present disclosure, the depth values ​​D of the plurality of adjacent pixels in the depth map 730 may be used to obtain the depth map 730. i-1,j , D i+1,j , D i,j-1 , D i,j+1 To calculate the first weight w q D , as shown in Equation 5 below.

[0118] Equation 5

[0119] Referring to the above equation 5, D q Denotes the depth values ​​D of multiple adjacent pixels in the depth map 730 i-1,j , D i+1,j , D i,j-1 , D i,j+1 , and as the depth values ​​D of multiple adjacent pixels i-1,j , D i+1,j , D i,j-1 , D i,j+1 The first weight w q D As the depth values ​​D of multiple adjacent pixels increase i-1,j , D i+1,j , D i,j-1, D i,j+1 (ie, the distance between the adjacent pixels and the camera 110) decreases, the first weight w q D The value of may be increased. Since the accuracy of the depth value decreases as the distance increases in the method of extracting the depth map 730 using the stereo camera, the above equation 5 is designed to apply more weight to the pixels close to the camera 110 .

[0120] In the above equation 4, the second weight w may be determined based on the difference between the pixel intensity values ​​between the first pixel and the plurality of neighboring pixels in the depth map 730. pq C In an embodiment of the present disclosure, the second weight w may be calculated based on the difference in intensity values ​​between pixels in the depth map 730. pq C , as shown in Equation 6 below.

[0121] Equation 6

[0122] Referring to FIG. 6, I p represents the intensity value of the first pixel in the depth map 730, I q represents the intensity values ​​of multiple adjacent pixels. As the difference between the intensity value of the first pixel and each of the intensity values ​​of the multiple adjacent pixels decreases, the second weight w pq C When the intensity values ​​of pixels on the depth map 730 are similar, the possibility of having the same or similar depth value increases. Therefore, the above equation 6 is designed to apply more weight to the difference in the intensity values ​​of pixels in the depth map 730.

[0123] Return to reference Figure 8 In operation S820, the augmented reality device 100 may execute the calculated loss Loss g (See Fig.10 ) is applied to the training of the AI ​​model, so that the depth value of at least one pixel in the depth map can be adjusted.

[0124] exist Figures 8 to 10 In the embodiment, the modified normal vector N' may be based on i,j (See Figure 6a and Figure 6b ) defined by the plane and the ray vector of the camera 110 ( ) causes the depth value D' i,j , D' i-1,j , D' i+1,j , D' i,j-1 , D' i,j+1 To determine the loss gIn particular, by applying a first weight w determined based on the distance from the camera 110 q D and a second weight w determined based on the intensity value difference between pixels in the depth map pq C To calculate the loss Loss g , so the loss Loss g Applied to AI model training to improve the accuracy of depth values.

[0125] Fig.11 11 is a diagram illustrating an operation of the augmented reality device 100 obtaining a depth map 1130 by using the AI ​​model 1100 according to an embodiment of the present disclosure.

[0126] Reference Fig.11 , the augmented reality device 100 may include a stereo camera including a left eye lens and a right eye lens. The augmented reality device 100 may obtain a depth map 1130 by inputting a left eye image 1110L obtained by using the left eye lens and a right eye image 1110R obtained by using the right eye lens and an IMU sensor measurement value 1120 into an AI model 1100 and performing reasoning using the AI ​​model 1100.

[0127] like Figure 7 As shown in the embodiment, the loss 740 (see Figure 7 ) to train the AI ​​model 1100. The augmented reality device 100 may be provided with the IMU sensor 120 (see Figure 2 ) is input to the AI ​​model 1100, and the input IMU sensor measurement value 1120 can be applied to the 3D cost volume. During the inference of the AI ​​model 1100, a depth map 1130 can be output through hierarchical refinement (such as up-convolution of the 3D cost volume, etc.).

[0128] exist Fig.11 In the illustrated embodiment, the augmented reality device 100 obtains a depth map 1130 by using an AI model 1100, which is trained by using an IMU sensor measurement value 1120 including information about the direction of gravity, and thus, the accuracy of the depth value in the depth map 1130 can be improved. In addition, the augmented reality device 100 according to an embodiment of the present disclosure trains the AI ​​model 1100 by using the IMU measurement value 1120 obtained from the IMU sensor 120 as a basic element, and thus, no additional hardware module is required. Therefore, a technical effect of achieving lower power consumption while maintaining a compact size specification can be provided.

[0129] Fig.12is a flowchart illustrating a method of adjusting a depth value of each pixel in a depth map performed by the augmented reality device 100 according to an embodiment of the present disclosure.

[0130] Fig.12 Operation S1210 includes Figure 3 In operation S330. Fig.12 Operations S1220 to S1240 shown in FIG. Figure 3 The detailed process of operation S340 is shown in FIG.

[0131] Fig.13 1 is a diagram illustrating an operation of adjusting the depth value of each pixel in the depth map 1320 by a plane scanning method performed by the augmented reality device 100 according to an embodiment of the present disclosure. Fig.12 and Fig.13 The operation of the augmented reality device 100 adjusting the depth value of each pixel in the depth map 1320 is described below.

[0132] Reference Fig.12 In operation S1210, the augmented reality device 100 may be based on the IMU sensor 120 (see Figure 2 ) to modify the direction of the normal vectors in the left-eye image and the right-eye image. Fig.13 , the augmented reality device 100 may include a stereo camera, the stereo camera including a left eye lens and a right eye lens. The processor 130 (see Figure 2 ) can be obtained from the IMU sensor 120 (see Figure 2 ) and can modify the direction of the normal vector of each pixel in the left eye image 1300L obtained by using the left eye lens and the right eye image 1300R obtained by using the right eye lens based on the obtained gravity direction information. In the disclosed embodiment, the processor 130 can modify the direction of the normal vector of each pixel in the gravity direction and the direction perpendicular to the gravity direction. The detailed method of modifying the normal vector direction is the same as that of the above reference. Figure 6a and Figure 6b The methods described are the same, so their description is omitted. Reference Fig.12 In operation S1220, the augmented reality device 100 performs plane hypothesis along the direction of the modified normal vector or a direction perpendicular to the modified normal vector. Fig.13, the processor 130 may define the plane of each pixel by plane assumption for each of the left-eye image 1300L and the right-eye image 1300R. The processor 130 may define the plane of each pixel in the left-eye image 1300L by performing plane assumption along the modification direction of the normal vector of each pixel included in the left-eye image 1300L or in a direction perpendicular to the modification direction of the normal vector of each pixel. In the same manner, the processor 130 may define the plane of each pixel in the right-eye image 1300R by performing plane assumption along the modification direction of the normal vector of each pixel included in the right-eye image 1300R or in a direction perpendicular to the modification direction of the normal vector of each pixel.

[0133] Reference Fig.12 In operation S1230, the augmented reality device 100 obtains a depth value of each pixel by performing plane scanning along a plane defined by the plane hypothesis. Fig.13 , the processor 130 may perform plane scanning, in which a matching point in the right eye image 1300R is searched based on the two-dimensional position coordinate value (x, y) of the reference pixel on the plane defined in the left eye image 1300L. In an embodiment of the present disclosure, the processor 130 may identify a plane of the right eye image 1300R corresponding to the plane defined in the left eye image 1300L through a plane hypothesis, and search for a matching point in the plane identified in the right eye image 1300R. In an embodiment of the present disclosure, the processor 130 may identify a plane of the right eye image 1300R corresponding to the plane defined in the left eye image 1300L through a plane hypothesis, and search for a matching point in the plane identified in the right eye image 1300R. 0 to d max The processor 130 searches for a matching point corresponding to the reference pixel of the left eye image 1300L within the parallax search range of . The processor 130 measures the intensity value similarity between the reference pixel in the plane of the left eye image 1300L and the pixel in the corresponding plane of the right eye image 1300R, and may identify the pixel with the lowest intensity value dissimilarity in the right eye image 1300R as a matching point based on the curve graph 1310 indicating the relationship between the intensity value dissimilarity between pixels and the parallax. The processor 130 may calculate the distance d between the position of the reference pixel in the left eye image 1300L and the pixel identified as the matching point in the right eye image 1300R. x The processor 130 may obtain a depth value of each pixel by performing plane scanning on each pixel in the left-eye image 1300L and the right-eye image 1300R in the above-described manner.

[0134] Reference Fig.12 In operation S1240, the augmented reality device 100 adjusts the depth value of at least one pixel in the depth map by using the obtained depth value. Fig.13, the processor 130 of the augmented reality device 100 may adjust the depth value of the depth map 1320 by using the obtained depth value of each pixel.

[0135] exist Fig.12 and Fig.13 In an embodiment of the invention, the augmented reality device 100 may obtain a depth map 1320 with high precision by modifying a normal vector of each pixel in the gravity direction and performing a plane scan on a plane defined by the modified normal vector.

[0136] Fig.14 is a flowchart illustrating a method of adjusting a depth value of each pixel in a depth map by the augmented reality device 100 according to an embodiment of the present disclosure.

[0137] Fig.14 Operations S1410 to S1430 are shown as follows: Figure 3 The details of operation S340 can be obtained by performing Figure 3 After the operation S330 is performed Fig.14 Operation S1410 is shown.

[0138] Fig.15 is a diagram illustrating an operation of adjusting a depth value of each pixel in a depth map 1510 obtained by a time-of-flight (ToF) method, performed by the augmented reality device 100 according to an embodiment of the present disclosure.

[0139] Reference Fig.14 In operation S1410, the augmented reality device 100 defines a plane for each pixel in the depth map based on the modified normal vector. In an embodiment of the present disclosure, the augmented reality device 100 may include a ToF camera. The ToF camera detects reflected light reflected from an object by irradiating light using a light source, and obtains a depth value of the object based on ToF (a time difference between a time point at which the reflected light is detected and a time point at which the light is irradiated). Fig.15 , the augmented reality device 100 can obtain an RGB image 1500 and a depth map image 1510 of the object by using a ToF camera. The processor 130 (see Figure 2 ) may obtain a normal vector of each pixel from the depth map image 1510, and may modify the direction of the normal vector along the gravity direction or a direction perpendicular to the gravity direction. The processor 130 may define a plane for each pixel in the depth map image 1510 based on the direction of the modified normal vector. In an embodiment of the present disclosure, the processor 130 may define a plane including a pixel having a modified normal vector and a plurality of adjacent pixels that are adjacent to the pixel in position.

[0140] Reference Fig.14In operation S1420, the augmented reality device 100 identifies a location area in the depth map based on the area segmented according to the color information of the RGB image. Fig.15 , the processor 130 of the augmented reality device 100 may segment the RGB image 1500 into a plurality of regions 1501 to 1506 based on color information of each of a plurality of pixels included in the RGB image 1500. In an embodiment of the present disclosure, the processor 130 may segment the RGB image 1500 into a plurality of regions 1501 to 1506 based on an intensity value of each of a plurality of pixels in the RGB image 1500. The processor 130 may segment the RGB image 1500 into a plurality of regions 1501 to 1506 by calculating a difference in intensity values ​​of each of a plurality of pixels of the RGB image 1500 and grouping pixels having a calculated difference equal to or less than a critical value. The processor 130 may identify a plane region in the depth map image 1510 based on at least one piece of information among positions, shapes, and sizes of the plurality of segmented regions 1501 to 1506 of the RGB image 1500. In Fig.15 In the embodiment of FIG. 1 , the processor 130 may identify a plurality of plane regions 1511 to 1516 corresponding to the plurality of segmented regions 1501 to 1506 in the RGB image 1500 from the depth map image 1510 .

[0141] Reference Fig.14 , in operation S1430, the augmented reality device 100 may adjust the depth values ​​of pixels in the identified plane area based on the depth values ​​of adjacent pixels. In an embodiment of the present disclosure, the augmented reality device 100 may identify pixels to be modified in the plane area of ​​the depth map, and may adjust the depth values ​​of the identified pixels to be adjusted based on the depth values ​​of adjacent pixels in the same plane area. In this specification, "pixels to be adjusted" means pixels whose depth values ​​need to be adjusted, as well as pixels whose depth values ​​are different from pixels that have not obtained depth values ​​in the plane area or from adjacent pixels in the plane area that exceed a preset critical value. Fig.15 In an embodiment of the present invention, pixels for which depth values ​​are not obtained and pixels having depth value differences equal to or greater than a preset critical value may not be identified in the first to fifth plane regions 1511 to 1515 among the plurality of plane regions 1511 to 1516 identified from the depth map image 1510. In the sixth plane region 1516 among the plurality of plane regions 1511 to 1516, at least one pixel to be adjusted 1526 for which depth values ​​are not obtained may be identified. The processor 130 may adjust the depth value of the identified pixel to be adjusted 1526 based on the depth values ​​of adjacent pixels among the plurality of pixels included in the sixth plane region 1516.

[0142] In the case of ordinary ToF cameras, the accuracy of the depth value may be reduced, or the depth value may not be obtained at a location far from the camera. Fig.14 and Fig.15 In the embodiment of the invention, the augmented reality device 100 modifies the normal vector of each pixel based on the gravity direction in the depth map image 1510, defines a plane based on the modified normal vector, identifies the plane area of ​​the depth map image 1510 based on the color information of each pixel of the RGB image, and adjusts the depth value of the pixel 1526 to be adjusted in the plane area in the depth map image 1520. Therefore, the accuracy of the depth map can be improved.

[0143] Fig.16 is a diagram illustrating a 3D space model 1610 reconstructed through a depth map obtained in a conventional manner and a 3D space model 1620 reconstructed through a depth map obtained by the augmented reality device 100 according to an embodiment of the present disclosure.

[0144] Reference Fig.16 According to the 3D space model 1610 of the real world space 1600 obtained by the conventional depth map acquisition method of the related art, the surface of the object is inclined, and the accuracy of the 3D space model 1610 is low. In the conventional depth map acquisition method, the accuracy of the depth value decreases as it moves away from the camera.

[0145] The augmented reality device 100 according to the embodiment of the present disclosure is provided with an IMU sensor 120 (see Figure 2 ) The direction of the normal vector of each pixel is modified according to the gravity direction measured by the augmented reality device 100 and the direction perpendicular to the gravity direction, and the depth value of each pixel is adjusted based on the modified normal vector. Therefore, compared with the depth map obtained in the traditional method, the accuracy and resolution of the obtained depth map can be improved. In addition, in the 3D space model 1620 reconstructed from the depth map obtained by the augmented reality device 100 according to an embodiment of the present disclosure, the direction of the recognized object is modified to be perpendicular to or parallel to the gravity direction.

[0146] In addition, the augmented reality device 100 according to the embodiment of the present disclosure obtains gravity direction information by using the IMU sensor 120 as a basic element, thereby achieving low power consumption while maintaining a compact size. Therefore, the augmented reality device 100 of the present disclosure can provide a technical effect of increasing portability and device usage time and improving user convenience.

[0147] The present invention provides an augmented reality device 100 that adjusts a depth value based on a gravity direction. The augmented reality device 100 according to an embodiment of the present disclosure may include a camera 110 (see Figure 2 )、IMU sensor 120 (see Figure 2 ), at least one processor 130 (see Figure 2) and memory 140 (see Figure 2 ). The at least one processor 130 may obtain a depth map from an image obtained by using the camera 110 by executing at least one instruction stored in the memory 140. The at least one processor 130 may obtain a normal vector of at least one pixel included in the depth map. The at least one processor 130 may modify the direction of the normal vector of at least one pixel based on the gravity direction measured by the IMU sensor 120. The at least one processor 130 may adjust the depth value of at least one pixel based on the modified direction of the normal vector.

[0148] In an embodiment of the present disclosure, at least one processor 130 can obtain a normal vector by converting at least one pixel into a 3D coordinate value based on the normal vector of at least one pixel included in the depth map and the depth value of at least one pixel, and calculating the vector cross product of the 3D coordinate values ​​of multiple adjacent pixels adjacent in each downward and left and right directions.

[0149] In an embodiment of the present disclosure, the camera 110 may include a left eye lens (including a left eye image) and a right eye lens (including a right eye image). At least one processor 130 may obtain a depth map by applying the left eye image and the right eye image as input to an AI model and calculating the disparity according to the similarity between the intensity values ​​of pixels in the left eye image and the right eye image using the AI ​​model.

[0150] In an embodiment of the present disclosure, at least one processor 130 may calculate the loss of the depth map obtained by the AI ​​model based on the depth value of the pixel on the plane defined by the modified normal vector and the depth values ​​of multiple adjacent pixels adjacent to the pixel in position. At least one processor 130 may adjust the depth value of at least one pixel by performing training that applies the calculated loss to the AI ​​model.

[0151] In an embodiment of the present disclosure, at least one processor 130 may define a plane including a pixel having a modified normal vector and a plurality of adjacent pixels that are positionally adjacent to the pixel, and obtain depth values ​​of the plurality of adjacent pixels based on a plurality of points where the defined plane intersects with a ray vector of the camera 110. The at least one processor 130 may respectively calculate a difference between the depth value of the obtained pixel and the depth values ​​of the plurality of adjacent pixels. The at least one processor 130 may obtain a loss by performing a weighted sum, wherein a weight is applied to a difference calculated from each of the plurality of adjacent pixels.

[0152] In an embodiment of the present disclosure, the weight may include a first weight determined based on a distance between each of a plurality of neighboring pixels in the depth map and the camera, and a second weight determined based on a difference in intensity values ​​between the pixel in the depth map and the plurality of neighboring pixels.

[0153] In an embodiment of the present disclosure, at least one processor 130 may obtain an adjusted depth map by performing inference of inputting the left-eye image and the right-eye image to a trained AI model.

[0154] In an embodiment of the present disclosure, at least one processor 130 may include a left eye lens for obtaining a left eye image and a right eye lens for obtaining a right eye image. At least one processor 130 may modify the direction of the normal vector of at least one pixel in the left eye image and the right eye image according to the direction of gravity or a direction perpendicular to the direction of gravity, and perform a plane assumption according to the direction of the modified normal vector or a direction perpendicular to the direction of the modified normal vector. At least one processor 130 may obtain a depth value of at least one pixel by performing a plane scan along a plane defined by the plane assumption. At least one processor 130 may adjust the depth value of at least one pixel in the depth map by using the obtained depth value.

[0155] In an embodiment of the present disclosure, the camera 110 may include a ToF camera, and the at least one processor 130 may obtain a depth map by using the ToF camera.

[0156] In an embodiment of the present disclosure, at least one processor 130 may define a plane for each pixel based on the modified normal vector. At least one processor 130 may identify a plane region of a plane defined in a depth map based on a region segmented according to color information of an RGB image. At least one processor 130 may adjust a depth value of at least one pixel in the depth map based on depth values ​​of adjacent pixels in the identified plane region.

[0157] The present disclosure provides a method for adjusting a depth value by an augmented reality device 100. In an embodiment of the present disclosure, the method may include a process of obtaining a depth map (S310) from an image obtained by using a camera 110. The method may include a process of obtaining a normal vector of at least one pixel included in the depth map (S320). The method may include a process of modifying a direction of the normal vector of at least one pixel based on a gravity direction measured by an IMU sensor 120 (S330). The method may include a process of adjusting a depth value of at least one pixel based on the modified direction of the normal vector (S340).

[0158] In an embodiment of the present disclosure, the process of obtaining a depth map (S310) may include a process of inputting a left eye image obtained by using a left eye lens and a right eye image obtained by using a right eye lens into an AI model, and a process of obtaining a depth map by calculating disparity based on the similarity of intensity values ​​between pixels of the left eye image and pixels of the right eye image using the AI ​​model.

[0159] In an embodiment of the present disclosure, the operation of adjusting the depth map of at least one pixel (S340) may include a process of calculating a loss (S810) of the depth map obtained by the AI ​​model based on the depth value of the pixel on the plane defined by the modified normal vector and the depth values ​​of a plurality of adjacent pixels that are positionally adjacent to the pixel. The operation of adjusting the depth value of at least one pixel (S340) may include a process of adjusting the depth value of at least one pixel in the depth map by performing a process of applying the calculated loss to the training of the AI ​​model (S820).

[0160] In an embodiment of the present disclosure, the operation of calculating the loss (S810) may include a process of defining a plane (S910) including a pixel having a modified normal vector and a plurality of neighboring pixels that are positionally adjacent to the pixel, and an operation of obtaining depth values ​​of the plurality of neighboring pixels (S920) based on a plurality of points where the defined plane intersects with a ray vector of the camera 110. The operation of calculating the loss (S810) may include a process of respectively calculating a difference between the depth value of the obtained pixel and the depth values ​​of the plurality of neighboring pixels (S930). The operation of calculating the loss (S810) may include a process of obtaining the loss (S940) by performing a weighted sum of applying a weight to the difference values ​​calculated from each of the plurality of neighboring pixels.

[0161] In an embodiment of the present disclosure, the weight may include a first weight determined based on a distance between each of a plurality of neighboring pixels in the depth map and the camera 110 , and a second weight determined based on a difference in intensity values ​​between a pixel in the depth map and the plurality of neighboring pixels.

[0162] In an embodiment of the present disclosure, the method may further include a process of obtaining an adjusted depth map by performing inference of inputting the left-eye image and the right-eye image into the trained AI model.

[0163] In an embodiment of the present disclosure, the process of modifying the direction of the normal vector of at least one pixel (S330) may include a process of modifying the direction of the normal vector of a left eye image obtained by using a left eye lens and a right eye image obtained by using a right eye lens according to a gravity direction or a direction perpendicular to the gravity direction. The process of adjusting the depth value of at least one pixel (S340) may include a process of performing a plane assumption (S1220) along the direction of the modified normal vector and a direction perpendicular to the direction of the modified normal vector. The process of adjusting the depth value of at least one pixel (S340) may include a process of obtaining a depth value of at least one pixel by performing a plane scan along a plane defined by the plane assumption. The process of adjusting the depth value of at least one pixel (S340) may include a process of adjusting the depth value of at least one pixel (S1240) by using the obtained depth value.

[0164] In an embodiment of the present disclosure, the process of obtaining the depth map ( S310 ) may obtain the depth map by using a ToF camera.

[0165] In an embodiment of the present disclosure, the process of adjusting the depth value of at least one pixel (S340) may include a process of defining a plane of the pixel based on the modified normal vector (S1410). The process of adjusting the depth value of at least one pixel (S340) may include a process of identifying a plane region of a plane defined in a depth map based on a region segmented according to color information of an RGB image (S1420). The process of adjusting the depth value of at least one pixel (S340) may include a process of adjusting the depth value of at least one pixel based on depth values ​​of adjacent pixels in the identified plane region (S1430).

[0166] The present disclosure provides a computer program product including a computer-readable storage medium. The storage medium may store instructions for an operation of obtaining a depth map from an image obtained by using the camera 110. The storage medium may store instructions for an operation of obtaining a normal vector of at least one pixel included in the depth map. The storage medium may store instructions related to an operation of modifying the direction of the normal vector of at least one pixel based on the gravity direction measured by the IMU sensor. The storage medium may store instructions related to an operation of adjusting a depth value of at least one pixel in the depth map based on the direction of the modified normal vector.

[0167] The program executed by the augmented reality device 100 described in the present disclosure may be implemented using hardware components, software components, and / or a combination of hardware components and software components. The program may be executed by all systems capable of executing computer-readable instructions.

[0168] Software may include computer programs, codes, instructions, or one or more combinations thereof to independently or collectively instruct or configure a processing device to operate as desired.

[0169] The software may be implemented as a computer program including instructions stored in a computer-readable storage medium. Examples of computer-readable recording media include magnetic storage media (e.g., ROM, RAM, floppy disks, hard disks, etc.) and optical recording media (e.g., CD-ROMs or digital versatile disks (DVDs)). The computer-readable recording medium may also be distributed on a network-coupled computer system so that the computer-readable code is stored and executed in a distributed manner. The medium may be stored in a memory, read by a computer, and executed by a processor.

[0170] The computer-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory" only means that the storage medium is a tangible device, does not include signals, and the term does not distinguish whether the data is semi-permanently stored in the storage medium or temporarily stored in the storage medium. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0171] In addition, the program according to the embodiment of the present disclosure may be provided and included in a computer program product. The computer program product may be traded between a seller and a buyer as a product.

[0172] The computer program product may include a software program or a computer-readable storage medium in which the software program is stored. For example, the computer program product may include a product (e.g., a downloadable application) in the form of a software program electronically distributed by the manufacturer of the augmented reality device 100 or an electronic market (e.g., Samsung Galaxystore®). For electronic distribution, at least a portion of the software program may be stored on a storage medium or may be temporarily generated. In this case, the storage medium may include a storage medium of a server of the manufacturer of the augmented reality device 100, a server of the electronic market, or a relay server that temporarily stores the software program.

[0173] In a system consisting of the augmented reality device 100 and / or a server, the computer program product may include a storage medium of the server or a storage medium of the augmented reality device 100. Alternatively, when there is a third device (e.g., a mobile device) communicating with the augmented reality device 100, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include the software program itself transmitted from the augmented reality device 100 to the third device or from the third device to the electronic device.

[0174] In this case, one of the augmented reality device 100 and the third device may run a computer program product to perform the method according to an embodiment of the present disclosure. Optionally, at least one or more of the augmented reality device 100 and the third device may run a computer program product to perform the method according to an embodiment of the present disclosure in a distributed manner.

[0175] For example, the augmented reality device 100 may execute the program stored in the memory 140 (see Figure 2 ) is used to control another electronic device (eg, a mobile device) that communicates with the augmented reality device 100 to execute a method according to an embodiment of the present disclosure.

[0176] In another example, the third device may run the computer program product, and may control an electronic device communicating with the third device to perform a method according to an embodiment of the present disclosure.

[0177] When the third device runs the computer program product, the third device downloads the computer program product from the augmented reality device 100 and runs the computer program product. Alternatively, the third device may run the computer program product provided in a preloaded state to perform the method according to an embodiment of the present disclosure.

[0178] Although the present disclosure has been described with reference to example embodiments, various changes and modifications may be made by those skilled in the art without departing from the spirit and scope of the present disclosure as set forth in the appended claims. For example, according to the present disclosure, even if the technology is implemented by other methods and processes different from those described above, and / or even if components (such as computer systems or modules) are coupled in other ways than those described above, or are replaced or substituted by other components or equivalents, the desired results can be achieved.

Claims

1. An augmented reality device, include: camera; An inertial measurement unit (IMU) sensor configured to measure the direction of gravity; A memory storing at least one instruction; as well as at least one processor configured to execute the at least one instruction to: obtaining a depth map from an image obtained by using the camera, obtaining a normal vector of at least one pixel included in the depth map, modifying a direction of a normal vector of the at least one pixel based on the direction of gravity measured by the IMU sensor, and The depth value of the at least one pixel is adjusted based on the direction of the modified normal vector.

2. The augmented reality device according to claim 1, in, The camera includes a left-eye lens configured to obtain a left-eye image and a right-eye lens configured to obtain a right-eye image, and Wherein, the at least one processor is further configured to execute the at least one instruction to: applying the left eye image and the right eye image as input to an artificial intelligence (AI) model, and The depth map is obtained by obtaining disparity according to similarity of intensity values ​​between pixels in the left-eye image and pixels in the right-eye image using the AI ​​model.

3. The augmented reality device according to claim 2, in, The at least one processor is further configured to execute the at least one instruction to: Obtaining a loss of the depth map obtained by the AI ​​model based on a depth value of a pixel on a plane defined by the modified normal vector and depth values ​​of the plurality of neighboring pixels adjacent to the pixel, and The depth value of the at least one pixel is adjusted by performing training that applies the loss to the AI ​​model.

4. The augmented reality device according to claim 3, in, The at least one processor is further configured to execute the at least one instruction to: defining the plane to include the pixel having the modified normal vector and the plurality of neighboring pixels adjacent to the pixel, obtaining depth values ​​of the plurality of adjacent pixels based on a plurality of points where the defined plane intersects with a ray vector of the camera, obtaining a difference between the depth value of the pixel and each of the depth values ​​of the plurality of adjacent pixels, and The loss is obtained by performing application of a weight to a weighted sum of the difference values ​​obtained for each of the plurality of neighboring pixels.

5. The augmented reality device according to claim 4, in, The weight includes a first weight determined based on a distance between each of the plurality of neighboring pixels in the depth map and the camera, and a second weight determined based on a difference in intensity values ​​between the pixel in the depth map and each of the plurality of neighboring pixels.

6. The augmented reality device according to claim 5, in, The at least one processor is also configured to execute the at least one instruction to obtain an adjusted depth map by performing inference providing the left-eye image and the right-eye image to the trained AI model.

7. The augmented reality device according to claim 1, in, The camera comprises a time-of-flight (ToF) camera, and Wherein, the at least one processor is further configured to execute the at least one instruction to: By using the ToF camera to obtain the depth map, defining a plane for each pixel of the at least one pixel based on the modified normal vector, identifying a plane region of the plane defined in the depth map based on a region segmented according to color information of the RGB image, and The depth value is adjusted based on depth values ​​of neighboring pixels in the planar region.

8. A method for adjusting a depth value performed by an augmented reality device, the method include: obtaining a depth map from an image obtained by using a camera; obtaining a normal vector of at least one pixel included in the depth map; modifying a direction of a normal vector of the at least one pixel based on a direction of gravity measured by an inertial measurement unit (IMU) sensor; as well as The depth value of the at least one pixel is adjusted based on the direction of the modified normal vector.

9. The method according to claim 8, in, Obtaining the depth map includes: providing a left eye image obtained by using a left eye lens and a right eye image obtained by using a right eye lens to an artificial intelligence AI model; and The depth map is obtained by obtaining disparity according to similarity of intensity values ​​between pixels in the left-eye image and pixels in the right-eye image using the AI ​​model.

10. The method according to claim 9, in, Adjusting the depth value of each pixel in the at least one pixel in the depth map comprises: Obtaining a loss of the depth map obtained by the AI ​​model based on a depth value of a pixel on a plane defined by the modified normal vector and depth values ​​of a plurality of neighboring pixels adjacent to the pixel; and The depth value of the at least one pixel is adjusted by performing training that applies the loss to the AI ​​model.

11. The method according to claim 10, in, Identifying the loss includes: defining the plane to include the pixel having the modified normal vector and the plurality of neighboring pixels adjacent to the pixel; Obtaining depth values ​​of the plurality of adjacent pixels based on a plurality of points where the defined plane intersects with a ray vector of the camera; obtaining a difference between the depth value of the pixel and each of the depth values ​​of the plurality of adjacent pixels; and The loss is obtained by performing a weighted sum of applying a weight to the difference values ​​obtained from each of the plurality of neighboring pixels.

12. The method according to claim 11, in, The weight includes a first weight determined based on a distance between each of the plurality of neighboring pixels in the depth map and the camera, and a second weight determined based on a difference in intensity values ​​between the pixel in the depth map and each of the plurality of neighboring pixels.

13. The method of claim 12, further comprising obtaining an adjusted depth map by performing inference providing the left-eye image and the right-eye image to the trained AI model.

14. The method according to claim 8, in, Obtaining the depth map includes obtaining the depth map by using a time-of-flight camera, and Wherein, modifying the depth value of the at least one pixel includes: defining a plane for each pixel of the at least one pixel based on the modified normal vector; identifying a plane region of the plane defined in the depth map based on a region segmented according to color information of the RGB image, and The depth value of the at least one pixel is adjusted based on depth values ​​of neighboring pixels in the planar region.

15. A non-transitory computer-readable storage medium comprising instructions executed by an augmented reality device to perform a method comprising the steps of: obtaining a depth map from an image obtained by using a camera; obtaining a normal vector of at least one pixel included in the depth map; modifying a direction of a normal vector of the at least one pixel based on a direction of gravity measured by an inertial measurement unit sensor; as well as The depth value of the at least one pixel is adjusted based on the direction of the modified normal vector.

Images