Pose determination method and device for augmented reality providing device

Abstract

To provide a method and device for pose determination of an augmented reality providing device.SOLUTION: A pose determination method of an augmented reality providing device includes steps of: estimating motion information of the augmented reality providing device using a motion sensor of the augmented reality providing device; determining first pose information of the augmented reality providing device on the basis of the motion information; acquiring a current image frame through an image sensor of the augmented reality providing device; extracting a feature point of an object from the current image frame; estimating second pose information of the augmented reality providing device on the basis of feature point position information that is pre-defined for the extracted feature point and the object; determining respective reliability values of the first pose information and the second pose information on the basis of the motion information and extraction information of the feature point; and determining a pose of the augmented reality providing device on the basis of the first pose information, the second pose information, and the reliability values.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a technique for determining a pose of an augmented reality providing device and an augmented reality providing technique. [Background technology]

[0002] Recently, electronic devices capable of realizing virtual reality (VR), augmented reality (AR), or mixed reality (MR) have been developed, drawing growing interest. Among these, augmented reality is a display technology that combines and displays virtual objects and information in a real-world environment. Augmented reality has the advantage of being applicable to various real-world environments and is attracting attention as a next-generation display technology suitable for ubiquitous environments and Internet of Things environments. Augmented reality provides users with new experiences by combining a panoramic view of the outside world with virtual images, and can be a means of conveying information more efficiently and realistically. Summary of the Invention [Problem to be solved by the invention]

[0003] An object of the present invention is to provide a technique for determining the pose of an augmented reality providing device and an augmented reality providing technique. [Means for solving the problem]

[0004] A pose determination method for an augmented reality providing device according to one embodiment includes the steps of estimating motion information of the augmented reality providing device using a motion sensor of the augmented reality providing device, determining first pose information of the augmented reality providing device based on the motion information, acquiring a current image frame via an image sensor of the augmented reality providing device, extracting feature points of an object from the current image frame, estimating second pose information of the augmented reality providing device based on the extracted feature points and predefined feature point position information for the object, determining reliability values ​​for each of the first pose information and the second pose information based on the motion information and the extracted feature point information, and determining a pose of the augmented reality providing device based on the first pose information, the second pose information, and the reliability value.

[0005] The step of determining the reliability value may include a step of determining the reliability value based on a difference between a feature point position of the object at the current time estimated based on the motion information and a feature point position of the object at the current time based on the feature point extraction information.

[0006] The step of determining the reliability value may include the step of determining the reliability value of the first pose information to be higher than the reliability value of the second pose information if the difference satisfies a condition.

[0007] The step of determining the reliability value may include determining the reliability value based on a degree of movement of the augmented reality providing device according to the movement information.

[0008] The step of determining the reliability value may include the step of determining the reliability value of the first pose information to be higher than the reliability value of the second pose information if the degree of movement satisfies a condition.

[0009] The step of determining the pose of the augmented reality providing device may include a step of determining the pose of the augmented reality providing device by combining the first pose information and the second pose information based on the reliability values ​​of the first pose information and the second pose information, respectively.

[0010] The predefined feature point position information may indicate three-dimensional coordinate information of a reference feature point located on the surface of the object when the object is in a reference pose.

[0011] According to one embodiment, a pose determination device for determining a pose of an augmented reality providing device includes: a motion sensor for measuring motion information of the augmented reality providing device; an image sensor for capturing an image of at least a portion of the periphery of the augmented reality providing device to acquire a current image frame; and a processor for determining a pose of the augmented reality providing device based on the measured motion information and the acquired current image frame. The processor determines first pose information of the augmented reality providing device based on the motion information, extracts feature points of an object from the current image frame, estimates second pose information of the augmented reality providing device based on the extracted feature points and predefined feature point position information for the object, determines reliability values ​​for the first pose information and the second pose information based on the motion information and the extracted feature point information, and determines the pose of the augmented reality providing device based on the first pose information, the second pose information, and the reliability values.

[0012] According to one embodiment, an augmented reality providing device includes a motion sensor that measures motion information of the augmented reality providing device, an image sensor that captures an image of at least a portion of the surroundings of the augmented reality providing device to acquire a current image frame, a processor that estimates a pose of the augmented reality providing device based on the motion information and generates augmented reality content that is matched to a real object based on the estimated pose, and a display that visualizes the augmented reality content. The processor determines first pose information of the augmented reality providing device based on the motion information, extracts feature points of the real object from the current image frame, estimates second pose information of the augmented reality providing device based on the extracted feature points and feature point position information defined for the real object, determines reliability values ​​for the first pose information and the second pose information based on the motion information and the extracted feature point information, and determines the pose of the augmented reality providing device based on the first pose information, the second pose information, and the reliability value. [Effects of the Invention]

[0013] According to one embodiment, the pose of the augmented reality providing device can be estimated accurately and efficiently by using both image data and motion data.

[0014] According to one embodiment, the pose determination device can improve the accuracy of the pose estimation result by adaptively using first pose information determined from movement information based on a reliability value and second pose information determined from image data according to the situation.

[0015] According to one embodiment, the pose determination device does not calculate a pose by accumulating relative movement changes over time, so that pose errors do not accumulate over time and poses can be calculated with high accuracy.

[0016] According to one embodiment, the pose determination device can improve the accuracy of pose estimation by using position information of predefined reference feature points for an object when estimating the pose of the augmented reality providing device. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a diagram illustrating an overview of an augmented reality providing device according to an embodiment; [Figure 2] 1 is a diagram illustrating an overview of an augmented reality providing device according to an embodiment; [Figure 3] 1 is a block diagram showing a configuration of a pose determination device according to an embodiment. [Figure 4] 1 is a flowchart illustrating a pose determination method of an augmented reality providing apparatus according to an embodiment. [Figure 5] 10 is a flowchart illustrating an operation of calculating a reliability value according to an embodiment. [Figure 6] 10 is a flowchart illustrating an operation of calculating a reliability value according to another embodiment. [Figure 7] FIG. 10 is a block diagram showing the configuration of a pose determination device according to another embodiment. [Figure 8] 1 is a block diagram illustrating a configuration of an augmented reality providing device according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0018] The specific structural or functional descriptions disclosed in this specification are merely examples for the purpose of describing the embodiments, and the embodiments may be implemented in various different forms. The present invention is not limited to the embodiments described in this specification, and the scope of the present invention includes modifications, equivalents, or alternatives that fall within the technical concept described in the embodiments.

[0019] Although terms such as "first" or "second" may be used to describe multiple components, such terms should be construed only for the purpose of distinguishing one component from the other components. For example, a first component may be designated as a second component, and similarly, a second component may be designated as a first component.

[0020] When a component is referred to as being "coupled" or "connected" to another component, it should be understood that although it is directly coupled or connected to the other component, there may be other components in between.

[0021] The singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, the words "comprise" or "have" and the like indicate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0022] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which the present invention pertains. Commonly used predefined terms should be interpreted as having a meaning consistent with the meaning they have in the context of the relevant art, and should not be interpreted as having an ideal or overly formal meaning unless expressly defined herein.

[0023] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same reference numerals will be used to designate the same components regardless of the reference numerals, and redundant description thereof will be omitted.

[0024] 1 and 2 are diagrams illustrating an overview of an augmented reality providing device according to an embodiment.

[0025] Referring to FIG. 1 , the augmented reality providing device 110 is a device that provides an augmented reality service to the user 100. The augmented reality service is a service that overlays a virtual image on an image of the real world viewed by the user 100. The augmented reality providing device 110 can provide the user with a virtual image including content related to an actual object 120 along with an image of the real world via a display 115. The virtual image viewed by the user 110 may be a 2D or 3D image, and may be a fixed image or a dynamic image such as an animation. The augmented reality providing device 110 may be a wearable device that can be worn by the user 100, such as a glasses-type device or a head-mounted display (HMD).

[0026] 2, an example of augmented reality content 210 provided via the display 115 of the augmented reality providing device 110 is illustrated. The augmented reality content 210 is formed by aligning a virtual image 220 containing virtual information with a real-world object 120 viewed by the user 100. The augmented reality content 210, for example, replicates a virtual object that can be perceived from the real-world object 120, but also includes realizing virtual computer graphic effects on the object 120 (for example, realizing additional information such as a guidebook on an actual mechanical device using a virtual image).

[0027] When the user 100 moves their head or gaze, the gaze direction in which the user is viewing the object 120 changes. Therefore, in order to provide the user 100 with highly accurate augmented reality content 210, the virtual image 220 must be precisely adjusted in accordance with the change in gaze direction. Here, in order to achieve natural alignment between the object 120 and the virtual image 220, it is necessary to precisely measure the change in gaze direction. In order to realize realistic augmented reality content, the virtual image 220 must be displayed at an accurate position from the scene in which the user 100 is viewing the object 120.

[0028] The augmented reality providing device 110 may include a pose determining device (e.g., a pose determining device 700 shown in FIG. 7) that determines the pose of the augmented reality providing device 110. The pose determining device performs localization of the augmented reality providing device 110, and the localization results may be used to estimate the head movement of the user 100 and the direction in which the user 100 is looking. The pose of the augmented reality providing device 110 determined by the pose determining device includes information about the position and orientation of the augmented reality providing device 110 and may be expressed as six degrees of freedom (DoF). The pose of the augmented reality providing device 110 corresponds to the pose of the user 100 or the pose of an image sensor (e.g., a camera) included in the augmented reality providing device 110.

[0029] The pose determination device can determine the pose of the augmented reality providing device 110 using image data acquired by the image sensor and motion data acquired by the motion sensor. The pose determination device can accurately and efficiently estimate the pose of the augmented reality providing device 110 by using both the image data and the motion data. The image sensor and the motion sensor may be included in the augmented reality providing device 110. The image sensor acquires image data by capturing an image of at least a portion of the area surrounding the augmented reality providing device 110 (e.g., an area in front of the user 100). The motion sensor measures the movement of the augmented reality providing device 110 or the user 100, and may include an inertial sensor (inertial measurement unit sensor) including an acceleration sensor and a gyro sensor.

[0030] The pose determination device determines first pose information of the augmented reality providing device 110 based on the motion data, and determines second pose information of the augmented reality providing device 110 based on the image data. The first pose information corresponds to a pose measurement value based on a motion sensor, and the second pose information corresponds to a pose measurement value based on an image sensor.

[0031] The pose determination device determines first pose information by reflecting the movement from the previous time to the current time on the position of the augmented reality providing device 110 at the previous time. When determining second pose information, the pose determination device may estimate a pose using advance information on the object 120. The advance information may include three-dimensional coordinate information of reference feature points located on the surface of the object 120 when the object 120 is in a reference pose. Using the advance information, the second pose information can be determined quickly and accurately. The pose determination device extracts feature points 232, 233, 234, 235, 236, 237, and 238 of the object 120 from image data obtained by capturing the object 120, and determines the second pose information by comparing the positional relationship between the extracted feature points 232, 233, 234, 235, 236, 237, and 238 and the reference feature points of the object 120 predefined as advance information. In this specification, "feature points" may also be referred to as landmarks. Thereafter, the pose determination device determines a reliability value for each of the first pose information and the second pose information, and based on the determined reliability values, determines a final pose for the augmented reality providing device 110. The pose determination process of the augmented reality providing device 110 will be described in detail below with reference to FIGS. 3 to 6.

[0032] FIG. 3 is a block diagram showing the configuration of a pose determination device according to an embodiment.

[0033] 3, the pose determination device includes a motion information determiner 330, an object feature point extractor 340, a reliability determiner 350, and a pose determiner 360. The pose determination device determines the pose of the augmented reality providing device (e.g., the augmented reality providing device 110 shown in FIG. 1) using detection information from a motion sensor (e.g., an IMU sensor) 310 and detection information from an image sensor 320.

[0034] The motion sensor 310 may include an acceleration sensor and a gyro sensor, and may be built into the augmented reality providing device. Acceleration measurement information measured by the acceleration sensor and angular velocity measurement information measured by the gyro sensor are transmitted to the motion information determiner 330. The motion information determiner 330 receives motion measurement values ​​of the augmented reality providing device from the motion sensor 310 and can estimate motion information of the augmented reality providing device based on the received motion measurement values.

[0035] The motion information determiner 330 estimates and updates motion information (e.g., acceleration, velocity, and angular velocity information) of the augmented reality providing device using the detection information of the motion sensor 310. The motion information determiner 330 acquires acceleration and / or velocity measurements related to the translational motion of the augmented reality providing device from the acceleration sensor and calculates the acceleration and velocity of the motion of the augmented reality providing device based on the acquired measurements. The motion information determiner 330 acquires angular velocity measurements related to the rotational motion of the augmented reality providing device from the gyro sensor and calculates the angular velocity of the motion of the augmented reality providing device based on the acquired measurements. The motion information determiner 330 transmits the estimated motion information of the augmented reality providing device to the pose determiner 360. The motion information determiner 330 receives correction information for correcting errors in the motion information from the pose determiner 360 and corrects the calculation of the motion information based on the received correction information. Through such correction, the motion information of the augmented reality providing device can be determined more accurately.

[0036] The image sensor 320 may include one or more cameras and may capture color or black-and-white image data. The image data captured by the image sensor 320 may include multiple video frames captured at different times. The image data captured by the image sensor 320 is transmitted to the object feature extractor 340.

[0037] The object feature point extractor 340 extracts feature points of an object from the image data. The object feature point extractor 340 may detect the position of a feature point of a predefined object in the image data using two-dimensional coordinates. The object feature point extractor 340 may extract feature points using, for example, a Harris Corner feature point detection algorithm, a scale invariant feature transform (SIFT) feature point detection algorithm, and / or a feature point detection algorithm using a neural network. However, the scope of the embodiments is not limited to these feature point detection algorithms, and various other feature point detection algorithms may be used.

[0038] The reliability determiner 350 receives motion information from the motion information determiner 330 and object feature point extraction information from the object feature point extractor 340, and can determine reliability values ​​in real time. The reliability determiner 350 determines reliability values ​​(or importance) of first pose information of the augmented reality providing device estimated based on the motion information and second pose information of the augmented reality providing device estimated based on the feature point extraction results. The reliability value of the first pose information indicates an estimate or probability value for how reliable or accurate the first pose information is, and the reliability value of the second pose information indicates an estimate or probability value for how reliable or accurate the second pose information is.

[0039] The first pose information may be determined by applying a movement value of the augmented reality providing device based on the movement information to pose information of the augmented reality providing device determined previously. The second pose information may be determined based on a comparison result between the positions of feature points extracted by the object feature point extractor 340 and the positions of reference feature points predefined for the object. Information about the positions of predefined feature points includes 3D coordinate information of reference feature points located on the surface of the object when the known object is in a reference pose. The pose determination device knows the shape information of the object in advance through prior information, and the object is considered to be immobile. In the case of prior information, the prior information is generated by previously defining the feature points of the object as reference feature points and recording 3D position information of the reference feature points when the object is in the reference pose. The second pose information may be derived by comparing the positions of feature points extracted from the image data with the positions of the reference feature points when the object is in the reference pose. The second pose information may be estimated through matching information between the extracted feature points and the reference feature points that correspond to each other.

[0040] The first pose information based on motion information and the second pose information based on feature point extraction information each serve different purposes. The second pose information serves to accurately correct the pose estimation of the augmented reality providing device based on the position of an object's feature points when there is little or no degradation in image data quality, and the first pose information serves to prevent a decrease in the accuracy of the pose estimation result even when quality degradation of the image data occurs due to, for example, movement of the image sensor 320. For example, if an object is partially or entirely outside the image data, the accuracy of the second pose information decreases, but this decrease in accuracy can be compensated for using the first pose information based on motion information. Furthermore, although acquiring image data and determining the second pose information from the image data has a relatively long delay compared to acquiring motion information and determining the first pose information from the motion information, using the first pose information based on motion information allows the pose of the augmented reality providing device to be quickly determined in situations where quick pose determination is required.

[0041] In one embodiment, the reliability determiner 350 may determine the reliability value using a filter-based algorithm such as a Kalman filter. The reliability value determined by the reliability determiner 350 may be determined to be, for example, a value within a certain range between a minimum and maximum value determined experimentally, and the reliability value may change over time. The reliability value determined by the reliability determiner 350 may be transmitted to the pose determiner 360.

[0042] The pose determiner 360 determines pose information indicating the position and orientation of the augmented reality providing device based on the first pose information, the second pose information, and the reliability value determined by the reliability determiner 350, and outputs the determined pose information. The pose information of the augmented reality providing device corresponds to the pose information of the image sensor 320. The pose information may be defined as variables indicating six degrees of freedom (DoF) (e.g., x, y, z, Rx, Ry, Rz) and may be expressed as a six-dimensional vector or a matrix corresponding to the vector.

[0043] Meanwhile, the period at which the motion measurement values ​​are output from the motion sensor 310 and the period at which the image data are output from the image sensor 320 may be the same or different. If the periods are different, for example, if the motion measurement values ​​are provided from the motion sensor 310 at an output frequency of 200 Hz and the image data are provided from the image sensor 320 at an output frequency of 30 Hz, the pose determiner 360 may output the final pose information at the output frequency of the sensor having the higher output frequency.

[0044] The pose determination device can improve the accuracy of estimating a final pose by adaptively using first pose information determined from movement information and second pose information determined from image data based on a reliability value according to a situation. The pose determination device does not calculate a pose by accumulating relative movement changes over time, so pose errors do not accumulate over time and poses can be calculated with high accuracy. In addition, the pose determination device can improve the accuracy of pose estimation by using position information of reference feature points predefined for an object in pose estimation of an augmented reality providing device.

[0045] 4 to 6 are flowcharts illustrating a pose determination method of an augmented reality providing device according to an embodiment. The pose determination method may be performed by a pose determination device (for example, a pose determination device 700 shown in FIG. 7).

[0046] 4, in step S410, the pose determination device estimates motion information of the augmented reality provision device using a motion sensor of the augmented reality provision device. The motion information of the augmented reality provision device corresponds to the motion of the user, more specifically, the motion of the user's head or gaze direction. The motion sensor may include an acceleration sensor that measures acceleration values ​​and a gyro sensor that measures angular velocity values. The pose determination device can estimate the acceleration, velocity, and angular velocity for the motion of the augmented reality provision device based on the motion measurements (e.g., acceleration values, angular velocity values) measured by the motion sensor.

[0047] In step S415, the pose determination device determines first pose information of the augmented reality providing device based on the motion information of the augmented reality providing device. The pose determination device may determine the first pose information based on feature points and motion information of an object extracted from a previous image frame at a previous time point. The pose determination device may reflect motion information measured between the previous time point and the current time point on position information of the feature points extracted from the previous image frame, calculate how the positions of the feature points have changed at the current time point, and determine first pose information at the current time point based on the calculation result.

[0048] In step S420, the pose determination device acquires a current image frame via an image sensor of the augmented reality providing device. The image sensor acquires image data by capturing at least a portion of an area around the augmented reality providing device (e.g., an area where the user's head is facing or an area where the user's gaze is facing). The image data may be acquired at specific time intervals, and may be acquired in the form of an image frame for each time.

[0049] In step S430, the pose determination device extracts feature points of an object from the current video frame and determines second pose information of the augmented reality providing device based on the extracted feature points. The pose determination device determines whether a predefined object exists in the current video frame and extracts feature points from the object. Prior information on an object existing in real space may be utilized to determine the second pose information. The prior information includes predefined feature point position information for the object's feature points. Here, the predefined feature point position information indicates 3D coordinate information of reference feature points located on the surface of the object when the object is in a reference pose. The reference feature point position information may be obtained by scanning the object in real space using a 3D scanner and determining the positions of the reference feature points. The pose determination device may determine the second pose information based on a comparison result between 2D position information of feature points extracted from the current video frame and 3D position information of predefined reference feature points. The pose determination device estimates the second pose information based on a position difference between corresponding feature points extracted from the current video frame and the reference feature points. The number of reference feature points used to determine the second pose information is not limited. All or part of the object's global reference feature points may be used.

[0050] In step S440, the pose determination device determines a reliability value for each of the first pose information and the second pose information based on the motion information and the feature point extraction information. In one embodiment, the pose determination device may determine the reliability value based on an estimation covariance derived by a Simultaneous Localization and Mapping (SLAM) method using a Kalman filter. For example, a large magnitude of the estimation covariance may be determined to have a small reliability value, and a small magnitude of the estimation covariance may be determined to have a large reliability value. When one or more of the feature point extraction information and the motion information are updated, the pose determination device may re-determine a reliability value for one or more of the first pose information and the second pose information. Therefore, the reliability value may change over time.

[0051] If the pose determination device determines that the movement of the augmented reality providing device is large, the pose determination device may set a reliability value of the first pose information based on the movement information high and a reliability value of the second pose information based on the feature point extraction information low. Also, if the pose determination device determines that the entire area of ​​the object is displayed in the current video frame and the video quality of the current video frame is good, the pose determination device may set a reliability value of the second pose information high and a reliability value of the first pose information low. If the object area is cut off in the current video frame or the video quality of the current video frame is poor, the pose determination device may set a reliability value of the second pose information low and a reliability value of the first pose information high.

[0052] The pose determination device may determine a reliability value based on a difference between a feature point position of an object at a current time estimated based on the motion information and a feature point position of the object at a current time based on the feature point extraction information. If the difference satisfies a condition (e.g., if the difference is greater than a threshold), the pose determination device may determine a reliability value of the first pose information based on the motion information to be higher than a reliability value of the second pose information based on the feature point extraction information. If the object is located in the current video frame and the video quality of the current video frame satisfies a condition (e.g., if a measured value of the video quality is greater than a threshold), the pose determination device may determine a reliability value of the second pose information to be higher than the reliability value of the first pose information. The pose determination device may determine a reliability value based on a movement degree of the augmented reality providing device according to the motion information. If the movement degree of the augmented reality providing device satisfies a condition (e.g., if the movement degree is greater than a threshold), the pose determination device may determine a reliability value of the first pose information to be higher than the reliability value of the second pose information. In another embodiment, the pose determination device may determine a reliability value of each feature point of the object based on a movement speed of each feature point of the object.

[0053] An embodiment in which the pose determination device determines a reliability value will be described below with reference to Figures 5 and 6. Referring to Figure 5, the pose determination device performs a process of performing a feature point-based reliability analysis, a movement speed-based reliability analysis, and determining a reliability value based on the reliability analysis results in order to determine the reliability.

[0054] In step S510, the pose determination apparatus performs a feature-point-based reliability analysis. The pose determination apparatus may compare feature point position measurements based on image data acquired by an image sensor with feature point position measurements based on motion information acquired by a motion sensor, and analyze the reliability based on the difference (or residual) between the two feature point position measurements. If there is a large difference between the feature point positions of an object at the current time estimated based on the motion information and the feature point positions of the object at the current time based on the feature point extraction information, the pose determination apparatus assigns a higher reliability value to the first pose information based on the motion information than to the second pose information. This is because the pose measurement values ​​of the second pose information derived based on the image sensor are significantly affected by the feature point detection performance of the object in the video frame. For example, if an object is partially or entirely outside the image sensor's capture area, the accuracy of the object's feature point positions may suddenly decrease, significantly degrading the feature point detection performance. Taking this situation into consideration, if there is a large difference between the feature point position determined based on the motion information and the feature point position extracted from the video frame, the pose determination device may determine a reliability value by regarding the feature point position determined based on the motion information as more reliable.

[0055] In step S520, the pose determination device performs a reliability analysis based on the movement speed of the augmented reality providing device (or the user). When the relative movement speed of the augmented reality providing device with respect to the object is high, the pose determination device sets a low reliability value of the second pose information based on the feature point extraction information. When the movement speed is high, image quality of the image data decreases due to image blur, etc., which reduces the accuracy of the feature point extraction information. Therefore, when the relative movement speed of the augmented reality providing device is high, the pose determination device sets a low reliability value of the second pose information and a relatively high reliability value of the first pose information based on the movement information, thereby preventing inaccurate pose estimation due to image quality degradation.

[0056] In step S530, the pose determination device determines a reliability value for each of the first pose information and the second pose information based on the reliability analysis results of steps S510 and S520. The pose determination device may determine the reliability value in real time within a predefined minimum-maximum range of reliability values. For example, the pose determination device may determine the reliability value as a sum or weighted sum of the difference between the feature point position measurements considered in the reliability analysis of step S510 and the relative movement speed of the augmented reality providing device considered in the reliability analysis of step S520. The weights used in the weighted sum may be experimentally determined. The pose determination device may use an accumulated value or average value of the sum or weighted sum as the reliability value.

[0057] In the embodiment shown in FIG. 5, all of the feature points of the object have the same reliability value. In other embodiments, the pose determination apparatus may instead determine a reliability value for each feature point. Referring to FIG. 6, the pose determination apparatus may perform steps S510 and S520 in the same manner as described above. In step S610, the pose determination apparatus determines a reliability value for each feature point of the object based on the movement speed of each feature point of the object. Therefore, different reliability values ​​may be assigned to each feature point. For example, assuming an object has 10 feature points, the 10 feature points do not have the same reliability value R, but each have a different reliability value R1, R2, R3, ..., R10. The pose determination apparatus calculates the movement speed at each feature point position of the object and calculates a reliability value for each feature point based on the magnitude of the calculated movement speed. The pose determination apparatus may determine the reliability value for each feature point as the sum or weighted sum of the difference between the feature point position measurements considered in the reliability analysis of step S510 and the movement speed at the corresponding feature point position.

[0058] Referring to FIG. 4, in step S450, the pose determination device determines a pose of the augmented reality providing device based on the first pose information, the second pose information, and a reliability value. The pose determination device determines feature values ​​indicating the position and orientation of the augmented reality providing device, and the feature values ​​have six degrees of freedom. The pose determination device may determine the pose of the augmented reality providing device by combining the first pose information and the second pose information based on the respective reliability values ​​of the first pose information and the second pose information. The pose determination device may use the reliability values ​​of the first pose information and the second pose information as weighting values ​​and determine a final pose of the augmented reality providing device based on pose information with a higher reliability value. The shape and position of a virtual image visualized in the augmented reality content may be adjusted based on the final pose determined by the pose determination device.

[0059] FIG. 7 is a block diagram showing the configuration of a pose determination device according to another embodiment.

[0060] 7, the pose determination device 700 is a device that determines the pose of an augmented reality providing device (e.g., the augmented reality providing device 110 shown in FIG. 1) and operates inside the augmented reality providing device. The pose determination device 700 may include a motion sensor 710, an image sensor 720, a processor 730, and a memory 740.

[0061] The motion sensor 710 (for example, the motion sensor 310 shown in FIG. 3) measures motion information indicating the motion of the augmented reality providing device. The motion of the augmented reality providing device corresponds to the motion of the user's head or the motion in the user's line of sight. The motion sensor 710 may include an acceleration sensor and a gyro sensor, and may be built into and disposed in the augmented reality providing device.

[0062] The image sensor 720 acquires image data. The image sensor 720 (e.g., the image sensor 320 shown in FIG. 3) may capture at least a portion of the surroundings of the augmented reality providing device to acquire a current image frame. The image sensor 720 may include one or more cameras and may be disposed at a specific position on the augmented reality providing device.

[0063] Memory 740 stores computer-readable instructions. When the instructions stored in memory 740 are executed by processor 730, processor 730 performs operations defined by the instructions. Memory 740 may include a computer-readable storage medium or device. For example, memory 740 may include random access memories (RAM), dynamic random access memories (DRAM), static random access memories (SRAM), or other forms of non-volatile memory known in the art.

[0064] The processor 730 controls the overall operation of the pose determination apparatus 700. The processor 730 may be a hardware device having a circuit with a physical structure for executing desired operations. The desired operations may include code or instructions included in a program. The hardware device may include a microprocessor, a central processing unit (CPU), a graphic processing unit (GPU), a processor core, a multi-core processor, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a neural processing unit (NPU), etc.

[0065] The processor 730 executes functions and commands to be executed within the pose determination apparatus 700. The processor 730 performs at least one of the operations and / or functions described above with reference to Figures 1 to 6. The processor 730 performs the functions and / or operations of the motion information determiner 330, the object feature point extractor 340, the confidence determiner 350, and the pose determiner 360 shown in Figure 3.

[0066] The processor 730 determines a pose of the augmented reality providing device based on the motion information measured by the motion sensor 710 and the current image frame acquired from the image sensor 720. The processor 730 determines first pose information of the augmented reality providing device based on the motion information. The processor 730 may extract feature points of an object from the current image frame and estimate second pose information of the augmented reality providing device based on the extracted feature points and feature point position information predefined for the object.

[0067] The processor 730 determines reliability values ​​of the first pose information and the second pose information based on the motion information and the feature point extraction information. As an example, if a difference between a feature point position of an object at a current time estimated based on the motion information and a feature point position of the object at a current time based on the feature point extraction information satisfies a predefined condition (e.g., the difference is greater than a threshold), the processor 730 may determine a reliability value of the first pose information higher than a reliability value of the second pose information. As another example, if a degree of movement of the augmented reality providing device based on the motion information satisfies a predefined condition (e.g., an acceleration change amount or a velocity change amount is greater than a threshold), the processor 730 may determine a reliability value of the first pose information higher than a reliability value of the second pose information. As a further example, if an object is located in a current video frame and the video quality of the current video frame satisfies a predefined condition (e.g., a video quality measurement value is greater than a threshold), the processor 730 may determine a reliability value of the second pose information higher than a reliability value of the first pose information.

[0068] The processor 730 may determine a final pose of the augmented reality providing device based on the first pose information, the second pose information, and the reliability value. The processor 730 may determine a final pose of the augmented reality providing device by combining the first pose information and the second pose information based on the reliability values ​​of the first pose information and the second pose information. The processor 730 may use the reliability values ​​of the first pose information and the second pose information as weights in determining the final pose. In another embodiment, the processor 730 may determine a final pose based on the second pose information (for another example, the first pose information) when the reliability value of the first pose information (for another example, the second pose information) is equal to or less than a certain value.

[0069] FIG. 8 is a block diagram showing a configuration of an augmented reality providing device according to an embodiment.

[0070] 8, the augmented reality providing device 800 is a device that provides augmented reality content to a user and can generate augmented reality content by matching real objects with virtual content according to the pose of the augmented reality providing device 800 determined through the above-described pose determination process. The augmented reality providing device 800 may be a device in various fields, such as an advanced driver assistance system (ADAS), a head-up display (HUD) device, a 3D digital information display (DID), a navigation device, a neuromorphic device, a 3D mobile device, a smartphone, a smart TV, a smart vehicle, an Internet of Things (IoT) device, a medical device, or a measurement device. Here, the 3D mobile device may include, for example, a display device for displaying augmented reality, virtual reality, and / or mixed reality, a head-mounted display (HMD), a face-mounted display (FMD), or augmented reality glasses (AR glasses).

[0071] The augmented reality providing device 800 may include a processor 810, a memory 820, a sensor 830, a storage device 840, an input device 850, an output device 860, and a communication device 870. Each component of the augmented reality providing device 800 can communicate via a communication bus 880.

[0072] The processor 810 controls the overall operation of the augmented reality providing device 800 and executes functions and instructions to be executed within the augmented reality providing device 800. The processor 810 may perform one or more of the operations described above with reference to Figures 1 to 7. The processor 810 may estimate a pose of the augmented reality providing device based on movement information measured by a sensor, and generate augmented reality content that is matched to a real object based on the estimated pose.

[0073] The memory 820 stores information necessary for the processor 810 to perform operations. For example, the memory 820 may store instructions to be executed by the processor 810 and store relevant information while software or programs are being executed in the augmented reality providing device 800. The memory 820 may include RAM, DRAM, SRAM, or other forms of non-volatile memory known in the art.

[0074] The sensor 830 may include a motion sensor that measures motion information of the augmented reality providing device 800 and an image sensor that captures an image of at least a portion of the surroundings of the augmented reality providing device 800 to acquire a current image frame. The motion sensor may include an inertial measurement sensor, and the image sensor may include an image capturing device such as a camera.

[0075] Storage device 840 may include a computer-readable storage medium or computer-readable storage device 840. For example, storage device 840 may include a storage, magnetic hard disk, optical disk, flash memory device, etc.

[0076] The input device 850 receives user input from a user. For example, the input device 850 may include a keyboard, a mouse, a touch screen, a microphone, or any other device capable of detecting user input from a user and communicating the detected user input to the augmented reality providing device 800.

[0077] The output device 860 can provide the output of the augmented reality providing device 800 to the user through a visual, auditory, or tactile channel. The output device 860 can include a display, a touch screen, a speaker, a vibration generator, or any other device capable of providing output to the user. The display can visualize and show the augmented reality content.

[0078] The communication device 870 can communicate with external devices via a wired or wireless network.

[0079] The above-described embodiments may be implemented using hardware components, software components, or a combination of hardware and software components. For example, the devices and components described herein may be implemented using one or more general-purpose or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU), a microprocessor, or other devices that execute and respond to instructions. The processing device may execute an operating system (OS) and one or more software applications that run on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of software. For convenience of understanding, a processing device may be described as being a single device, but those skilled in the art will recognize that a processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing device may include multiple processors or a single processor and a single controller. Other processing configurations, such as parallel processors, are also possible.

[0080] Software includes computer programs, codes, instructions, or a combination of one or more thereof, which can configure a processing device to operate as desired or can independently or in combination instruct the processing device. The software and / or data can be permanently or temporarily embodied in any type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave to be interpreted by the processing device or to provide instructions or data to the processing device. The software can be distributed across computer systems coupled to a network and stored and executed in a distributed manner. The software and data can be stored on one or more computer-readable recording media.

[0081] The method according to the present invention may be embodied in the form of program instructions that can be executed by various computer means and recorded on a computer-readable recording medium. The recording medium may include program instructions, data files, data structures, and the like, alone or in combination. The recording medium and program instructions may be specially designed and constructed for the purposes of the present invention, or they may be well-known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language code, such as that produced by a compiler, but also high-level language code that is executed by a computer using an interpreter, for example.

[0082] The hardware devices described above may be configured to operate as one or more software modules to perform the operations described in this invention, and vice versa.

[0083] Although the embodiments have been described with limited drawings as mentioned above, those skilled in the art can apply various technical modifications and variations based on the above description. For example, the described techniques may be performed in a different order than the described method, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different manner than the described method, or may be replaced or substituted by other components or equivalents, and still achieve suitable results.

[0084] Therefore, the scope of the present invention should not be limited to the disclosed embodiments, but should be determined by the appended claims and their equivalents. [Explanation of symbols]

[0085] 110, 800: Augmented reality device 115: Display 310 710:Motion sensor 320, 720: Image sensor 330: Motion information determiner 340: Object feature point extractor 350: Reliability determiner 360: Pose Determiner 700: Pose determination device 730, 810: Processor 740, 820: Memory 830: Sensor 840: Storage device 850: Input device 860: Output device 870:Communication equipment 880:Communication bus

Claims

1. A pose determination method for an augmented reality providing device, comprising: estimating motion information of the augmented reality providing device using a motion sensor of the augmented reality providing device; determining first pose information of the augmented reality providing device based on the movement information; acquiring a current image frame through an image sensor of the augmented reality providing device; extracting feature points of an object from the current video frame; estimating second pose information of the augmented reality providing device based on the extracted feature points and feature point position information predefined for the object; determining a reliability value of each of the first pose information and the second pose information based on the movement information and the feature point extraction information; determining a pose of the augmented reality providing device based on the first pose information, the second pose information, and the reliability value; Including, The step of determining the confidence value comprises: determining the reliability value based on a difference between feature point positions of the object at the current time estimated based on the motion information and feature point positions of the object at the current time based on feature point extraction information; method.

2. The step of determining the confidence value comprises: If the difference satisfies a condition, determining a reliability value of the first pose information to be higher than a reliability value of the second pose information. The method of claim 1.

3. The step of determining the confidence value comprises: determining the reliability value based on a degree of movement of the augmented reality providing device according to the movement information; The method according to claim 1 or 2.

4. The step of determining the confidence value comprises: If the degree of movement satisfies a condition, determining a reliability value of the first pose information to be higher than a reliability value of the second pose information. The method of claim 3.

5. The step of determining the confidence value comprises: determining a reliability value of the second pose information to be higher than a reliability value of the first pose information when the object is located in the current video frame and the video quality of the current video frame satisfies a condition; 5. The method according to any one of claims 1 to 4.

6. The step of determining the confidence value comprises: determining the confidence value using a Kalman filter; 6. The method according to any one of claims 1 to 5.

7. The step of determining the confidence value comprises: determining a confidence value for each of the feature points of the object based on a speed of movement of each of the feature points of the object; 7. The method according to any one of claims 1 to 6.

8. The step of determining the confidence value comprises: and when at least one of the feature point extraction information and the movement information is updated, determining again a reliability value of at least one of the first pose information and the second pose information.

8. The method according to any one of claims 1 to 7.

9. The step of determining a pose of the augmented reality providing device includes: determining a pose of the augmented reality providing device by combining the first pose information and the second pose information based on respective reliability values ​​of the first pose information and the second pose information; 9. The method according to any one of claims 1 to 8.

10. the predefined feature point position information indicates three-dimensional coordinate information of a reference feature point located on a surface of the object when the object is in a reference pose; 10. The method according to any one of claims 1 to 9.

11. The step of determining the first pose information includes: determining the first pose information based on feature points of the object extracted from a previous video frame and the motion information; 11. The method according to any one of claims 1 to 10.

12. The step of estimating second pose information includes: estimating the second pose information based on positional differences between the extracted feature points and the reference feature points corresponding to each other; The method of claim 10.

13. The step of determining a pose of the augmented reality providing device includes: determining feature values ​​indicative of a position and orientation of the augmented reality providing device; 13. The method according to any one of claims 1 to 12.

14. 14. A method for implementing the method of any one of claims 1 to 13, comprising: storing one or more computer programs containing instructions for implementing the method of any one of claims 1 to 13; A computer-readable recording medium.

15. A pose determination device that determines a pose of an augmented reality providing device, a motion sensor for measuring motion information of the augmented reality providing device; an image sensor that captures at least a portion of the periphery of the augmented reality providing device and acquires a current image frame; a processor that determines a pose of the augmented reality providing device based on the measured movement information and the captured current video frame; The processor: determining first pose information of the augmented reality providing device based on the movement information; extracting feature points of an object from the current image frame; estimating second pose information of the augmented reality providing device based on the extracted feature points and feature point position information predefined for the object; determining a reliability value for each of the first pose information and the second pose information based on the movement information and the feature point extraction information; determining the pose of the augmented reality providing device based on the first pose information, the second pose information, and the reliability value; determining the reliability value based on a difference between the feature point positions of the object at the current time estimated based on the motion information and the feature point positions of the object at the current time based on the feature point extraction information; Device.

16. The processor: If the difference satisfies a condition, the reliability value of the first pose information is determined to be higher than the reliability value of the second pose information.

16. The apparatus of claim 15.

17. The processor: If a degree of movement of the augmented reality providing device according to the movement information satisfies a condition, a reliability value of the first pose information is determined to be higher than a reliability value of the second pose information.

17. Apparatus according to claim 15 or 16.

18. The processor: determining a reliability value of the second pose information to be higher than a reliability value of the first pose information when the object is located within the current video frame and the video quality of the current video frame satisfies a condition; 18. Apparatus according to any one of claims 15 to 17.

19. An augmented reality providing device, a motion sensor for measuring motion information of the augmented reality providing device; an image sensor that captures at least a portion of the periphery of the augmented reality providing device and acquires a current image frame; a processor that estimates a pose of the augmented reality providing device according to the motion information and generates augmented reality content that is matched to a real object according to the estimated pose; a display for visualizing the augmented reality content; The processor: determining first pose information of the augmented reality providing device based on the movement information; extracting feature points of the actual object from the current video frame; estimating second pose information of the augmented reality providing device based on the extracted feature points and feature point position information predefined for the real object; determining a reliability value for each of the first pose information and the second pose information based on the movement information and the feature point extraction information; determining the pose of the augmented reality providing device based on the first pose information, the second pose information, and the reliability value; determining the reliability value based on a difference between the feature point positions of the object at the current time estimated based on the motion information and the feature point positions of the object at the current time based on the feature point extraction information; Device.

Images