Method and apparatus for estimating posture

Abstract

Disclosed are a method and device for posture estimation, which includes: acquiring an original image before geometric correction from an image sensor; determining feature points in the original image; and estimating the posture based on the feature points.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Korean Patent Application No. 10-2019-0118690 filed on September 26, 2019, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety for all purposes by reference. Technical Field

[0003] The following description relates to methods and apparatus for estimating pose. Background Art

[0004] Camera pose estimation involves determining the translation and rotation of a dynamically changing camera viewpoint. The application of pose estimation is increasing and is being applied in many fields, such as simultaneous localization and mapping (SLAM), mixed reality, augmented reality, robotic navigation, and three-dimensional (3D) scene reconstruction. Summary of the Invention

[0005] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0006] In one general aspect, a method for estimating a pose is provided, the method comprising: acquiring, through an image sensor, an original image before geometric correction; determining feature points in the original image; and estimating the pose based on the feature points.

[0007] The raw image may include an image in which geometric distortion of a lens of the image sensor is not corrected.

[0008] Estimating the pose may include estimating the pose based on changes in feature points occurring due to geometric distortion of a lens of the image sensor.

[0009] The geometric distortion may include one of barrel distortion and pincushion distortion of a lens of the image sensor.

[0010] Acquiring the raw image may include acquiring the raw image before an image signal processor (ISP) that processes the image captured in the image sensor removes geometric distortion.

[0011] The method may include: acquiring a corrected image after geometric correction of the original image; and determining feature points in the corrected image, wherein estimating the pose may include: estimating the pose based on changes between the feature points of the original image and the feature points of the corrected image.

[0012] The image sensor may be a dual-pixel or higher pixel sensor, each pixel of the dual-pixel or higher pixel sensor including a photodiode.

[0013] The method may include determining a vector toward the focal plane based on a phase difference obtained from two photodiodes selected from photodiodes in each of the dual or more pixels, wherein estimating the pose may include estimating the pose based on the vector.

[0014] Estimating the pose may include determining an angle between the focal plane and the captured object based on the vector, and estimating the pose based on changes in the angle.

[0015] Phase difference may include light intensity differences based on the relative distance between the focal plane and the captured object.

[0016] Estimating the pose may include estimating the pose by applying a parameter of the image sensor related to the optical zoom to the vector.

[0017] The method may include detecting a plane in the original image based on the feature points.

[0018] Estimating the posture may include estimating the posture of the image sensor or a mobile terminal including the image sensor.

[0019] In another general aspect, a method for estimating a pose is provided, the method comprising: determining a vector toward a focal plane of a corresponding pixel based on a phase difference obtained from two photodiodes among photodiodes in each pixel selected from pixels of a dual-pixel or higher-pixel sensor; and estimating the pose by determining a relative movement change of the dual-pixel or higher-pixel sensor relative to the focal plane based on the vector.

[0020] Phase difference may include light intensity differences based on the relative distance between the focal plane and the captured object.

[0021] Estimating the pose may include estimating the pose by determining an angle between the focal plane and the captured object based on the vector and determining a change in relative movement based on a change in the angle.

[0022] A dual or higher pixel sensor may include pixels that each include a photodiode.

[0023] In another general aspect, an apparatus for estimating a pose is provided, the apparatus including: a processor configured to acquire, through an image sensor, an original image before geometric correction; determine feature points in the original image; and estimate the pose based on the feature points.

[0024] The processor may be configured to estimate the pose based on changes in the feature points that occur due to geometric distortion of a lens of the image sensor.

[0025] In another general aspect, an apparatus for estimating a pose is provided, the apparatus comprising: an image sensor configured to obtain an original image; and a processor configured to: select a first feature point from the original image, obtain a corrected image by correcting geometric distortion of a lens of the image sensor in the original image, select a second feature point from the corrected image, the second feature point corresponding to the first feature point, and estimate the pose based on a change between the first feature point and the second feature point.

[0026] The geometric distortion of the image sensor lens may not be corrected in the original image.

[0027] Other features and aspects will become apparent from the following detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 An example of a process performed in the posture estimation apparatus is shown.

[0029] Figure 2 An example of processing an image captured by a sensor is shown.

[0030] Figure 3 and Figure 4 An example of estimating a pose based on changes in feature points caused by geometric distortion is shown.

[0031] Figure 5 Examples of pose estimation and / or plane detection operations are shown.

[0032] Figures 6 to 8 An example is shown in which a posture is estimated using a phase difference acquired from two photodiodes selected from among a plurality of photodiodes in a pixel.

[0033] Figure 9 An example of plane detection and tracking operation is shown.

[0034] Figure 10 An example of a pose estimation process is shown.

[0035] Figure 11 An example of a process of estimating a pose using sensors is shown.

[0036] Figure 12 and Figure 13 An example of a pose estimation method is shown.

[0037] Figure 14 An example of a posture estimation device is shown.

[0038] Throughout the drawings and detailed description, unless otherwise described or provided, the same reference numerals should be understood to refer to the same elements, features, and structures. The drawings may not be drawn to scale, and the relative sizes, proportions, and depictions of elements in the drawings may be exaggerated for clarity, illustration, and convenience. DETAILED DESCRIPTION

[0039] The following detailed description is provided to help the reader obtain a comprehensive understanding of the methods, devices and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be apparent. For example, it will be clear that the order of operations described herein is merely an example and is not limited to the order of those operations set forth herein, but can be changed after understanding the disclosure of the present application, except for operations that must appear in a certain order. In addition, for greater clarity and brevity, descriptions of features known in the art may be omitted.

[0040] The features described herein may be implemented in different forms and are not to be construed as limited to the examples described herein. Rather, the examples described herein are provided merely to illustrate some of the many possible ways to implement the methods, devices, and / or systems described herein, which will become apparent upon understanding the disclosure of this application.

[0041] Although terms such as "first," "second," and "third" may be used herein to describe various components, assemblies, regions, layers, or portions, these components, components, regions, layers, or portions should not be limited by these terms. Instead, these terms are merely used to distinguish one component, component, region, layer, or portion from another component, component, region, layer, or portion. Thus, a first component, component, region, layer, or portion mentioned in the examples described herein may also be referred to as a second component, component, region, layer, or portion without departing from the teachings of the examples.

[0042] Throughout the specification, when an element such as a layer, a region, or a substrate is described as being “on,” “connected to,” or “coupled to” another element, it may be directly “on,” “connected to,” or “coupled to” the other element, or one or more other elements may be present therebetween. In contrast, when an element is described as being “directly on,” “directly connected to,” or “directly coupled to” another element, there may be no other elements present therebetween.

[0043] The terms used herein are only used to describe various examples and are not intended to limit the present disclosure. Unless the context clearly indicates otherwise, the articles "a", "an" and "the" are also intended to include plural forms. The terms "include", "comprising" and "having" indicate the presence of the recited features, numbers, operations, components, elements and / or combinations thereof, but do not preclude the presence or addition of one or more other features, numbers, operations, components, elements and / or combinations thereof.

[0044] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. Terms (e.g., those defined in commonly used dictionaries) will be interpreted to have the same meaning as they have in the context of the relevant art and should not be interpreted as having an ideal or overly formal meaning unless expressly defined as such herein.

[0045] Regarding the reference numerals assigned to the elements in the drawings, it should be noted that although the same elements are shown in different drawings, the same elements will be represented by the same reference numerals when possible. In addition, in the description of the embodiments, when it is considered that a detailed description of known related structures or functions will lead to an obscure interpretation of the present disclosure, such description will be omitted.

[0046] Figure 1 An example of a process performed in the posture estimation apparatus is shown.

[0047] Image sensor 110 generates a raw image by capturing a scene within a field of view (FoV). The raw image may have geometric distortion due to the lens of image sensor 110, and therefore may require correction in image signal processor (ISP) 120 to resemble a normal image perceived by the human eye. The image corrected to resemble a normal image by ISP 120 is output as, for example, a preview image of image sensor 110 or a final captured image of image sensor 110.

[0048] The pose estimation device estimates the pose of image sensor 110 using geometric distortion in the original image. Correcting geometric distortion in ISP 120 is called geometric correction. Pose estimation is performed using the original image obtained before geometric correction. In some examples, the pose estimation device estimates the pose using a corrected image obtained after geometric correction, or estimates the pose using both the image obtained before and after geometric correction.

[0049] In an example, the posture estimation device estimates the posture based on the phase difference acquired from the pixels of the image sensor 110. In an example, the image sensor 110 is a dual-pixel or higher-pixel sensor including pixels each including multiple photodiodes. In a dual-pixel or higher-pixel sensor, each pixel includes two or more photodiodes.

[0050] The pose estimation device estimates a pose of six degrees of freedom (6DOF). In an example, 6DOF includes three-dimensional (3D) translation and 3D rotation information.

[0051] In one example, the pose estimation apparatus is a terminal including the image sensor 110, and is, for example, any terminal such as a mobile device, a smart phone, a wearable smart device (e.g., a ring, a watch, a pair of glasses, a glasses-like device, a wristband, an ankle brace, a belt, a necklace, an earring, a headband, a helmet, a device embedded in clothing, or an eyeglass display (EGD)), a computing device (e.g., a server, a laptop, a notebook computer, a small notebook computer, a netbook, an ultra-portable PC (UMPC), a tablet personal computer (tablet computer), a tablet phone, a mobile internet device (MID), a personal digital assistant (PDA), an enterprise digital assistant (EDA), an ultra-portable personal computer (UMPC), a portable laboratory desktop PC), an electronic product (e.g., a computer robots, digital cameras, digital video cameras, portable game consoles, MP3 players, portable / personal multimedia players (PMPs), handheld e-books, global positioning system (GPS) navigation, personal navigation devices, portable navigation devices (PNDs), handheld game consoles, e-books, televisions (TVs), high-definition televisions (HDTVs), smart TVs, smart devices, smart vacuum cleaners, smart home devices or security devices for access control, walking assistance devices, kiosks, robots, indoor autonomous robots, outdoor delivery robots, underwater and underground exploration robots, various Internet of Things (IoT) devices, self-driving cars, automatic or autonomous driving systems, smart vehicles, unmanned aerial vehicles, advanced driver assistance systems (ADAS), heads-up displays (HUDs) and augmented reality heads-up displays (AR HUDs), or any other device capable of wireless or network communication with the devices disclosed herein.

[0052] In this example, pose estimation 130 is performed by a processor (e.g., a central processing unit (CPU), a processor core, a multi-core processor, a reconfigurable processor, a multi-core processor, a multiprocessor, an application specific integrated circuit (ASIC) and a field programmable gate array (FPGA), a graphics processing unit (GPU), an application processor (AP), or any other type of multi-processor or single-processor structure included in a pose estimation apparatus). Additional details regarding the processor are provided below.

[0053] In this example, the pose estimation apparatus is a device other than the mobile terminal including the image sensor 110 or the image sensor 110, and is a computing device, a remote server, or the like with high-performance processing capabilities. In this example, pose estimation 130 is performed by the computing device, the remote server, or the like that receives pixel phase differences and / or raw images acquired by the image sensor 110. For ease of description, the following description will be made with respect to an example in which the pose estimation apparatus is a mobile terminal including the image sensor 110.

[0054] Figure 2 An example of processing an image captured in a sensor is shown.

[0055] Figure 2 The figure shows the normal path and accelerated path for processing raw images in the image signal processor. The normal path is a process in which the raw image captured by the sensor is corrected to an image similar to a normal image perceived by the human eye using various correction techniques of the image signal processor. The corrected image similar to the normal image is output as, for example, a preview image or the final captured image. The accelerated path does not employ the image signal processor's correction techniques. In the accelerated path, pose estimation is performed using the raw image captured by the sensor and having geometric distortion caused by the lens. In this way, pose estimation is performed using information present in the uncorrected raw image in the image processor.

[0056] Figure 3 and Figure 4 An example of a process of estimating a pose based on changes in feature points due to geometric distortion is shown.

[0057] Figure 3 Examples of no distortion, barrel distortion, and pincushion distortion are shown.

[0058] The original image captured by an image sensor has geometric distortion caused by the image sensor's lens. Geometric distortion occurs because the lens diffracts light incident on the image sensor. Furthermore, the closer to the edge of the original image, the greater the degree of geometric distortion. This geometric distortion varies depending on the characteristics of the lens.

[0059] Barrel distortion occurs when straight lines bend inward in the shape of a barrel. Common in wide-angle lenses, barrel distortion occurs because the field of view of the lens is much larger than the size of the image sensor and therefore needs to be "compressed" to fit. In barrel distortion, the image magnification decreases as the distance from the optical axis increases. Barrel distortion appears as a shape that bends like a circle near the edges. Figure 3As shown in , it can be seen that the horizontal line in the middle of the original image with barrel distortion is a straight line, while the other horizontal lines bend like circles as they approach the upper or lower edge. Similarly, it can be seen that the vertical lines bend like circles as they approach the left or right edge.

[0060] Pincushion distortion is a lens effect that causes the image to appear constricted in the middle, like the effect on a pillow when a needle is inserted into it. In pincushion distortion, the image magnification increases with distance from the optical axis. Common in telephoto lenses, pincushion distortion appears as a curved shape in the opposite direction to barrel distortion. Figure 3 As shown in , it can be seen that the horizontal line in the middle of the original image with pincushion distortion is a straight line, while the other horizontal lines are concave toward the center of the image as they approach the upper or lower edge. Similarly, it can be seen that the vertical lines are concave toward the center of the image as they approach the left or right edge.

[0061] Such geometric distortion characteristics are used to estimate the pose, and this will be described with reference to the following drawings.

[0062] Figure 4 An example is shown for explaining a process of estimating a pose based on changes in feature points due to geometric distortion. Figure 4 In the example of , it is assumed that the image sensor has a rotational movement followed by a translation and barrel distortion occurs in the original image. Figure 4 , it is assumed that the feature points determined for the reference line are not displayed separately, but the feature points for detecting the translation and / or rotation of the image sensor are appropriately determined.

[0063] This example describes changes in feature points due to translation / rotation in a corrected image after geometric correction. The corrected image is an image in which geometric distortion has been corrected by an image signal processor. Because changes in the position of reference lines due to image sensor movement can be detected in the corrected image, image sensor movement can be estimated. In contrast, changes in reference lines due to image sensor rotation are not detected in the corrected image. Therefore, the corrected image is not used to estimate image sensor rotation.

[0064] In another example, feature point changes due to translation / rotation in a raw image before geometric correction are described. The raw image is obtained before geometric distortion is corrected in the image signal processor. Geometric distortion may occur in the reference line when a reference line located in the middle of the raw image is moved to the edge of the raw image due to image sensor movement. Furthermore, geometric distortion may occur in the reference line when the position of the reference line in the raw image changes due to image sensor rotation. Consequently, the reference line's movement appears differently in the raw image, which can be used as information useful for pose estimation.

[0065] In other words, the changes in feature points extracted from the original image can be used as additional movement information that may not be obtained from a typical rectified image. Therefore, the accuracy of pose estimation can be improved using a small number of feature points.

[0066] Figure 5 Examples of pose estimation and / or plane detection operations are shown.

[0067] Figure 5 An example of a process for performing pose estimation and / or plane detection using images obtained before geometric correction and images obtained after geometric correction is shown.

[0068] In operation 510 , geometric distortion in the original image is removed through geometric correction (GC), thereby generating a corrected image.

[0069] In operation 520, shape and / or surface differences between the original image and the corrected image are determined. For example, the same object may have different shapes and / or surfaces in the original image and the corrected image based on the presence or absence of geometric correction. Furthermore, due to translation and / or rotation of the image sensor, changes in feature points of the corresponding object in the original image and the corrected image may differ from each other.

[0070] In operation 530, a posture is estimated based on the difference sensed in operation 520. In addition, a plane in the image is detected based on the difference sensed in operation 520. The plane in the image is used, for example, in the case of enhancing a virtual object in augmented reality (AR).

[0071] In addition, since operations described with reference to other drawings are also applicable to plane detection and / or tracking, repeated descriptions will be omitted.

[0072] Figures 6 to 8 An example of a process of estimating a posture using a phase difference acquired from two photodiodes selected from among a plurality of photodiodes in a pixel is shown.

[0073] The image sensor may be a dual-pixel or higher-pixel sensor 610 . Figure 6 An example of the structure of a dual-pixel or more pixel sensor 610 is shown. In the example, the dual-pixel or more pixel sensor 610 is a single camera including pixels each including a plurality of photodiodes. Such a single camera has an advantage in fast focusing. Figure 6 In the example shown in FIG, photodiode A and photodiode B are two photodiodes selected from the plurality of photodiodes included in each pixel. Each of photodiode A and photodiode B independently receives light energy and outputs electrical energy (e.g., DP data). When capturing an image, the plurality of photodiodes included in each pixel or two photodiodes A and photodiode B selected from the plurality of photodiodes included in each pixel can be used together to capture the image.

[0074] Figure 7a and 7b An example of a process of capturing a raw image in a dual-pixel or higher-pixel sensor is shown.

[0075] In this example, the position of the focal plane is determined based on the distance between the lens and the sensor. The lens' characteristics are also taken into account. Objects within the focal plane are collected at a single point on the sensor, resulting in no blur. However, objects outside the focal plane are distributed across the sensor, potentially causing blur.

[0076] refer to Figure 7a , since an object at a distance z1 from the lens is not located on the focal plane, it can be seen from the DP data output by two photodiodes selected from the plurality of photodiodes that parallax d and blur b exist in the image generated during capture. In another example, since an object at a distance g1 from the lens is located on the focal plane, there is no parallax or blur.

[0077] refer to Figure 7b , since an object located at a distance z2 from the lens is not located on the focal plane, it can be seen from the DP data output by the two selected photodiodes that parallax d and blur b exist in the image generated during capture. In another example, since an object located at a distance g2 from the lens is located on the focal plane, there is no parallax or blur.

[0078] exist Figure 7a and Figure 7b In the example shown in FIG, even if the object is located at a different position, the same parallax and blur are generated. In this regard, although it is difficult to obtain the absolute depth information of the object based on the phase difference obtained from two selected photodiodes in each pixel, the relative depth information about the object relative to the reference plane can be known. Thus, the movement change of a short period of time is detected within a short distance and used for posture estimation. Figure 8Further detailed description is given.

[0079] Figure 8 An example of a process of determining information about a change in relative movement with respect to a focal plane is shown.

[0080] For ease of description, it is assumed that a dual-pixel or higher pixel sensor that captures a static object is translated (moved) and / or rotated. In two photodiodes A and photodiode B selected from a plurality of photodiodes included in each pixel of the dual-pixel or higher pixel sensor, a light intensity difference based on the relative distance between the object and the focal plane is generated. For example, a light intensity difference is generated in a pixel that senses the left end of the object, and the light intensity difference indicates that photodiode A has a higher light intensity than the light intensity of photodiode B. In addition, a light intensity difference is generated in a pixel that senses the right end of the object, and the light intensity difference indicates that photodiode B has a higher light intensity than the light intensity of photodiode A. Although for ease of description, Figure 8 The light intensity difference between pixels sensing the two ends of the object is shown, but a light intensity difference can also be generated for pixels sensing the middle portion of the object. Based on the light intensity differences generated between the pixels, a vector from the object to the focal plane is determined for each pair of pixels. Using these vectors, the angle φ1 between the object and the focal plane is determined.

[0081] Furthermore, the angle between the object and the focal plane can be changed based on translation and / or rotation of the dual-pixel or multi-pixel sensor. Similarly, based on the light intensity difference generated by two selected photodiodes in each pixel, a vector from the object to the focal plane can be determined for each pair of pixels. In this example, the angle φ2 between the object and the focal plane can be determined using this determined vector.

[0082] In an example, the pose of the dual pixel or higher pixel sensor is estimated by determining a motion vector representing relative movement of an object in an image based on vectors determined before and / or after translation and / or rotation of the dual pixel or higher pixel sensor.

[0083] This light intensity difference is also known as "phase difference." Phase difference images function as if they were captured using multiple narrow-baseline stereo cameras. While extracting meaningful information from long distances using this phase difference is difficult, it can detect changes in motion relative to the focal plane at short distances. This allows for functionality similar to that of an inertial measurement unit (IMU), without the accumulated errors.

[0084] The vector from the object to the focal plane generated for each pair of pixels based on the phase difference acquired from each of the pixels as described above is converted into a movement vector of the world coordinate system according to the following Equation 1.

[0085] [Equation 1]

[0086]

[0087] In Equation 1, represents the relative motion vector of the nth pixel relative to the focal plane, represents the change in coordinate system from the previous focal plane to the new focal plane due to optical zoom ("1" if there is no optical zoom), represents the transformation from the focal plane coordinate system to the world coordinate system, and Represents the movement vector of the nth pixel in the world coordinate system.

[0088] In this way, based on phase difference information acquired at different times, relative movement information indicating whether movement toward or away from the focal plane or rotation relative to the focal plane is performed is acquired, thereby recognizing movement changes at high speed and performing processing accordingly.

[0089] Figure 9 An example of plane detection and tracking operation is shown.

[0090] Figure 9 An example of performing optical zoom while performing plane detection and tracking using a dual-pixel or higher-pixel sensor is shown. Figure 9 The operations in the description may be performed in the order and manner shown, but the order of some of the operations may be changed or some of the operations may be omitted without departing from the spirit and scope of the described illustrative examples. Figure 9 Many of the operations shown in may be performed in parallel or concurrently. Figure 9 One or more blocks and combinations of blocks may be implemented by a computer based on dedicated hardware and a device (such as a processor) that performs specified functions, or a combination of dedicated hardware and computer instructions. Figure 9 In addition to the description, Figures 1-8 The description also applies to Figure 9 , and is incorporated herein by reference. Therefore, the above description may not be repeated here.

[0091] In operation 910 , a plurality of points included in a single target in an image are selected as piece-wise targets. For example, predetermined selected points on a table object included in an image are piece-wise targets.

[0092] In operation 920, phase differences are determined for multiple segmented targets, obtained from two photodiodes selected from the plurality of photodiodes included in each pixel. In this case, a vector from the target to the focal plane is determined for each pixel. For example, when the vectors of at least three targets are on the same plane, the target is identified as a single plane. When there are multiple targets, the target is identified as a more complex shape.

[0093] In operation 930, the optical zoom is controlled based on the motion vector and the blur size. For example, when the change amount is slightly sensed due to a small motion vector and / or blur size, the optical zoom is controlled to increase. After controlling the optical zoom, operation 910 is repeated.

[0094] In operation 950, a plane is detected and / or tracked based on the phase difference of operation 920. In this case, movement of the plane relative to the camera position may be tracked.

[0095] In operation 940, an IMU-based attitude is additionally estimated. For example, values ​​of additional sensors such as a gyroscope sensor and an accelerometer are used to estimate the attitude. Furthermore, information about the estimated attitude may be additionally used for plane detection and / or tracking in operation 950.

[0096] In addition, since operations described with reference to other drawings are also applicable to plane detection and / or tracking, repeated descriptions will be omitted.

[0097] Figure 10 An example of a pose estimation process is shown. Figure 10 The operations in the description may be performed in the order and manner shown, but the order of some of the operations may be changed or some of the operations may be omitted without departing from the spirit and scope of the described illustrative examples. Figure 10 Many of the operations shown in may be performed in parallel or concurrently. Figure 10 One or more blocks and combinations of blocks may be implemented by a computer based on dedicated hardware and a device (such as a processor) that performs specified functions, or a combination of dedicated hardware and computer instructions. Figure 10 In addition to the description, Figures 1-9 The description also applies to Figure 10 , and is incorporated herein by reference. Therefore, the above description may not be repeated here.

[0098] Figure 10 An example is shown for explaining a posture estimation process performed by a processor in the posture estimation apparatus.

[0099] In operation 1010, a raw image is acquired from an image sensor. The raw image is an image obtained before geometric correction is performed in an image signal processor.

[0100] In operation 1021, geometric correction is performed on the original image to remove geometric distortion included in the original image. Geometric distortion occurs due to the lens of the image sensor and includes, for example, barrel distortion and / or pincushion distortion. The image obtained by removing the geometric distortion of the original image is called a corrected image.

[0101] In operation 1022, feature points are selected from the corrected image. For example, the feature points are used to perform SLAM optimization, which can be selected sparsely or densely depending on the computational requirements. In this example, feature points are selected as random points, edges, and textures that are easy to identify and match. In addition, feature points are also selected from the original image acquired from the image sensor.

[0102] In operation 1023, the movement of the feature points in the corrected image is determined. In one example, the movement of the feature points in the corrected image is determined by comparing the corrected image corresponding to the current time point t with the corrected image corresponding to the previous time point t-1. In one example, the final movement is determined by accumulating the movement determined for each time point. In another example, the movement of the feature points in the corrected image is determined by comparing the corrected image corresponding to the current time point with the corrected image corresponding to a reference time point.

[0103] In operation 1024, feature point movement in the original image is determined. Unlike the corrected image, feature point movement in the original image is determined based on geometric distortion caused by the image sensor lens. For example, whether the geometric distortion is barrel distortion or pincushion distortion, and the degree of distortion of objects located at the edge of the image are considered.

[0104] As with the corrected image, the movement of feature points in the original image is determined by comparing the original image corresponding to the current time point t with the original image corresponding to the previous time point t-1. The final movement is determined by accumulating the movement determined for each time point. In another example, the movement of feature points in the original image is determined once by comparing the original image corresponding to the current time point with the original image corresponding to the reference time point.

[0105] In operation 1025, a motion vector is generated based on the determined feature point movement and converted into a world coordinate system. For example, the motion vector is determined based on a camera coordinate system centered on the image sensor and the coordinate system of the motion vector is converted from the camera coordinate system to the world coordinate system.

[0106] In operation 1031 , the image sensor is a dual-pixel or greater sensor including pixels each including a plurality of photodiodes, and a phase difference is detected from two photodiodes selected from among the plurality of photodiodes in each of the pixels.

[0107] In operation 1032 , a vector toward a focal plane is determined using the phase difference detected for each of the pixels.

[0108] In operation 1033, a motion vector indicating relative motion is determined based on the vector of the focal plane. Since the motion vector is based on the camera coordinate system, the coordinate system of the motion vector is converted from the camera coordinate system to the world coordinate system.

[0109] In operation 1040, a detection and / or tracking mode is selected. Detection mode uses all or at least a threshold proportion of the feature points detected in the image. In contrast, tracking mode uses a portion (e.g., less than a threshold proportion) of the feature points detected in the image. For example, considering the output values ​​of another sensor such as an IMU, it may be possible to confirm that the pose has changed slightly. If it is confirmed in operation 1031 that only some pixels have changed in phase difference, tracking mode is selected, thereby estimating the pose based on the phase difference of the corresponding pixels. In some cases, operation 1040 may be omitted.

[0110] In operation 1050 , a pose is estimated based on a phase difference and / or a change in movement of feature points in the original image and / or the corrected image sensed by the dual-pixel or greater pixel sensor.

[0111] because Figures 1 to 9 The description also applies to Figure 10 , and therefore repeated descriptions will be omitted.

[0112] Figure 11 An example of a process for estimating pose using multiple sensors is shown.

[0113] Figure 11 An example is shown to illustrate the process of estimating a pose using a first sensor 1110 and a second sensor 1120. In this example, the first sensor 1110 and the second sensor 1120 are different from each other and are, for example, an RGB sensor and a depth sensor, a wide-angle sensor and a telephoto sensor, and cameras with different fields of view. Because the first sensor 1110 and the second sensor 1120 are different sensor types, the characteristics of the geometric distortion occurring therein are also different. By utilizing this difference in the characteristics of geometric distortion, additional feature points that cannot be extracted by typical sensors are considered, which can improve the efficiency of pose estimation 1130. Furthermore, the first sensor 1110 and the second sensor 1120 are different sensor types, such as an image sensor and an IMU. Pose estimation 1130 is also performed by comprehensively considering changes in the movement of feature points in the image acquired by the image sensor and the movement information sensed by the IMU.

[0114] Figure 12 and Figure 13 An example of a pose estimation method is shown. Figure 12 and Figure 13The operations in the description may be performed in the order and manner shown, but the order of some of the operations may be changed or some of the operations may be omitted without departing from the spirit and scope of the described illustrative examples. Figure 12 and Figure 13 Many of the operations shown in may be performed in parallel or concurrently. Figure 12 and Figure 13 One or more blocks and combinations of blocks may be implemented by a computer based on dedicated hardware and a device (such as a processor) that performs specified functions, or a combination of dedicated hardware and computer instructions. Figure 12 and Figure 13 In addition to the description, Figures 1-11 The description also applies to Figure 12 and Figure 13 , and is incorporated herein by reference. Therefore, the above description may not be repeated here.

[0115] Figure 12 and Figure 13 An example of a posture estimation method executed by a processor included in a posture estimation apparatus is shown.

[0116] In operation 1210, a pose estimation apparatus obtains a raw image from an image sensor before geometric correction. The raw image is an image in which geometric distortion caused by a lens of the image sensor has not been corrected. In an example, the pose estimation apparatus obtains the raw image before an image signal processor that processes the image captured by the image sensor removes the geometric distortion.

[0117] In operation 1220 , the pose estimation apparatus determines feature points in the original image.

[0118] In operation 1230, the pose estimation apparatus estimates the pose based on the feature points. In an example, the pose estimation apparatus estimates the pose according to changes in the feature points that occur based on geometric distortion caused by a lens of the image sensor.

[0119] In operation 1310, a pose estimation apparatus determines a vector toward a focal plane for a corresponding pixel based on a phase difference obtained from two photodiodes selected from a plurality of photodiodes in each pixel of a dual-pixel or multi-pixel sensor. In an example, the phase difference is a light intensity difference based on the relative distance between a captured object and the focal plane. The dual-pixel or multi-pixel sensor includes pixels each including a plurality of photodiodes.

[0120] In operation 1320, the pose estimation apparatus estimates the pose by determining a relative movement change relative to the focal plane based on the vector. The pose estimation apparatus may estimate the pose by determining an angle between the captured object and the focal plane based on the vector and determining a relative movement change based on a change in the angle.

[0121] Figure 14 An example of a posture estimation device is shown.

[0122] refer to Figure 14 The posture estimation apparatus 1400 includes a memory 1410 , a processor 1420 , and an input and output (I / O) interface 1430 . The memory 1410 , the processor 1420 , and the I / O interface 1430 communicate with each other through a bus 1440 .

[0123] Memory 1410 includes computer-readable instructions; processor 1420 performs the above-described operations in response to the instructions stored in memory 1410 being executed by processor 1420. Memory 1410 is a volatile memory or a non-volatile memory. Memory 1410 includes a large-capacity storage medium, such as a hard disk, for storing various data. Further details regarding memory 1410 are provided below.

[0124] Processor 1420 is a device for executing instructions or programs or controlling attitude estimation apparatus 1400. For example, the desired operations include instructions or code included in the program. For example, a data processing device implemented by hardware includes a microprocessor, a central processing unit (CPU), a processor core, a multi-core processor, a reconfigurable processor, a multi-core processor, a multiprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a graphics processor unit (GPU), or any other type of multi-processor or single-processor architecture. Further details regarding processor 1420 are provided below.

[0125] In an example, the posture estimation apparatus 1400 uses an I / O interface 1430 connected to another component in the posture estimation apparatus 1400 (e.g., an image sensor) to connect to an external device and exchange data. In an example, the posture estimation apparatus 1400 visually presents an image with a corrected posture on the I / O interface 1430. In an example, the I / O interface 1430 may be a display that receives input from a user or provides output. In an example, the I / O interface 1430 may serve as an input device and receive input from the user through traditional input methods (e.g., a keyboard and mouse) and new input methods (e.g., touch input, voice input, and image input). Therefore, the I / O interface 1430 may include, for example, a keyboard, a mouse, a touch screen, a microphone, and other devices that can detect input from the user and transmit the detected input to the posture estimation apparatus 1400.

[0126] In an example, the I / O interface 1430 can be used as an output device and provide the output of the posture estimation apparatus 1400 to the user through visual, auditory, or tactile channels. The I / O interface 1430 may include, for example, a display, a touch screen, a speaker, a vibration generator, and other devices that can provide output to the user.

[0127] However, the I / O interface 1430 is not limited to the above examples, and any other display operably connected to the pose estimation device 1400, such as a head-up display (HUD), an augmented reality head-up display (AR 3D HUD), and an eyewear display (EGD), as well as a computer monitor, may be used without departing from the spirit and scope of the described exemplary embodiments. In an example, the I / O interface 1430 is a physical structure including one or more hardware components that provide the ability to render a user interface, render a display, and / or receive user input.

[0128] Processor 1420 acquires a raw image from the image sensor before geometric correction, identifies feature points in the raw image, and estimates the pose based on the feature points. Furthermore, based on phase differences obtained from two photodiodes selected from a plurality of photodiodes in each pixel of a dual-pixel or multi-pixel sensor, processor 1420 determines a vector pointing toward the focal plane of the corresponding pixel and a relative motion change relative to the focal plane based on the vector, thereby estimating the pose.

[0129] Using additional motion information not obtained from typical geometrically corrected images, it is possible to stably estimate position / orientation using a small number of feature points and reduce the data acquisition time required for signal processing, thereby achieving fast signal processing.

[0130] Furthermore, it is possible to estimate a pose using a minimum amount of input information by extracting a relatively large amount of information from an original image without applying human visual recognition to a pose estimation device for data processing.

[0131] By obtaining additional motion information not obtained from typical images from the phase difference between pixels before geometric correction and / or original images, the effect of estimating pose using cameras with different characteristics can be achieved using only a single camera.

[0132] In addition, the posture estimation device 1400 may be considered to process the above operations.

[0133] The devices, units, modules, equipment and other components described herein are implemented by hardware components. Examples of hardware components that can be used to perform the operations described in this application include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application, where appropriate. In other examples, one or more hardware components in the hardware components for performing the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer can be implemented by one or more processing elements (e.g., logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors, or any other device or combination of devices configured to respond and execute instructions in a defined manner to achieve desired results). In one example, a processor or computer includes (or is connected to) one or more memories storing instructions or software executed by the processor or computer. Hardware components implemented by a processor or computer can execute instructions or software, such as an operating system (OS) and one or more software applications running on the OS, to perform the operations described in this application. The hardware components can also access, manipulate, process, create, and store data in response to the execution of instructions or software. For the sake of brevity, the singular term "processor" or "computer" may be used in the description of the examples described in this application, but in other examples, multiple processors or computers may be used, or the processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors or another processor and another controller. One or more processors or a processor and a controller may implement a single hardware component or two or more hardware components. The hardware components may have any one or more of different processing configurations, examples of which include a single processor, independent processors, parallel processors, single instruction single data (SISD) multiprocessing, single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0134] The method for performing the operations described in this application is performed by computing hardware, for example, by one or more processors or computers implemented as described above that execute instructions or software to perform the operations performed by these methods described in this application. For example, a single operation or two or more operations can be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations can be performed by one or more processors or a processor and a controller, and one or more other operations can be performed by one or more other processors or another processor and another controller. One or more processors or a processor and a controller can perform a single operation or two or more operations.

[0135] The instructions or software for controlling a processor or computer to implement the hardware components and perform the methods described above are written as a computer program, code segments, instructions, or any combination thereof, which, individually or collectively, instruct or configure the processor or computer to operate as a machine or special-purpose computer to perform the operations performed by the hardware components and the methods described above. In one example, the instructions or software include at least one of an applet, a dynamic link library (DLL), middleware, firmware, a device driver, or an application storing the method for estimating pose. In one example, the instructions or software include machine code directly executed by the processor or computer, such as machine code generated by a compiler. In another example, the instructions or software include high-level code executed by the processor or computer using an interpreter. A programmer of ordinary skill in the art can readily write the instructions or software based on the block diagrams and flow charts shown in the accompanying drawings and the corresponding description in the specification, which disclose algorithms for performing the operations performed by the hardware components and the methods described above.

[0136] The instructions or software that controls computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, as well as any associated data, data files, and data structures, can be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random-access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random-access memory (DRAM), static random-access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage device, hard disk drive (HDD), solid state drive (SSD), flash memory, card type memory (such as, multimedia card or micro card (for example, secure digital (SD) card or extreme digital (XD) card)), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, and any other device configured as follows: to store instructions or software and any associated data, data files and data structures in a non-transitory manner, and to provide instructions or software and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the instructions. In one example, the instructions or software, and any associated data, data files and data structures are distributed on a network connected computer system so that the instructions and software and any associated data, data files and data structures are stored, accessed and executed by one or more processors or computers in a distributed manner.

[0137] Although this disclosure includes specific examples, it will be apparent after an understanding of the disclosure of this application that various changes in form and detail can be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered merely descriptive and not for purposes of limitation. The description of features or aspects in each example is considered to be applicable to similar features or aspects in other examples. Suitable results can be achieved if the described techniques are performed in a different order and / or if components in the described systems, architectures, devices, or circuits are combined in a different manner and / or replaced or supplemented by other components or their equivalents. Therefore, the scope of the disclosure is not limited by the detailed description, but by the claims and their equivalents, and all changes within the scope of the claims and their equivalents are interpreted as being included in this disclosure.

Claims

1. A method for estimating a posture, the method comprising: Acquire the original image before geometric correction through the image sensor; Determining feature points in the original image; as well as estimating the pose of the image sensor, Wherein, estimating the posture includes: estimating the posture based on the movement of feature points in the original image due to geometric distortion of the lens of the image sensor, wherein the movement of the feature points in the original image is determined by comparing the original image corresponding to the current time point with the original image corresponding to the previous time point.

2. The method according to claim 1, wherein The original image includes an image in which geometric distortion of a lens of the image sensor is not corrected.

3. The method according to claim 1, wherein The geometric distortion includes one of barrel distortion and pincushion distortion of a lens of the image sensor.

4. The method according to claim 1, wherein Acquiring the original image includes: Before an image signal processor (ISP) processes the image captured by the image sensor to remove geometric distortion, an original image is acquired.

5. The method according to claim 1, wherein The image sensor is a dual-pixel or higher pixel sensor, each pixel of the dual-pixel or higher pixel sensor includes a photodiode.

6. The method according to claim 5, further comprising: determining a vector toward the focal plane based on a phase difference obtained from two photodiodes selected from among the photodiodes in each pixel of the dual pixel or more pixels, Wherein, estimating the posture further comprises: The pose is estimated based on the vector.

7. The method according to claim 6, wherein: Estimating the pose further comprises: An angle between the focal plane and the captured object is determined based on the vector, and the pose is estimated based on a change in the angle.

8. The method according to claim 6, wherein: The phase difference includes a light intensity difference based on a relative distance between the focal plane and a captured object.

9. The method according to claim 6, wherein: Estimating the pose further comprises: The pose is estimated by applying optical zoom-related parameters of the image sensor to the vector.

10. The method according to claim 1, further comprising: A plane in the original image is detected based on the feature points.

11. The method according to claim 1, wherein Estimating the pose further comprises: A posture of a mobile terminal including the image sensor is estimated.

12. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method according to claim 1.

13. A method for estimating a posture, the method comprising: Acquire the original image before geometric correction through the image sensor; Determining feature points in the original image; Acquire a corrected image after geometric correction is performed on the original image; determining feature points in the corrected image; as well as The posture is estimated based on a change between feature point movements of the original image and feature point movements of the corrected image, wherein the feature point movements in the original image are determined by comparing the original image corresponding to a current time point with the original image corresponding to a previous time point, and the feature point movements in the corrected image are determined by comparing the corrected image corresponding to the current time point with the corrected image corresponding to the previous time point.

14. The method according to claim 13, wherein The original image includes an image in which geometric distortion of a lens of the image sensor is not corrected.

15. The method according to claim 14, wherein The geometric distortion includes one of barrel distortion and pincushion distortion of a lens of the image sensor.

16. The method according to claim 13, wherein: Acquiring the original image includes: Before an image signal processor (ISP) processes the image captured by the image sensor to remove geometric distortion, an original image is acquired.

17. A device for estimating posture, the device comprising: The processor is configured to: Acquire the original image before geometric correction through the image sensor; Determining feature points in the original image; as well as estimating the pose of the image sensor, The processor is further configured to estimate the posture based on a movement of feature points in the original image due to geometric distortion of a lens of the image sensor, wherein the movement of feature points in the original image is determined by comparing the original image corresponding to a current time point with the original image corresponding to a previous time point.

Images