Positioning method and device based on image and map data

Abstract

To provide a positioning method based on an image and map data.SOLUTION: A positioning method comprises: extracting a feature of an object from an input image; generating an image in which the object is projected from positioning of a device on the basis of map data; and evaluating the positioning on the basis of a feature value corresponding to an apex contained in the projection image.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a method and apparatus for positioning based on image and map data. [Background technology]

[0002] Various types of augmented reality services are being provided in various fields, such as driving assistance for vehicles and other means of transport, games, and entertainment. Various localization methods can be used to provide a more accurate sense of augmented reality. For example, a sensor-based localization method may use a combination of various sensors, such as a Global Positioning System (GPS) sensor and an Inertial Measurement Unit (IMU) sensor, to grasp the position and direction of an object. There is also a vision-based localization method that uses camera information. Summary of the Invention [Problem to be solved by the invention]

[0003] The embodiments described below provide a method for positioning based on video and map data. The embodiments provide a technique that allows positioning without establishing correspondences between vertices of the video and vertices of the map data. The embodiments also provide a technique that allows positioning without parameterizing features of the video, extracting relationships that are invariant to 3D transformations and perspective transformations, and easily identifying such invariant relationships when searching the map data. [Means for solving the problem]

[0004] A positioning method according to one aspect includes the steps of generating a first image of an object from an input image, generating a second image that projects the object based on map data including a position of the object using a candidate positioning of a device, pooling feature values ​​from the first image that correspond to vertices in the second image, and determining a score for the candidate positioning based on the pooled feature values.

[0005] The step of generating the first image may include generating a plurality of feature maps corresponding to a plurality of features.

[0006] The step of generating the second image may include the steps of extracting an area corresponding to a field of view range of the candidate positioning from the map data, and projecting vertices included in the area onto projection points corresponding to the candidate positioning.

[0007] The step of pooling the feature values ​​may include selecting pixels in the first image based on the coordinates of the vertices, and obtaining feature values ​​of the selected pixels.

[0008] Determining a score for the candidate positioning may include summing the pooled feature values.

[0009] The summing step may include, when the first image includes a plurality of feature maps corresponding to a plurality of features, summing the feature values ​​based on weights determined corresponding to the features.

[0010] The positioning method may further comprise determining a positioning for the device based on the scores of the candidate positions.

[0011] Determining a position measurement of the device may include determining a candidate position measurement corresponding to a maximum score among scores of a plurality of candidate position measurements as the position measurement of the device.

[0012] The step of determining the positioning of the device may include a step of dividing the second image into a plurality of regions, and a step of sequentially determining values ​​of a plurality of degrees of freedom included in the candidate positioning by using scores calculated for the divided regions.

[0013] The plurality of degrees of freedom values ​​may include three translational degrees of freedom values ​​and three rotational degrees of freedom values.

[0014] The dividing step may include dividing the second image into a far distance area and a near distance area based on a first criterion related to distance, and dividing the near distance area into a vanishing point-oriented near distance area and a non-vanishing point-oriented near distance area based on a second criterion related to a vanishing point.

[0015] The step of sequentially determining values ​​of the plurality of degrees of freedom may include a step of determining a rotation-related degree of freedom based on the far-distance region, a step of determining a left-right movement-related degree of freedom based on the vanishing-point-oriented close-distance region, and a step of determining a front-back movement-related degree of freedom based on the non-vanishing-point-oriented close-distance region.

[0016] The step of determining the positioning of the device may include the steps of determining a rotation-related degree of freedom based on far-distance vertices included in the second image whose influence due to a movement-related degree of freedom is equal to or less than a first threshold, determining a left-right movement-related degree of freedom based on vanishing-point-oriented close-distance vertices among close-distance vertices excluding the far-distance vertices from the second image whose influence due to a movement-related degree of freedom in the forward-backward direction is equal to or less than a second threshold, and determining the movement-related degree of freedom in the forward-backward direction based on non-vanishing-point-oriented close-distance vertices among the close-distance vertices excluding the vanishing-point-oriented close-distance vertices.

[0017] Determining a position fix for the device may include determining a direction to improve the score based on the distribution of the pooled feature values, and correcting the candidate position fix based on the direction.

[0018] The first image may include a probability distribution indicating a degree of proximity to the object, and determining the direction may include determining the direction based on the probability distribution.

[0019] The step of determining the positioning of the device may include the steps of generating a corrected second image onto which the object is projected according to the corrected candidate positioning, and determining a corrected score of the corrected candidate positioning by pooling feature values ​​from the first image corresponding to vertices included in the corrected second image, wherein the steps of determining the direction, correcting the candidate positioning, generating the corrected second image, and determining a corrected score of the corrected candidate positioning are repeated until a condition for determining the corrected score of the corrected candidate positioning is satisfied.

[0020] The positioning method may further include determining a virtual object on the map data for an augmented reality service, and displaying the virtual object based on the determined positioning.

[0021] A positioning method according to one embodiment includes the steps of generating a first image of an object from an input image, generating a second image projecting the object according to a candidate positioning of a device based on map data including the position of the object, dividing the second image into a plurality of regions, and determining values ​​of a plurality of degrees of freedom included in the candidate positioning by matching between the divided regions and the first image.

[0022] The step of determining values ​​of the multiple degrees of freedom may include a step of sequentially determining values ​​of the multiple degrees of freedom included in the candidate positioning by using a score calculated by matching with the first image in the divided area.

[0023] The step of determining the values ​​of the degrees of freedom may include a step of calculating a score based on the changed values ​​of the degrees of freedom by changing the determined values ​​of the degrees of freedom corresponding to the divided regions and pooling feature values ​​from the first image corresponding to vertices within the corresponding regions, and a step of selecting the value of the degrees of freedom corresponding to the maximum score.

[0024] The plurality of degrees of freedom values ​​may include three translational degrees of freedom values ​​and three rotational degrees of freedom values.

[0025] The dividing step may include dividing the second image into a far distance area and a near distance area based on a first criterion related to distance, and dividing the near distance area into a vanishing point-oriented near distance area and a non-vanishing point-oriented near distance area based on a second criterion related to a vanishing point.

[0026] The step of determining values ​​of the plurality of degrees of freedom may include the steps of determining a rotation-related degree of freedom based on the far-distance region, determining a left-right movement-related degree of freedom based on the vanishing-point-oriented close-distance region, and determining a front-back movement-related degree of freedom based on the non-vanishing-point-oriented close-distance region.

[0027] The step of determining the values ​​of the multiple degrees of freedom may include the steps of determining a rotation-related degree of freedom based on far-distance vertices included in the second image whose influence due to a movement-related degree of freedom is equal to or less than a first threshold value; determining a left-right movement-related degree of freedom based on vanishing-point-oriented close-distance vertices among close-distance vertices excluding the far-distance vertices from the second image whose influence due to a movement-related degree of freedom in the forward-backward direction is equal to or less than a second threshold value; and determining the movement-related degree of freedom in the forward-backward direction based on non-vanishing-point-oriented close-distance vertices excluding the vanishing-point-oriented close-distance vertices among the close-distance vertices.

[0028] The positioning method may further include determining a virtual object on the map data for an augmented reality service, and displaying the virtual object based on the determined values ​​of the degrees of freedom.

[0029] The input image may include an image of a vehicle traveling, and the virtual object may indicate traveling route information.

[0030] A positioning device according to one embodiment includes at least one processor that generates a first image of an object from an input image, generates a second image that projects the object based on map data including the position of the object according to a candidate positioning of the device, pools feature values ​​from the first image that correspond to vertices in the second image, and determines a score for the candidate positioning based on the pooled feature values.

[0031] A positioning device according to an embodiment generates a first image of an object from an input image, generates a second image onto which the object is projected by a candidate positioning of the device based on map data including a position of the object, and divides the second image into a plurality of regions; At least one processor is configured to determine values ​​of a plurality of degrees of freedom included in the candidate positioning by matching between the segmented regions and the first image.

[0032] A positioning device according to one embodiment includes a sensor attached to a device and detecting one or more images and candidate positioning information of the device, a processor that generates a first image of an object from the images, generates a second image projecting the object with information on the candidate side based on map data including a position of the object, determines a score for the information on the candidate side by pooling feature values ​​corresponding to vertices in the second image from the first image, and determines positioning information of the device based on the score, and a head-up display (HUD) that visualizes a virtual object on the map data based on the determined positioning information.

[0033] The processor may divide the second image into a far distance region and a near distance region based on distance, and divide the near distance region into a vanishing point oriented near distance region and a non-vanishing point oriented near distance region based on a vanishing point.

[0034] The processor may determine a rotational degree of freedom based on the far distance region, determine a left / right translation degree of freedom based on the vanishing point oriented near distance region, and determine a front / back translation degree of freedom based on the non-vanishing point oriented near distance region.

[0035] The processor may generate the first image including a feature map corresponding to a plurality of features using a neural network.

[0036] The second image may include a projection of two-dimensional vertices corresponding to the object.

[0037] The positioning device may further include a memory including instructions for controlling the processor to determine at least one of the map data, the image, the first image, the second image, the score, and the determined positioning information and the virtual object. Effect of the Invention

[0038] According to an embodiment of the present invention, a method for positioning based on an image and map data can be provided. The embodiment can provide a technique that allows positioning without establishing a correspondence between vertices of the image and vertices of the map data. The embodiment can also provide a technique that allows positioning without parameterizing image features, extracting relationships that are invariant to 3D transformations and perspective transformations, or easily identifying such invariant relationships during search from the map data. [Brief description of the drawings]

[0039] [Figure 1A] FIG. 1 illustrates the importance of positioning accuracy in an augmented reality application according to one embodiment. [Figure 1B] FIG. 1 illustrates the importance of positioning accuracy in an augmented reality application according to one embodiment. [Figure 1C] FIG. 1 illustrates the importance of positioning accuracy in an augmented reality application according to one embodiment. [Diagram 2] 11 is a diagram illustrating an operation of calculating a positioning score according to an embodiment. FIG. [Diagram 3] FIG. 11 is a diagram illustrating a specific example of an operation for calculating a positioning score according to an embodiment. [Figure 4] FIG. 13 is a diagram illustrating an operation of determining a positioning of a device using a positioning score according to one embodiment. [Diagram 5] FIG. 13 illustrates scores of candidate positions according to an embodiment. [Figure 6] 1 is an operational flowchart illustrating a positioning method according to an embodiment. [Figure 7] FIG. 13 is a diagram illustrating an operation of determining device positioning using an optimization scheme according to an embodiment. [Figure 8A] 1 is an operational flow diagram illustrating an optimization method according to one embodiment. [Figure 8B] 1 is an operational flow diagram illustrating an optimization method according to one embodiment. [Figure 9A] 1A-1C are diagrams illustrating the steps of applying an optimization method according to an embodiment; [Figure 9B] 1A-1C are diagrams illustrating the steps of applying an optimization method according to an embodiment; [Figure 9C] 1A-1C are diagrams illustrating the steps of applying an optimization method according to an embodiment; [Figure 9D] 1A-1C are diagrams illustrating the steps of applying an optimization method according to an embodiment; [Figure 9E] 1A-1C are diagrams illustrating the steps of applying an optimization method according to an embodiment; [Figure 10] 1 is an operational flowchart illustrating a positioning method using parameter updates according to an embodiment. [Figure 11] 11A and 11B are diagrams illustrating a result of applying a positioning method using parameter updates according to an embodiment in a step-by-step manner. [Figure 12] FIG. 1 illustrates a neural network for generating a feature map according to one embodiment. [Figure 13] 1 is a block diagram showing a positioning device according to an embodiment; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0040] The specific structural or functional description of the embodiment is disclosed for the purpose of illustration only and may be modified in various forms. Therefore, the embodiment is not limited to the specific disclosed form, and the scope of the present specification includes modifications, equivalents, or alternatives within the technical spirit.

[0041] Although terms such as first or second may be used to describe multiple components, such terms should be construed only for the purpose of distinguishing one component from the other components. For example, a first component may be named a second component, and similarly, a second component may be named a first component.

[0042] When a component is referred to as being "coupled" or "connected" to another component, it should be understood that although it is directly coupled or connected to the other component, there may be other components in between.

[0043] The terms used in this specification are merely used to describe certain embodiments and are not intended to limit the present invention. A singular expression includes a plural expression unless the context clearly indicates otherwise. In this specification, the terms "include" or "have" indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, and should be understood as not precluding the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0044] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which the present invention belongs. Commonly used predefined terms should be interpreted as having a meaning consistent with the meaning they have in the context of the relevant art, and should not be interpreted as ideal or overly formal unless expressly defined in this specification.

[0045] The embodiments described below may be implemented in the form of software and applied to positioning technology based on image and map data. For example, the embodiments may be used to improve the accuracy of positioning, which is a key technology for realizing an Augmented Reality Head-Up Display (AR HUD). In addition, positioning is a necessary technology for many location-based services other than HUD, and the embodiments may be used for estimating position and orientation in an environment where HD map data is provided for high-precision positioning.

[0046] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same reference numerals will be used to refer to the same components regardless of the reference numerals, and duplicated description thereof will be omitted.

[0047] FIG. 1 is a diagram illustrating the importance of positioning accuracy in an augmented reality application according to one embodiment.

[0048] Referring to FIG. 1, augmented reality is a technology that adds or enhances information based on reality, and can display, for example, a virtual object corresponding to a virtual image on an image or background image of the real world. Augmented reality can provide a sense of immersion by blending the real world and the virtual world well, allowing the user to interact in real time between the virtual world without the user being aware that the real and virtual environments are separate. In order to match a virtual object with a real image, the position and orientation of the user device (or user) that provides the augmented reality, in other words, localization information, must be known.

[0049] The positioning information for providing the augmented reality is used to position a virtual object at a desired location in an image. Hereinafter, for convenience of explanation, the present specification will be described by taking as an example a case where a driving guide road corresponding to a virtual object is displayed on a road surface, but the present invention is not limited thereto.

[0050] 1A shows an augmented reality image 120 when the positioning error is small, and FIG 1B shows an augmented reality image 140 when the positioning error is large.

[0051] For example, assume that a reference route of a vehicle is displayed on a road image based on the positioning information of an object 110. Here, the object corresponds to a user terminal and / or a vehicle performing positioning. When the positioning error of the object 110 is small, a driving guidance road 115, which is a virtual object to be displayed by the device, may be visually well aligned with the actual road image, such as an augmented reality image 120. In contrast, when the positioning error of the object 130 is large, a driving guidance road 135, which is a virtual object to be displayed by the device, may not be visually well aligned with the actual road image, such as an augmented reality image 140.

[0052] 1C, the positioning information according to one embodiment includes the position of the device and the attitude of the device. The position is a movement-related degree of freedom, which is a lateral t x , Vertical y , and length (Longitudinal) t z In other words, it corresponds to three-dimensional coordinates such as (x, y, z). Also, the attitude is a rotation-related degree of freedom, and it corresponds to pitch r x , Yawr y , and Rollr zThe position is acquired by, for example, a GPS sensor, a lidar, etc., and the attitude is acquired by, for example, an IMU sensor, a gyro sensor, etc. Positioning information according to one embodiment is understood to have six degrees of freedom (DoF) including the position and attitude described above.

[0053] Hereinafter, in this specification, the term "vehicle" refers to a vehicle traveling on a road, and includes, for example, an autonomous vehicle and an intelligent vehicle equipped with an advanced driver assistance system (ADAS). The term "road" refers to a road on which a vehicle travels, and includes, for example, various types of roads such as expressways, national roads, local roads, express national highways, and motorways. A road includes one or more lanes. A "lane" corresponds to a road space divided by lanes displayed on a road surface. A lane is divided by left and right lanes (or lane boundaries) adjacent to the corresponding lane. In addition, a "lane" is understood to be a line of various shapes, such as a solid line, a dotted line, a curved line, and a zigzag line, displayed on a road surface in white, blue, or yellow. A lane may correspond to a lane on one side of a road, or may correspond to a pair of lanes that divide a road, in other words, a left lane and a right lane that correspond to a road boundary line.

[0054] The embodiments described below are utilized to display lanes in an Augmented Reality Navigation system such as a smart vehicle, generate visual information to assist steering of an autonomous vehicle, or provide various control information for vehicle driving. The embodiments are also used to assist safe and comfortable driving by providing visual information to a device including an intelligent system such as a Head Up Display (HUD) installed for driving assistance or fully autonomous driving in a vehicle. For example, the embodiments can be used to interpret visual information of an intelligent system installed for autonomous driving or driving assistance in a vehicle. The embodiments are applied to, for example, an autonomous vehicle, an intelligent vehicle, an advanced driver assistance system (ADAS), a navigation system, a smartphone, and a mobile device.

[0055] FIG. 2 is a diagram illustrating an operation of calculating a positioning score according to an embodiment.

[0056] 2, a positioning device according to an embodiment calculates a score s(θ) of a positioning parameter θ based on map data Q and an image I. The positioning device may be implemented as one or more software modules, one or more hardware modules, or various combinations thereof.

[0057] As will be described in detail below, the embodiment does not require a correspondence between the vertices of the map data and the image features as a prerequisite. For example, the map data may be a point cloud including a plurality of 3D vertices corresponding to objects such as lanes. The 3D vertices of the map data are projected to 2D vertices based on the positioning parameters. The image features include feature values ​​extracted for each pixel included in the image.

[0058] The embodiments described below do not require information about how vertices (e.g., 2D vertices) of the map data and features (or pixels) of the image correspond (or match) to each other, and the features extracted in the image do not need to be parameterized, and relationships between features do not need to be separately analyzed or searched for in the map data.

[0059] The positioning parameter θ is a position / attitude information parameter and is defined as a 6-degree-of-freedom variable as described with reference to FIG. 1C. According to an embodiment, the positioning parameter θ corresponds to rough position / attitude information. The positioning device can improve the accuracy of positioning by correcting the positioning parameter θ using a scoring technique based on image and map data.

[0060] For example, the positioning device extracts features from an image I to construct a feature map. The positioning device calculates a matching score for a positioning parameter θ. More specifically, the positioning device projects vertices from map data Q based on the positioning parameters, and calculates the matching score by pooling feature values ​​of pixels in the feature map that correspond to the two-dimensional coordinates of the projected vertices. The positioning device updates the positioning parameter θ in a direction that increases the matching score.

[0061] Here, the device may be, for example, a vehicle or a user device such as a navigation system or a smartphone, as a device that performs a positioning method according to an embodiment. The positioning information has six degrees of freedom including the position and attitude of the device as described above. The positioning information may be obtained based on the output of a sensor such as an IMU sensor, a GPS sensor, a lidar sensor, and a radar.

[0062] The input image corresponds to a background image or other image to be displayed together with a virtual object for an augmented reality service in the future. The input image includes, for example, a driving image of a vehicle. Here, the driving image may be, for example, a driving image captured by a photographing device mounted on the vehicle, and includes one or more frames.

[0063] The positioning device acquires an input image based on the output of the photographing device. The photographing device is fixed to a predetermined position such as a windshield, dashboard, or rear-view mirror of a vehicle to photograph a driving image in front of the vehicle. The photographing device may include, for example, a vision sensor, an image sensor, or a device performing a similar function, and may photograph a single image or, in some cases, photograph frame-by-frame images. The image captured by another device other than the photographing device fixed to the vehicle may be a driving image. The object may include, for example, a lane, a road marking, a traffic light, a sign, a curb, a pedestrian, and a structure. The lane may include, for example, lines such as a road boundary line, a road center line, and a stop line, and the road marking may include, for example, signs such as no parking, a crosswalk, a tow area, and a speed limit sign.

[0064] The map data may be, for example, high density (HD) map data. The high precision map is a 3D map with high precision (for example, centimeter-level precision) for autonomous driving. The high precision map may include, for example, lane-level information such as road centerlines and boundaries, as well as information on traffic lights, signboards, curbs, road signs, and various structures in a 3D digital form. The high precision map is constructed, for example, by a Mobile Mapping System (MMS). The MMS, which is a 3D spatial information projection system equipped with various sensors, can obtain detailed position information using a mobile body equipped with sensors such as a camera, a LIDAR, and a GPS for position measurement and topographical object surveying.

[0065] FIG. 3 is a diagram illustrating a specific operation of calculating a positioning score according to an embodiment.

[0066] Referring to FIG. 3, the positioning device 200 according to an embodiment includes converters 210 and 220, a feature extractor 230, and a pooler 240. The converter 210 receives a parameter θ and map data Q, and applies a 3D position and 3D orientation of the device according to the parameter θ to the map data Q through a 3D transformation (T). As an example, the converter 210 extracts an area corresponding to a field of view range at the position and orientation corresponding to the parameter θ from the map data. The converter 220 generates a projected image at the viewpoint of the device through a perspective transformation (P). As an example, the converter 220 may project 3D vertices included in the area extracted by the converter 210 onto a 2D projection plane corresponding to the parameter θ. In this case, the 3D vertices q included in the map data Q are projected onto a 2D projection plane corresponding to the parameter θ. i k is transformed into a 2D vertex p in the projected image by T-transformation and P-transformation with the parameter θ. i k where k can be an index indicating distinct features or classes, and i can be an index indicating a vertex within the feature or class.

[0067] The feature extractor 230 extracts features from the image I. The features are stored in one or more feature maps F according to the object type (or class). 1 , F 2 235. For example, the feature map F 1 contains features related to lanes in the image, and the feature map F 2 contains features related to the traffic lights in the video. For convenience of explanation, we use two feature maps F 1 , F 2 Although a case where 235 is extracted will be described, the embodiment is not limited to such a case.

[0068] The positioning device may include separate feature extractors for extracting multiple feature maps, or the positioning device may include a single feature extractor (e.g., a deep neural network) that outputs multiple feature maps for each channel.

[0069] Extracted feature map F1 , F 2 may contain errors that prevent the feature value from being determined precisely on a pixel-by-pixel basis. In this case, each feature map may have a value between 0 and 1 for each pixel. The pixel feature value indicates the intensity of the corresponding feature for each pixel.

[0070] 2D vertex p of the projected image i k is the 3D vertex p of the map data. i k means a pixel mapped to image I. Referring to Equation (1), the scores for each feature are summed for pixels mapped to image I.

[0071]

number

[0072] Here, T() denotes the T transformation and P() denotes the P transformation. In this case, P(T(q i k , θ)) denotes the mapping point, and F k () indicates the feature value or score by the mapping point in the feature map corresponding to the kth feature or class. Here, the mapping point P(T(q i k , θ)) are not integers, operations such as rounding or interpolation may be performed. Referring to Equation (2), the scores for each feature may be added together to calculate the final score.

[0073]

number

[0074] Here, the weight value w k is set in any way. For example, the weights w k may be set to a weighted value that is uniformly assigned to all the nodes, or may be set to a value tuned based on training data.

[0075] 4 is a diagram illustrating an operation of determining the positioning of a device using a positioning score according to an embodiment. The order of the operations shown in FIG. 4 may be changed, some of the operations shown in FIG. 4 may be omitted, or at least some of the operations shown in FIG. 4 may be processed in parallel, without departing from the scope of the rights according to the embodiment.

[0076] 4, an input video is received in step S410, features are extracted in step S430, map data is obtained in step S420, and a Positioning candidates is obtained, where Positioning candidates is multiple Positioning candidates Includes.

[0077] In step S450, Positioning candidates A projected image of the map data is generated. Positioning candidates If there are multiple, multiple Positioning candidates A number of projection images are generated by the

[0078] In step S460, feature values ​​are pooled from the feature map corresponding to the two-dimensional vertices in the projection image. Based on the feature values ​​pooled in step S460, Positioning candidates A score is calculated. Positioning candidates If there are multiple Positioning candidates A score is calculated for each.

[0079] In step S470, the best score (eg, the maximum score) is retrieved. In step S480, the candidate positioning solution having the retrieved best score is determined as the positioning solution of the device.

[0080] Although not shown in the drawings, the positioning device 200 determines a virtual object on the map data Q for the augmented reality service. For example, the virtual object is expressed as a virtual object indicating driving route information in the form of an arrow or a road display indicating a traveling direction. Based on the positioning information determined in step S480, the positioning device may display the virtual object together with the input image on a head-up display, a navigation system, or a display of a user device.

[0081] FIG. 5 is a block diagram of an embodiment of the present invention. Positioning candidates FIG.

[0082] 5, the degree of visual alignment between the projected image and image 510 based on the first candidate positioning is lower than the degree of visual alignment between the projected image and image 520 based on the second candidate positioning. Therefore, the score pooled from the feature map of image 510 based on the first candidate positioning is calculated to be lower than the score pooled from the feature map of image 520 based on the second candidate positioning.

[0083] Fig. 6 is an operational flowchart showing a positioning method according to an embodiment. Within the scope of the rights according to the embodiment, the order of the operations shown in Fig. 6 may be changed, some of the operations shown in Fig. 6 may be omitted, or at least some of the operations shown in Fig. 6 may be processed in parallel.

[0084] Referring to FIG. 6, in step S610, at least one feature map is extracted from an input video. In step S620, a set of candidates for positioning parameters (e.g., position / pose parameters) is selected. In step S630, a set of candidates for positioning parameters (e.g., position / pose parameters) is evaluated. Positioning candidates Determine whether exists. Evaluate Positioning candidates If exists, in step S640, Positioning candidates A projection image is generated by the above-mentioned method, and in step S650, a score for each feature is calculated. In step S660, a final score is calculated by adding and accumulating the scores for each feature. In step S670, a final score is calculated by adding and accumulating the scores for each feature. Positioning candidates Best of Positioning candidates and the final score to determine the best Positioning candidates The information will be updated.

[0085] Further evaluation in step S680 Positioning candidates If does not exist, the pre-evaluated Positioning candidates Best of Positioning candidates is determined as the device's positioning. In this case, the best Positioning candidates The parameters are chosen as the device position / orientation parameters.

[0086] Fig. 7 is a diagram illustrating an operation of determining the positioning of a device by an optimization method according to an embodiment. Within the scope of the rights according to the embodiment, the order of the operations shown in Fig. 7 may be changed, some of the operations shown in Fig. 7 may be omitted, or at least some of the operations shown in Fig. 7 may be processed in parallel.

[0087] 7, an input image is received in step S710, a feature map is extracted in step S730, map data is received in step S720, an initial positioning is received in step S740, and a projected image according to the initial positioning is generated in step S750.

[0088] In step S760, the initial positioning is updated according to an optimization scheme according to an embodiment. In step S770, the positioning of the device is determined to be the optimized positioning.

[0089] The optimization method of step S760 will now be described in detail.

[0090] Fig. 8 is an operational flowchart showing an optimization method according to an embodiment. Within the scope of the rights according to the embodiment, the order of the operations shown in Fig. 8 may be changed, some of the operations shown in Fig. 8 may be omitted, or at least some of the operations shown in Fig. 8 may be processed in parallel.

[0091] The positioning device according to an embodiment supports a global optimization process, classifying 2D vertices projected from map data based on criteria other than features (e.g., distance, presence or absence of vanishing point orientation, etc.), and then using the classified 2D vertices to estimate different degrees of freedom of positioning parameters.

[0092] A positioning device according to an embodiment divides a projected image into a plurality of regions, and determines the position of the device through matching between the divided regions and a feature map. More specifically, the positioning device sequentially determines values ​​of a plurality of degrees of freedom included in the positioning by sequentially using scores calculated through matching between the divided regions and the feature map. For example, the positioning device pools feature values ​​from the feature map corresponding to two-dimensional vertices included in the corresponding regions while changing the values ​​of the degrees of freedom determined corresponding to the divided regions. The positioning device calculates a score according to the changed values ​​of the degrees of freedom based on the pooled feature values. Thereafter, the positioning device determines the corresponding degrees of freedom to be a value corresponding to the maximum score.

[0093] A vertex far away in a projected image has a characteristic that is almost invariant to a change in a position parameter. Based on this characteristic, the positioning device first performs a process of calculating scores at a far-away vertex to determine an attitude parameter, and then performs a process of calculating scores at a close-away vertex to determine a position parameter separately. Therefore, the degree of freedom that must be estimated at each step is reduced, which reduces the complexity during search and the possibility of local convergence during optimization.

[0094] As described below, the positioning device divides the projected image into a far distance region and a near distance region based on a first criterion related to distance, and divides the near distance region into a vanishing point-oriented near distance region and a non-vanishing point-oriented near distance region based on a second criterion related to a vanishing point. Here, the far distance region includes two-dimensional vertices whose influence due to movement-related degrees of freedom is equal to or less than a predetermined level (or threshold). Also, the vanishing point-oriented near distance region includes two-dimensional vertices whose influence due to movement-related degrees of freedom in the forward / rearward direction is equal to or less than a predetermined level (or threshold).

[0095] In addition, the positioning device may use some of the degrees of freedom of the positioning parameters as values ​​determined by prior calibration. For example, z and y is determined by pre-calibration. The height t y and role r z is fixed.

[0096] Referring to FIG. 8, in step S810, a feature map is extracted from an input image. Then, in step S820, a far-distance vertex Q is extracted from a projected image of the map data. 1 is selected. Far-distance vertex Q 1 is the degree of freedom related to movement among the degrees of freedom of the positioning parameters, t x , t z Therefore, in step S830, the far vertex Q 1 Based on the rotation-related degree of freedom r x , r y is determined. The rotation-related degree of freedom r x , r y may be referred to as a pose parameter.

[0097] More specifically, the positioning device is x While changing the far-away vertex Q 1 is translated vertically, and r y While changing the far-away vertex Q 1 The positioning device translates the far vertex Q 1 The score calculated corresponding to r is set to be equal to or greater than the determined target value. x and r y Search for the value of .

[0098] In step S840, the near vertices are selected from the map data. z and t y , and the distant vertex Q 1 Determined by r x and ry With is applied, close vertices are selected.

[0099] The positioning device detects the vertex Q that corresponds to the lane heading toward the vanishing point among the near vertices. 2 Select the other vertices Q 3 Select the vertex Q. 2 is the degree of freedom related to forward and backward movement among the degrees of freedom of the positioning parameters, t z Therefore, in step S850, the vertex Q 2 Based on the translation related degrees of freedom t x Also, vertex Q 3 Based on the translation related degrees of freedom t z is determined. The movement-related degree of freedom t x , t z may be referred to as a position parameter.

[0100] FIG. 9 is a diagram showing a step-by-step result of applying an optimization method according to an embodiment.

[0101] 9A, there is shown a feature 911 extracted from a feature map 910 and a vertex 921 projected from map data 920. Because the initial positioning on the map data 920 is inaccurate, the feature 911 and the vertex 921 do not match each other. Hereinafter, a process of sequentially determining the degrees of freedom of the positioning parameters so that the feature 911 and the vertex 921 match each other will be described.

[0102] Taking into consideration that a camera installed on a vehicle has a certain height and a certain roll relative to the road surface, y and r z is pre-calibrated.

[0103] 9B, the positioning device may remove the roll effect with the feature map 910 and the map data 920. For example, the positioning device may remove the roll effect with a pre-calibrated r zThe roll effect is removed in the feature map 910 by rotating 930 the feature map 910 based on the above. The positioning device also detects peripheral vertices of the initial positioning in the map data 920, approximates the detected peripheral vertices to a plane, and then rotates the map data 920 so that the roll of the plane is removed.

[0104] In addition, the positioning device is calibrated in advance. y The height of the vertices of the map data 920 is corrected using

[0105] Referring to FIG. 9C, the positioning device uses a far vertex 940 in the projected image of the map data as a reference for the initial positioning to perform a translational shift r x andr y Here, the positioning device may translate the far-distance vertex 940 so that the correlation between the far-distance vertex 940 and the feature map 910 is equal to or greater than a predetermined target value. For example, r x By adjusting and rotating 945 the vertices on the map data 920 , the far vertices 940 in the projected image are smoothly aligned with the features in the feature map 910 .

[0106] Referring to FIG. 9D, the positioning device can analyze vertices in the projected image to determine the vanishing points of the surrounding lanes 950. The positioning device aligns the vanishing points of the surrounding lanes 950 to a predetermined position (e.g., the center of the feature map) in the feature map 910. In this case, the surrounding lanes have a characteristic that is invariant to the z-direction movement. The positioning device moves the vertices corresponding to the surrounding lanes 950 in the x direction so that the correlation between the surrounding lanes 950 and the feature map 910 is equal to or greater than a predetermined target value. For example, x By adjusting the vertices on the map data 920 and moving 955 the vertices on the map data 920, the vertices corresponding to the surrounding lanes 950 in the projected image are smoothly matched with the features of the feature map 910.

[0107] Referring to FIG. 9E, the positioning device searches for a z-direction movement using the remaining vertices 960, excluding the lanes in the vanishing point direction, among the close-distance vertices in the projected image. The positioning device moves the remaining vertices 960 in the z direction so that the correlation between the remaining vertices 960 and the feature map 910 is equal to or greater than a predetermined target value. For example, z 9 to move 965 the vertices on the map data 920 so that the remaining vertices 960 in the projected image are smoothly aligned with the features in the feature map 910.

[0108] Fig. 10 is an operational flowchart showing a positioning method using parameter updates according to an embodiment. Within the scope of the rights according to the embodiment, the order of the operations shown in Fig. 10 may be changed, some of the operations shown in Fig. 10 may be omitted, or at least some of the operations shown in Fig. 10 may be processed in parallel.

[0109] 10, in step S1010, a feature map is extracted from an input video, and in step S1020, an initial positioning (e.g., initial values ​​for position / orientation parameters) is selected. In step S1030, a score for the current parameters is calculated, and a direction in which the score for the current parameters is improved is calculated.

[0110] According to one embodiment, the feature map includes a probability distribution indicating the degree of proximity to the object. For example, the features included in the feature map may include information about the distance to the nearest object as a normalized value between 0 and 1. In this case, the feature map provides information about the direction toward the object. The positioning device pools the feature values ​​of the feature map, which correspond to the two-dimensional vertices projected from the map data by the current parameters. The positioning device determines a direction in which the score of the current parameters may be improved based on the pooled feature values.

[0111] In step S1040, it is determined whether the iteration end condition is satisfied, and if the iteration end condition is not satisfied, the parameters are updated in step S1050. The positioning device updates the parameters according to the direction calculated in step S1030. Steps S1050, S1030, and S1040 are repeatedly executed until the iteration end condition is satisfied. The iteration end condition includes whether the score of the parameter is equal to or greater than a determined target value. According to an embodiment, for system stability, the iteration end condition may further include whether the number of iterations exceeds a determined threshold.

[0112] If an iteration termination condition is met, then in step S1060, the current parameters are selected as the final fix (eg, final position / attitude parameters).

[0113] The embodiment shown in FIG. 10 is a process of gradually arriving at good parameters starting from the initial value θ_0, but such a local optimization method cannot achieve good performance unless the initial value is good. Therefore, the embodiment shown in FIG. 10 may selectively include a step of separately estimating the initial value. As an example, the parameters obtained by the embodiment shown in FIG. 8 are regarded as the initial values, and the embodiment shown in FIG. 10 is executed. In this case, it is possible to achieve the effect of correcting values ​​such as roll and camera height fixed through pre-calibration to suit changes occurring in the actual driving environment.

[0114] FIG. 11 is a diagram showing a step-by-step result of applying a positioning method using parameter updates according to an embodiment.

[0115] 11, an input image 1105, a first image 1110, and a second image 1120 are shown. The first image 1110 is generated in response to the input image 1105. The second image 1120 is generated based on positioning information x, y, z, r obtained by initial positioning based on map data. x , r y , r zThe second image 1120 may be an image generated by projecting an object using the projection method. The second image 1120 is a projected image that includes a plurality of two-dimensional vertices corresponding to the object.

[0116] The positioning device calculates a score Score 1 by matching the first image 1110 and the second image 1120 as in the image 1130. For example, the positioning device calculates the score by adding up pixel values ​​corresponding to an object included in the second image 1120 among a plurality of pixels included in the first image 1110.

[0117] For example, a plurality of pixels included in the first image 1110 has a value between 0 and 1 depending on the distance to an adjacent object. The closer a pixel is to an adjacent object, the closer it is to 1, and the farther a pixel is from the adjacent object, the closer it is to 0. The positioning device extracts pixels that match with the second image 1120 from the plurality of pixels included in the first image 1110, and calculates a score by adding up the values ​​of the extracted pixels.

[0118] The positioning device corrects the positioning information based on the directionality of the first image 1110 so that the degree of visual alignment, in other words, the score l, increases. The positioning device calculates a positioning correction value so that the positioning of the object included in the second image 1120 is suitable for the directionality of the first image 1110. The positioning device updates the positioning information from l to l' 1140 by applying the positioning correction value to the positioning information by the initial positioning. For example, the positioning device determines in which direction the object in the second image 1120 should be moved to increase the score based on the directionality included in the first image 1110. When updating the positioning information, since the object in the second image 1120 moves, the positioning device updates the positioning information based on the directionality included in the first image 1110.

[0119] The positioning device generates an updated second image 1150 based on the updated positioning information l'. The positioning device matches the updated second image 1150 with the first image 1110 to calculate a score l'.

[0120] The positioning device calculates a positioning correction value that makes the score equal to or higher than a predetermined standard through the above-mentioned process, and finally obtains optimized positioning information. * Output.

[0121] FIG. 12 is a diagram illustrating a neural network for generating a feature map according to one embodiment.

[0122] Referring to FIG. 12, a process of generating a distance field map 1250 corresponding to a first image by applying an input image 1210 to a neural network 1230 is shown.

[0123] The neural network 1230 may be a neural network that has been trained in advance to generate a first image including a direction corresponding to an object included in the input image 1210 based on the input image 1210. The neural network 1230 may represent a learning model implemented in software or hardware using a plurality of artificial neurons (nodes). The neural network 1230 includes artificial neurons. The neural network 1230 may be realized in various structures, such as a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), or a bidirectional long short term memory (BLSTM).

[0124] The neural network 1230 may estimate an object included in the input image 1210 in the form of a distance field map 1250. For example, if the first image includes directional information toward a nearby object as in the distance field map 1250, the direction of optimization may be easily determined by using a gradient descent method, etc. Also, if a probability distribution indicating the degree of proximity to an object is distributed throughout the image as in the distance field map 1250, the amount of data for learning may be increased, and the performance of the neural network may be improved compared to when learning using sparse data.

[0125] FIG. 13 is a block diagram showing a positioning device according to an embodiment.

[0126] Referring to FIG. 13, a positioning device 1300 according to an embodiment includes a sensor 1310 and a processor 1330. The positioning device 1300 further includes a memory 1350, a communication interface 1370, and a display device 1390. The sensor 1310, the processor 1330, the memory 1350, the communication interface 1370, and the display device 1390 are connected via a communication bus 1305. The display device 1390 includes one or more hardware configurations that provide a function of rendering a user interface, rendering a display, or receiving a user input. The display device 1390 can be realized in various forms, such as a smartphone, a glasses-type display, etc., without being limited to the above examples.

[0127] The sensor 1310 includes, for example, an image sensor, a vision sensor, an acceleration sensor, a gyro sensor, a Global Positioning System (GPS) sensor, an Inertial Measurement Unit (IMU) sensor, a radar, a lidar, etc. The multiple sensors 1310 acquire (take) input images including images of a vehicle traveling. The sensor 1310 detects detection information such as the speed, acceleration, traveling direction, the steering angle of the vehicle, and the speed of the vehicle in addition to positioning information such as the GPS coordinates, position, and attitude of the vehicle.

[0128] The embodiments provide a technique that allows positioning without establishing a correspondence between vertices of an image and vertices of map data, and also without parameterizing image features, extracting relationships that are invariant to 3D transformations and perspective transformations, or easily identifying such invariant relationships when searching map data.

[0129] According to an embodiment, the positioning device 1300 acquires detection information of various sensors including input images via the communication interface 1370. According to an embodiment, the communication interface 1370 receives detection information including driving images from other sensors existing outside the positioning device 1300.

[0130] The processor 1330 may provide an augmented reality service by outputting the corrected positioning information via the communication interface 1370 and / or the display device 1390, or by displaying a virtual object on the map data together with the input image based on the corrected positioning information. The processor 1330 may also perform at least one of the methods or an algorithm corresponding to at least one of the methods described above with reference to FIGS. 1 to 13.

[0131] The processor 1330 may be a data processing device implemented in hardware having circuits with physical structures for performing desired operations. For example, the desired operations may include code or instructions contained in a program. For example, data processing devices implemented in hardware include a microprocessor, a central processing unit, a processor core, a multi-core processor, a multiprocessor, an application-specific integrated circuit (ASIC), and a field programmable gate array (FPGA).

[0132] The processor 1330 executes programs and controls the positioning device 1300. The program codes executed by the processor 1330 are stored in the memory 1350.

[0133] The memory 1350 stores the positioning information, the first image, the second image, and / or the corrected positioning information of the positioning device 1300. The memory 1350 stores various information generated in the processing of the processor 1330 described above. In addition, the memory 1350 stores various data and programs. The memory 1350 includes a volatile memory or a non-volatile memory. The memory 1350 includes a large-capacity storage medium such as a hard disk and stores various data.

[0134] The display device 1390 may output the positioning information corrected by the processor 1330, or may display a virtual object together with the input image on the map data based on the corrected positioning information.

[0135] The positioning device according to an embodiment can perform viewpoint-independent execution by updating 3D positioning information of the positioning device using the result of performing the above-mentioned positioning method based on the imaging device even when the viewpoint between the imaging device and the positioning device is not the same, for example, as in a head-up display (HUD) or augmented reality glasses (AR Glasses), etc. Also, the positioning device according to an embodiment can update 3D positioning information when the viewpoint between the imaging device and the positioning device is the same, as in a mobile terminal or smartphone, and can also be used to directly correct 2D position in an image.

[0136] The above-described embodiments may be implemented with hardware components, software components, or a combination of hardware and software components. For example, the devices and components described herein may be implemented using one or more general-purpose or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU), a microprocessor, or other devices that execute and respond to instructions. The processing device executes an operating system (OS) and one or more software applications that run on the operating system. The processing device also accesses, stores, manipulates, processes, and generates data in response to the execution of the software. For ease of understanding, the description may be given with one processing device, but one skilled in the art will appreciate that the processing device includes multiple processing elements and / or multiple types of processing elements. For example, the processing device includes multiple processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

[0137] The software may include computer programs, codes, instructions, or any combination of one or more thereof, capable of configuring or instructing a processing device to operate as desired, either independently or in combination. The software and / or data may be embodied in any type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave, either permanently or temporarily, to be interpreted by the processing device or to provide instructions or data to the processing device. The software may be distributed across computer systems coupled to a network, and may be stored and executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

[0138] The method according to the present invention may be embodied in the form of program instructions to be executed by various computer means and recorded on a computer readable recording medium. The recording medium may include program instructions, data files, data structures, and the like, alone or in combination. The recording medium and the program instructions may be specially designed and constructed for the purposes of the present invention, or may be well known and available to those skilled in the art of computer software. Examples of computer readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs, DYIJDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROMs, RAMs, flash memories, and the like. Examples of program instructions include not only machine code, such as produced by a compiler, but also high-level language code, which is executed by a computer using an interpreter, and the like. The hardware devices may be configured to operate as one or more software modules to perform the operations shown in the present invention, and vice versa.

[0139] As mentioned above, even if the embodiments are described by limited drawings, those having ordinary skill in the art can apply various technical modifications and variations based on the above description. For example, the described techniques can be performed in a different order from the described method, and / or the components of the described system, structure, device, circuit, etc. can be combined or combined in a different form from the described method, or replaced or substituted by other components or equivalents to achieve appropriate results.

[0140] Therefore, the scope of the present invention should not be limited to the disclosed embodiment, but should be determined by the appended claims and their equivalents. [Explanation of symbols]

[0141] 110, 130 Objects 115, 135 Driving directions 120, 140 Augmented reality video 200 Positioning device 910 Feature Map 920 Map Data

Claims

1. generating a first image of the object from an input image; generating a second image projecting the object for a plurality of candidate positioning information associated with the device based on map data including a position of the object; pooling feature values ​​from the first image that correspond to vertices in the second image; determining a score for the plurality of candidate positioning information based on the pooled feature values; determining the candidate positioning information corresponding to the maximum score as the positioning information of the device; A positioning method including:

2. generating the first image includes generating a plurality of feature maps corresponding to a plurality of features. The positioning method according to claim 1 .

3. The step of generating the second image includes: extracting an area corresponding to a visual field range in the candidate positioning information from the map data; projecting vertices included in the region onto projection points corresponding to the candidate positioning information; The positioning method according to claim 1 or 2, comprising:

4. The step of pooling the feature values ​​includes: selecting a pixel in the first image based on the coordinates of the vertices; obtaining feature values ​​of the selected pixels; The method of claim 1 , further comprising:

5. determining a score for the candidate positioning information comprises summing the pooled feature values. A positioning method according to any one of claims 1 to 4.

6. When the first image includes a plurality of feature maps corresponding to a plurality of features, the summing step includes summing the feature values ​​according to weights determined for the plurality of features. The positioning method according to claim 5.

7. The positioning method further comprises: determining a position for the device based on a score of the candidate positioning information; The positioning method according to claim 1 .

8. The step of determining a position of the device further comprises: Dividing the second image into a plurality of regions; sequentially determining values ​​of a plurality of degrees of freedom included in the candidate positioning information by using the scores calculated for the divided regions; The method of claim 7, comprising:

9. The values ​​of the plurality of degrees of freedom are Values ​​of the three degrees of freedom of movement; The values ​​of the three rotational degrees of freedom; The method of claim 8 , comprising:

10. The dividing step includes: Dividing the second image into a far distance region and a near distance region based on a first criterion related to distance; dividing the near distance region into a vanishing point oriented near distance region and a non-vanishing point oriented near distance region based on a second criterion related to the vanishing point; The method of claim 8 , comprising:

11. The step of sequentially determining values ​​of the plurality of degrees of freedom includes: determining a rotation-related degree of freedom based on the far-field region; determining a lateral movement-related degree of freedom based on the vanishing point-oriented near field; determining a forward / backward movement-related degree of freedom based on the non-vanishing-point-oriented near field; The method of claim 10, comprising:

12. The step of determining a position of the device further comprises: determining a rotation-related degree of freedom based on a far-distance vertex having an influence of a movement-related degree of freedom equal to or less than a first threshold value among the vertices included in the second image; determining a left-right movement-related degree of freedom based on vanishing-point-oriented near vertices, among near vertices excluding the far vertices from the second image, which are affected by a front-rear movement-related degree of freedom equal to or less than a second threshold value after determining the rotation-related degree of freedom; determining a degree of freedom related to movement in the forward / backward direction based on non-vanishing-point-oriented close-distance vertices excluding the vanishing-point-oriented close-distance vertices among the close-distance vertices after determining the degree of freedom related to movement in the left / right direction; The method of claim 7, comprising:

13. The step of determining a position of the device further comprises: determining a direction for improving the score based on the pooled feature values; correcting the candidate positioning information based on the direction; The method of claim 7, comprising:

14. the first image includes a probability distribution indicating a degree of proximity to the object; determining the direction includes determining the direction based on the probability distribution. The positioning method according to claim 13.

15. The step of determining a position of the device further comprises: generating a corrected second image onto which the object is projected according to the corrected candidate positioning information; determining a corrected score of the corrected candidate positioning information by pooling feature values ​​from the first image corresponding to vertices included in the corrected second image; Including, The steps of determining the direction, correcting the candidate positioning information, generating the corrected second image, and determining the corrected score of the corrected candidate positioning information are repeated until a condition for determining the corrected score of the corrected candidate positioning information is satisfied. The positioning method according to claim 13.

16. The positioning method further comprises: determining a virtual object on the map data for an augmented reality service; displaying the virtual object based on the determined positioning; The method of claim 7, comprising:

17. The input image includes an image of a vehicle running, The virtual object indicates driving route information. The positioning method according to claim 16.

18. A computer program stored on a medium and including instructions, comprising: The instructions, when executed by hardware, cause the hardware to perform the method of any one of claims 1 to 17. Computer program.

19. generating a first image of the object from an input image; generating a second image projecting the object for a plurality of candidate positioning information associated with the device based on map data including a position of the object; Dividing the second image into a plurality of regions; determining values ​​of a plurality of degrees of freedom included in the plurality of candidate positioning information by matching between the divided regions and the first image; A positioning method including:

20. The step of determining the values ​​of the plurality of degrees of freedom includes a step of sequentially determining values ​​of the plurality of degrees of freedom included in the plurality of candidate positioning information by using a score calculated by matching with the first image in the divided region.

20. The positioning method according to claim 19.

21. The step of determining values ​​of the plurality of degrees of freedom comprises: calculating a score according to the changed value of the degree of freedom by pooling feature values ​​from the first image corresponding to vertices in the divided regions while changing the value of the degree of freedom determined corresponding to the divided regions; selecting the value of the degrees of freedom that corresponds to the maximum score; 20. The method of claim 19, comprising:

22. The values ​​of the plurality of degrees of freedom are Values ​​of the three degrees of freedom of movement; The values ​​of the three rotational degrees of freedom; 22. The method of claim 19, further comprising:

23. The dividing step includes: Dividing the second image into a far distance region and a near distance region based on a first criterion related to distance; dividing the near distance region into a vanishing point oriented near distance region and a non-vanishing point oriented near distance region based on a second criterion related to the vanishing point; 23. The method of claim 19, further comprising:

24. The step of determining values ​​of the plurality of degrees of freedom comprises: determining a rotation-related degree of freedom based on the far-field region; determining a lateral movement-related degree of freedom based on the vanishing point-oriented near field; determining a forward / backward movement-related degree of freedom based on the non-vanishing-point-oriented near field; 24. The method of claim 23, comprising:

25. The step of determining values ​​of the plurality of degrees of freedom comprises: determining a rotation-related degree of freedom based on a far-distance vertex having an influence of a movement-related degree of freedom equal to or less than a first threshold value among the vertices included in the second image; determining a degree of freedom related to movement in a left-right direction based on vanishing-point-oriented near vertices, among the near vertices excluding the far vertices from the second image, in which an influence of a degree of freedom related to movement in a forward-backward direction is equal to or less than a second threshold; determining a movement-related degree of freedom in the forward / backward direction based on non-vanishing point-oriented close-distance vertices excluding the vanishing point-oriented close-distance vertices among the close-distance vertices; 20. The method of claim 19, comprising:

26. The positioning method further comprises: determining a virtual object on the map data for an augmented reality service; displaying the virtual object based on the determined values ​​of the degrees of freedom; 20. The method of claim 19, comprising:

27. The input image includes an image of a vehicle running, The positioning method according to claim 26 , wherein the virtual object indicates driving route information.

28. A computer program stored on a medium and including instructions, comprising: The instructions, when executed by hardware, cause the hardware to perform the method of any one of claims 19 to 27. Computer program.

Images