Apparatus and method for simultaneous localization and mapping

Abstract

The present invention introduces an apparatus and a method for simultaneous localization and mapping, the apparatus comprising: a sensor unit that photographs the surroundings of a moving object and is capable of adjusting a parameter for a photographing condition; a parameter adjustment unit for adjusting the parameter so that a visual field condition is satisfied; and a visual map generating unit for generating a visual map configured such that one or more features each having a plurality of models are connected to each other, wherein the location of the moving object is estimated by matching between an extracted feature and the feature of the visual map.

Description

Simultaneous position estimation and mapping device and method

[0001] The present invention relates to a simultaneous position estimation and map creation device and method that are robust to various environmental conditions.

[0002]

[0003] Simultaneous localization and mapping (SLAM) technology is a technology that simultaneously creates a map of the surrounding environment while autonomous vehicles, robots, etc. are moving, and estimates the location of autonomous vehicles, robots, etc. on the created map.

[0004] Simultaneous position estimation and mapping technology can help autonomous vehicles, robots, etc. to identify their locations even in unknown environments, and based on this, can help autonomous vehicles, robots, etc. to create movement paths or avoid obstacles.

[0005] For this simultaneous location estimation and map creation, images of the surrounding environment acquired through sensors such as cameras can be utilized, and the performance of location estimation and map creation depends on the quality of the images.

[0006] In general, simultaneous position estimation and map creation assume ideal conditions such as little or no occlusion or changes in the surrounding environment. However, depending on the actual application environment, there may be limitations such as the existence of occlusion areas due to surrounding obstacles or changes in illumination.

[0007] Therefore, to improve the performance of simultaneous location estimation and map creation, a method needs to be proposed that can mitigate the impact of changes in the surrounding environment.

[0008]

[0009] The matters described as background technology above are only intended to enhance understanding of the background of the present invention, and should not be taken as an acknowledgment that they correspond to prior art already known to those skilled in the art.

[0010]

[0011] An object of the present invention is to provide a simultaneous position estimation and map creation device and method that are robust to various environmental conditions by adjusting parameters for shooting conditions and reflecting various parameters in features for position estimation and map creation.

[0012]

[0013] The tasks of the present invention are not limited to the tasks mentioned above, and other tasks not mentioned will be clearly understood by those skilled in the art from the description below.

[0014]

[0015] According to one embodiment of the present invention for realizing the above-described task, a simultaneous position estimation and map creation device includes: a sensor unit including at least one camera equipped on a mobile body to photograph the surroundings of the mobile body while the mobile body is moving, and capable of adjusting parameters for photographing conditions; a parameter adjustment unit to determine whether a field of view condition according to the at least one parameter is satisfied with respect to an image collected through the photographing, and to adjust the parameter so that the field of view condition is satisfied; a visual map generation unit to generate a visual map in which one or more features having a plurality of models are connected according to the position and parameter of the at least one camera; and a position estimation unit to estimate the position of the mobile body through matching between features extracted from an image satisfying the field of view condition and features of the visual map, and to output the estimated position of the mobile body.

[0016] For example, the method further includes a feature map generation unit that generates a feature map for each of the at least one camera by linking the extracted feature with a parameter corresponding to each of the at least one camera, and the visual map generation unit can generate a visual map based on a relative position between the feature map and the at least one camera.

[0017] For example, the at least one camera may be provided at different locations on the mobile body, and each camera may photograph the surroundings of the mobile body in different directions.

[0018] For example, the at least one camera may be provided on at least one of the front, side, rear, and front-side of the mobile body.

[0019] For example, the above image may be added with time information according to the shooting time.

[0020] For example, the above parameters may include at least one of focus, focal length, exposure, and gain.

[0021] For example, the viewing condition may be determined based on at least one of the occlusion range of the image, the number of objects included in the image, and the brightness of the image.

[0022] For example, the position estimation unit can determine the reliability of the estimated position of the moving object based on the number of extracted features and the number of features on the visual map corresponding to the extracted features, and output the position of the estimated moving object based on the determined reliability.

[0023] For example, the position estimation unit can further output route guidance information based on the estimated position of the moving object.

[0024] For example, the route guidance information may include waypoints for reaching a destination.

[0025] For example, the visual map generation unit can add a model for each feature on the visual map based on the extracted features.

[0026] For example, the visual map generation unit may replace a model for each feature on the visual map so that the number of models does not exceed the preset value based on the weight of each model when the number of models reaches a preset value.

[0027] For example, each of the plurality of models may be defined according to the location of the feature on the visual map, a descriptor corresponding to the surrounding characteristics of the location information, and the parameters.

[0028] For example, the position estimation unit can determine the relative position between the extracted feature and the feature of the visual map through matching between the feature extracted from the image satisfying the field of view condition and the feature of the visual map, and estimate the position of the moving object based on the determined relative position.

[0029]

[0030] According to one embodiment of the present invention for realizing the above-described task, a simultaneous location estimation and map creation method comprises the steps of: collecting images through a sensor unit including at least one camera equipped on a mobile body to photograph the surroundings of the mobile body while the mobile body is moving and capable of adjusting parameters for photographing conditions; determining whether a field of view condition is satisfied for an image collected through the photographing based on the at least one parameter, and adjusting the parameter so that the field of view condition is satisfied; generating a visual map in which at least one feature having a plurality of models is connected according to the position and parameter of the at least one camera; and estimating a location of the mobile body through matching between a feature extracted from an image satisfying the field of view condition and a feature of the visual map, and outputting the estimated location of the mobile body.

[0031]

[0032] According to various embodiments of the present invention as described above, a wider field of view can be secured by adjusting parameters for shooting conditions, and the quality of images for feature extraction can be improved by alleviating the influence of changes in illumination of the surrounding environment.

[0033] Additionally, improving the quality of images can improve the performance of feature extraction, which can secure the performance of location estimation and map creation even in open spaces where feature extraction is difficult, and can help driving in unknown environments where coordinate or map information is not available.

[0034] Additionally, by reflecting various parameters in the features, location estimation and map creation performance can be improved for various environmental conditions.

[0035]

[0036] *

[0037] The effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by a person having ordinary skill in the art to which the present invention belongs from the description below.

[0038]

[0039] FIG. 1 is a diagram showing the configuration of a simultaneous position estimation and map creation device according to one embodiment of the present invention.

[0040] FIG. 2 is a diagram for explaining a feature extraction process according to one embodiment of the present invention.

[0041] FIG. 3 is a flowchart of a simultaneous location estimation and map creation process according to one embodiment of the present invention.

[0042]

[0043] Specific structural and functional descriptions of the embodiments of the present invention disclosed in this specification or application are merely illustrative for the purpose of explaining the embodiments according to the present invention, and the embodiments according to the present invention may be implemented in various forms and should not be construed as limited to the embodiments described in this specification or application.

[0044] Since embodiments of the present invention can be modified in various ways and take various forms, specific embodiments are illustrated in the drawings and described in detail in this specification or application. However, this is not intended to limit embodiments of the present invention to specific disclosed forms, and it should be understood that all modifications, equivalents, and alternatives fall within the spirit and technical scope of the present invention.

[0045] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and shall not be construed in an idealized or overly formal sense unless explicitly defined herein.

[0046] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components will be given the same reference numbers and redundant descriptions thereof will be omitted.

[0047] In the description of the following embodiments, the term "pre-set" means that when a parameter is used in a process or algorithm, the value of the parameter is predetermined. Depending on the embodiment, the value of the parameter may be set when the process or algorithm starts or may be set during the execution of the process or algorithm.

[0048] The suffixes "module" and "part" used for components in the following description are given or used interchangeably only for the convenience of writing specifications, and do not have distinct meanings or roles in themselves.

[0049] In describing the embodiments disclosed in this specification, detailed descriptions of related known technologies will be omitted if it is determined that such detailed descriptions may obscure the gist of the embodiments disclosed in this specification. In addition, the attached drawings are provided solely to facilitate understanding of the embodiments disclosed in this specification, and the technical concepts disclosed in this specification are not limited by the attached drawings, and should be understood to include all modifications, equivalents, and substitutes included within the spirit and technical scope of the present invention.

[0050] Terms that include ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0051] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0052] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0053] In this specification, terms such as “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but should be understood not to exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0054]

[0055] The simultaneous position estimation and map creation device and method according to embodiments of the present invention propose to improve the quality of an image for feature extraction by adjusting parameters for shooting conditions and to enable simultaneous position estimation and map creation to be robust under various environmental conditions by reflecting various parameters to the extracted features.

[0056] Before describing a simultaneous location estimation and map creation method according to an embodiment of the present invention, the configuration of a simultaneous location estimation and map creation device according to an embodiment of the present invention will first be described with reference to FIG. 1.

[0057]

[0058] FIG. 1 is a diagram showing the configuration of a simultaneous position estimation and map creation device according to one embodiment of the present invention.

[0059] Referring to FIG. 1, the simultaneous position estimation and map creation device (10) may include a sensor unit (100), a parameter adjustment unit (200), a feature map creation unit (300), a position estimation unit (400), and a visual map creation unit (500). However, FIG. 1 mainly illustrates components related to the description of one embodiment of the present invention, and it is obvious that the actual simultaneous position estimation and map creation device (10) may be implemented by including more or fewer components than this.

[0060] Below, each component is explained.

[0061] First, the sensor unit (100) is equipped on a mobile body and captures the surroundings of the mobile body while the mobile body is moving, and includes at least one camera capable of adjusting parameters for shooting conditions.

[0062] Here, the parameters for the shooting conditions may include at least one of focus, focal length, exposure, and gain, and the resolution, sharpness, brightness, magnification, etc. of the captured image may vary depending on the parameters for the shooting conditions.

[0063] Additionally, at least one camera is installed at different locations on the mobile device, each capable of capturing images of the mobile device's surroundings from different directions. The more directions the cameras capture, the wider the field of view of the images for position estimation and map creation, allowing for more features to be extracted.

[0064] To this end, a camera may be installed at least in the front, side, rear, and front-side of the moving object, and cameras installed in different locations may capture images having non-overlapping field of view ranges based on the moving object.

[0065] Meanwhile, when images are collected through multiple cameras in this way, time information according to the shooting time can be added to the collected images.

[0066] Temporal information may include, for example, the time an image was captured or stored, or the order in which it was captured or stored.

[0067] The above time information can be utilized to classify images with the same or corresponding time information among images collected through multiple cameras.

[0068] In particular, images classified according to time information can be combined to derive a full field of view of the surrounding environment of the moving object.

[0069]

[0070] The parameter adjustment unit (200) can determine whether the field of view conditions are satisfied according to the parameters for the images collected through shooting, and adjust the parameters so that the field of view conditions are satisfied.

[0071] Here, the field of view condition can be understood as a criterion for evaluating the reliability of the collected image, and whether the field of view condition is satisfied can depend on parameters.

[0072] That is, the parameter adjustment unit (200) can determine the reliability of the collected image by determining whether the field of view conditions are satisfied, and can adjust the shooting conditions of the camera to improve the reliability of the collected image.

[0073] In this case, the viewing conditions can be determined based on at least one of the occlusion range of the image, the number of objects included in the image, and the brightness of the image.

[0074] For example, the parameter adjustment unit (200) may determine that the field of view condition is not satisfied when the occlusion range in the collected image exceeds a preset ratio in the entire range of the image, the number of objects included in the image is less than a preset number, the brightness of the image is outside the preset brightness range, or the sharpness is less than a preset value.

[0075] In this case, when the field of view condition is not satisfied, the parameter adjustment unit (200) can adjust the parameters so that the field of view condition is satisfied.

[0076] For example, the parameter adjustment unit (200) can adjust the focus of the camera so that the sharpness satisfies the viewing conditions, or adjust the gamma so that the brightness of the image satisfies the viewing conditions.

[0077] In this way, the parameter adjustment unit (200) can adjust parameters for shooting conditions in response to changes in the surrounding environment, thereby enabling the collection of images for location estimation and map creation effectively even in various environments.

[0078]

[0079] Meanwhile, the feature map generation unit (300) can generate a feature map for at least one camera by linking the parameters corresponding to at least one camera and the features extracted from the collected images.

[0080] The number of feature maps generated can be determined based on the total number of cameras, and each feature map can include features extracted from images captured by the corresponding camera, and each feature can be associated with parameters for the camera corresponding to the feature map.

[0081]

[0082] The position estimation unit (400) can estimate the position of a moving object through matching between features extracted from an image that satisfies the field of view conditions and features of a visual map generated through the visual map generation unit (500).

[0083] More specifically, the position estimation unit (400) can determine the relative position between the extracted feature and the feature of the visual map through matching between the feature extracted from the image satisfying the view condition and the feature of the visual map.

[0084] When the relative position between the feature extracted from the image and the feature of the visual map is determined, the position estimation unit (400) can estimate the position of the moving object based on the determined relative position.

[0085] When the location of the moving object is estimated, the location estimation unit (400) can output the estimated location of the moving object.

[0086] Additionally, the location estimation unit (400) can determine the reliability of the estimated location of the moving object based on the number of extracted features and the number of features on the visual map corresponding to the extracted features.

[0087] The position estimation unit (400) can output the estimated position of the moving object based on the determined reliability, thereby improving the position estimation performance of the moving object.

[0088] Furthermore, the position estimation unit (400) not only provides the estimated position of the moving object, but can also output further route guidance information based on the estimated position of the moving object.

[0089] Such route guidance information may include waypoints for reaching a destination, thereby helping a vehicle to move efficiently toward its destination even in conditions where GPS (Global Positioning System) or other navigational aids are not available.

[0090]

[0091] Meanwhile, the visual map may be composed of one or more features having multiple models connected according to the position and parameters of at least one camera.

[0092] More specifically, each of the multiple models for a feature can be defined according to the location of the feature on the visual map, descriptors corresponding to the surrounding characteristics of the location information, and parameters.

[0093] For example, one model (X1) among multiple models for a feature can be expressed as follows.

[0094] X1=(P1,[(D1,C1)1,(D1,C1) 2,..., (D1,C1) m ])

[0095] Here, P can have values ​​for the x, y, and z axis coordinates on the visual map for the model, D can have values ​​for the parameters of the camera, and C can have values ​​for the surrounding characteristics of the location information.

[0096] Additionally, the visual map generation unit (500) can generate a visual map based on the feature map generated by the feature map generation unit (300) and the relative position between at least one camera.

[0097] For example, the visual map generation unit (500) can determine the relative positions between each feature map based on the relative positions between cameras, and generate a visual map by combining features included in each feature map based on the determined relative positions.

[0098] Additionally, the visual map generation unit can add a model for each feature on the visual map based on the extracted features.

[0099] In this case, when the number of models reaches a preset value, the visual map generation unit can replace the model for each feature on the visual map based on the weight of each model so that the number of models does not exceed the preset value.

[0100] Each component of the simultaneous position estimation and map creation device (10) can be implemented by including a communication device that communicates with a sensor to perform the functions described above, a memory that stores an operating system or logic commands and input / output information, and one or more processors that perform judgments, calculations, decisions, etc. necessary for controlling the responsible function.

[0101] Below, the feature extraction process will be described with reference to Fig. 2.

[0102]

[0103] FIG. 2 is a diagram for explaining a feature extraction process according to one embodiment of the present invention.

[0104] Referring to FIG. 2, cameras may be provided at the front, front side, side, and rear of the moving device, and each camera may capture the surrounding environment in different directions based on the moving device.

[0105] Features can be extracted from images captured by each camera, and can be extracted based on objects included in the captured images, for example.

[0106] Feature maps (Feature map #1-#3) can be generated for each camera, and each feature map (Feature map #1-#3) can contain features extracted from images captured by each camera.

[0107] Additionally, each feature map (Feature map #1-#3) can reflect the parameters of the corresponding camera (Camera Parameters #1-#3).

[0108] Below, the process of simultaneous location estimation and map creation described so far will be explained with reference to Fig. 3.

[0109]

[0110] FIG. 3 is a flowchart of a simultaneous location estimation and map creation process according to one embodiment of the present invention.

[0111] Referring to FIG. 3, first, the parameter adjustment unit (200) collects images captured through each camera of the sensor unit (100) and parameters for each camera (S301), and can determine whether the collected images satisfy the field of view conditions (S302).

[0112] If the field of view condition is not satisfied (No in S302), the parameter adjustment unit (200) adjusts the parameters so that the field of view condition is satisfied (S303), and if the field of view condition is satisfied (Yes in S302), the parameter adjustment unit (200) stores the parameters (S304).

[0113] The feature map generation unit (300) generates a feature map including features extracted from an image that satisfies a field of view condition (S305), and the position estimation unit (400) matches the generated feature map and the field of view map (S306) to determine the relative position between features (S307), and estimates the position of the moving object based on the determination result (S308).

[0114] Thereafter, the visual map generation unit (500) can update the features of the visual map using the newly extracted features, and the updated visual map can be used again for location estimation.

[0115]

[0116] According to various embodiments of the present invention as described above, a wider field of view can be secured by adjusting parameters for shooting conditions, and the quality of images for feature extraction can be improved by alleviating the influence of changes in illumination of the surrounding environment.

[0117] Additionally, improving the quality of images can improve the performance of feature extraction, which can secure the performance of location estimation and map creation even in open spaces where feature extraction is difficult, and can help driving in unknown environments where coordinate or map information is not available.

[0118] Additionally, by reflecting various parameters in the features, location estimation and map creation performance can be improved for various environmental conditions.

[0119]

[0120] Although the present invention has been illustrated and described with respect to specific embodiments thereof as described above, it will be apparent to those skilled in the art that the present invention may be variously improved and modified without departing from the technical spirit of the present invention as defined by the following claims.

[0121]

[0122] [Explanation of symbols]

[0123] 100: Sensor section

[0124] 200: Parameter adjustment section

[0125] 300: Feature map generation unit

[0126] 400: Location estimation unit

[0127] 500: Visual Map Generation Unit

Claims

1. A sensor unit including at least one camera equipped on a mobile body to photograph the surroundings of the mobile body while the mobile body is moving and capable of adjusting parameters for photographing conditions; A parameter adjustment unit that determines whether a field of view condition is satisfied according to at least one parameter for an image collected through the above shooting, and adjusts the parameter so that the field of view condition is satisfied; A visual map generation unit generating a visual map in which one or more features having multiple models are connected according to the position and parameters of at least one camera; and A simultaneous position estimation and map creation device including a position estimation unit that estimates the position of the moving object through matching between features extracted from an image satisfying the above-mentioned field of view conditions and features of the visual map, and outputs the estimated position of the moving object.

2. In claim 1, Further comprising a feature map generation unit that generates a feature map for each of the at least one camera by linking the extracted feature with a parameter corresponding to each of the at least one camera, The above visual map generation unit, A simultaneous position estimation and map creation device characterized in that it generates a visual map based on the relative position between the feature map and the at least one camera.

3. In claim 1, At least one camera above, A simultaneous position estimation and map creation device characterized in that it is provided at different locations of the above-mentioned mobile body and each of the devices photographs the surroundings of the above-mentioned mobile body in different directions.

4. In claim 2, At least one camera above, A simultaneous position estimation and map creation device characterized in that it is provided on at least one of the front, side, rear and front side of the above-mentioned moving body.

5. In claim 2, In the image above, A simultaneous position estimation and map creation device characterized by adding time information according to the shooting time.

6. In claim 1, The above parameters are, A simultaneous position estimation and mapping device characterized by including at least one of focus, focal length, exposure and gain.

7. In claim 1, The above viewing conditions are, A simultaneous location estimation and map creation device characterized in that the determination is made based on at least one of the occlusion range of the image, the number of objects included in the image, and the brightness of the image.

8. In claim 1, The above location estimation unit, A simultaneous location estimation and map creation device characterized in that the reliability of the estimated location of the moving object is determined based on the number of extracted features and the number of features on the visual map corresponding to the extracted features, and the location of the estimated moving object is output based on the determined reliability.

9. In claim 1, The above location estimation unit, A simultaneous position estimation and map creation device characterized in that it further outputs route guidance information based on the estimated position of the moving object.

10. In claim 9, The above route guidance information is, A simultaneous position estimation and mapping device characterized by including waypoints for reaching a destination.

11. In claim 1, The above visual map generation unit, A simultaneous location estimation and map creation device characterized by adding a model for each feature on the visual map based on the extracted features.

12. In claim 10, The above visual map generation unit, A simultaneous location estimation and map creation device characterized in that when the number of the above models reaches a preset value, the model for each feature on the visual map is replaced so that the number of the models does not exceed the preset value based on the weight of each model.

13. In claim 1, Each of the above multiple models, A simultaneous location estimation and map creation device characterized in that the location of the feature on the visual map, a descriptor corresponding to the surrounding characteristics of the location information, and the parameters are defined.

14. In claim 1, The above location estimation unit, A simultaneous position estimation and map creation device characterized in that the relative position between the extracted feature and the feature of the visual map is determined through matching between the feature extracted from the image satisfying the above field of view condition and the feature of the visual map, and the position of the moving object is estimated based on the determined relative position.

15. A step of collecting images through a sensor unit including at least one camera equipped on a mobile body and capable of photographing the surroundings of the mobile body while the mobile body is moving and adjusting parameters for photographing conditions; A step of determining whether a field of view condition is satisfied for an image collected through the shooting based on at least one parameter, and adjusting the parameter so that the field of view condition is satisfied; A step of generating a visual map in which one or more features having multiple models are connected according to the position and parameters of at least one camera; and A method for simultaneous position estimation and map creation, comprising: a step of estimating the position of the moving object through matching between features extracted from an image satisfying the above-mentioned field of view conditions and features of the visual map, and outputting the position of the estimated moving object.

Images