Device and method for synchronous positioning and mapping

Abstract

The present invention introduces an apparatus and a method for synchronous positioning and mapping, the apparatus comprising: a sensor unit that captures an environment surrounding a moving object and is capable of adjusting parameters of a capture condition; the parameter adjusting unit is used for adjusting parameters to meet the visual field condition; and a vision map generation unit that generates a vision map configured such that one or more features, each having a plurality of models, are connected to each other, in which the position of the moving object is estimated by matching between the extracted features and the features of the vision map.

Description

Technical Field

[0001] This invention relates to a synchronous positioning and mapping device and method that is robust to various environmental conditions. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) is a technique that creates a map of the surrounding environment and simultaneously estimates the position of the autonomous vehicle, robot, etc., on the created map while the vehicle or robot is moving.

[0003] Based on simultaneous localization and mapping (SLAM) technology, the location of autonomous vehicles, robots, etc. can be identified even in unknown environments. Furthermore, it can assist autonomous vehicles and robots in their movement, such as generating movement paths or avoiding obstacles.

[0004] For this type of simultaneous localization and mapping, images of the surrounding environment can be obtained using sensors such as cameras, and the performance of position estimation and mapping varies with image quality.

[0005] Typically, simultaneous localization and mapping (SMR) assumes ideal conditions such as minimal obstruction or change in the surrounding environment. However, depending on the actual application environment, there may be limitations such as obstructed areas caused by surrounding obstacles or changes in lighting.

[0006] Therefore, in order to improve the performance of simultaneous localization and mapping, it is necessary to propose a method that can mitigate the impact of changes in the surrounding environment.

[0007] The above description of the background art is merely for enhancing the understanding of the background of the present invention and should not be construed as an admission that this content corresponds to the prior art known to those skilled in the art. Summary of the Invention

[0008] Technical issues The purpose of this invention is to provide a synchronous positioning and mapping apparatus and method that are robust under various environmental conditions by adjusting the parameters of the shooting conditions and reflecting various parameters in the features used for position estimation and mapping.

[0009] The problems of this invention are not limited to those described above, and other problems not mentioned will be clearly understood by those skilled in the art through the following description.

[0010] Technical solution According to an embodiment of the present invention, a synchronous positioning and mapping device for achieving the above-mentioned objective includes: a sensor unit disposed on a moving object, for capturing images of the surrounding environment of the moving object during its movement, and including at least one camera capable of adjusting parameters of the capturing conditions; a parameter adjustment unit for determining, based on at least one parameter, the satisfaction state of the field of view conditions of the captured images, and adjusting the parameters to satisfy the field of view conditions; a visual map generation unit for generating a visual map configured to be composed of one or more features, each feature having multiple models based on the position and parameters of at least one camera; and a position estimation unit for estimating the position of the moving object by matching features extracted from the images satisfying the field of view conditions with features of the visual map, and outputting the estimated position of the moving object.

[0011] For example, the apparatus may further include: a feature map generation unit that generates a feature map for each of the at least one camera by associating parameters corresponding to each of the at least one camera with the extracted features, and a visual map generation unit that can generate a visual map based on the relative position between the feature map and the at least one camera.

[0012] For example, at least one camera can be set at different locations on the moving object, with each camera capturing images of the surrounding environment from different directions.

[0013] For example, at least one camera may be positioned at at least one of the front, side, back, and front ramp of the moving object.

[0014] For example, time information based on the time the image was captured can be added to it.

[0015] For example, parameters may include at least one of focus, focal length, exposure, and gain.

[0016] For example, the field of view conditions can be determined based on at least one of the following: the extent of occlusion in the image, the number of objects contained in the image, and the brightness of the image.

[0017] For example, the position estimation unit can determine the reliability of the estimated position of a moving object based on the number of extracted features and the number of features on the visual map corresponding to the extracted features, and output the estimated position of the moving object based on the determined reliability.

[0018] For example, the position estimation unit can further output path guidance information based on the estimated position of the moving object.

[0019] For example, route guidance information may include waypoints to the destination.

[0020] For example, the visual graph generation unit can add a model to each feature on the visual graph based on the extracted features.

[0021] For example, when the number of models reaches a predetermined value, the visual graph generation unit can replace the model of each feature on the visual graph based on the weight of each model, so that the number of models does not exceed the predetermined value.

[0022] For example, each of the multiple models can be defined based on the location of features on the visual graph, descriptors corresponding to the surrounding environmental characteristics of the location information, and parameters.

[0023] For example, a position estimation unit can determine the relative position between the extracted features and the features of the visual image by matching the features extracted from the image that satisfies the visual conditions with the features of the visual image, and estimate the position of the moving object based on the determined relative position.

[0024] According to an embodiment of the present invention, a synchronous localization and mapping method for achieving the above-mentioned objective includes: acquiring images via a sensor unit disposed on a moving object, capturing images of the surrounding environment of the moving object during its movement, and including at least one camera capable of adjusting parameters of the capturing conditions; determining, based on at least one parameter, the satisfaction state of the field of view conditions of the captured images, and adjusting the parameters to satisfy the field of view conditions; generating a visual map configured to be composed of one or more features, each feature having multiple models based on the position and parameters of at least one camera; and estimating the position of the moving object by matching features extracted from the images satisfying the field of view conditions with features of the visual map, and outputting the estimated position of the moving object.

[0025] Technical effect As described above, according to various embodiments of the present invention, a wider field of view can be obtained by adjusting the parameters of the shooting conditions, and the quality of the feature extraction image can be improved by mitigating the influence of changes in ambient lighting.

[0026] In addition, improving image quality can enhance feature extraction performance, thereby ensuring location estimation and mapping performance even in open spaces where feature extraction is difficult, and facilitating driving in unknown environments without coordinate or map information.

[0027] In addition, by incorporating various parameters into the features, the performance of location estimation and mapping under various environmental conditions can be improved.

[0028] The effects that can be obtained by the present invention are not limited to those described above. For those skilled in the art, other effects not mentioned will be clearly understood from the following description. Attached Figure Description

[0029] Figure 1 This is a diagram illustrating the configuration of a synchronous positioning and mapping device according to an embodiment of the present invention.

[0030] Figure 2 This is a diagram illustrating the feature extraction process according to an embodiment of the present invention.

[0031] Figure 3 This is a flowchart of the synchronous positioning and mapping process according to an embodiment of the present invention. Detailed Implementation

[0032] The specific structural or functional descriptions of the embodiments of the present invention disclosed in this specification or application are for illustrative purposes only. The embodiments of the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described in this specification or application.

[0033] Since embodiments of the present invention can be modified and take various forms, specific embodiments are illustrated in the accompanying drawings and described in detail in this specification or application. However, this is not intended to limit the embodiments according to the concepts of the present invention to the specific forms disclosed, but should be understood to include all variations, equivalents, and alternatives contained within the spirit and scope of the present invention.

[0034] Unless otherwise defined, all terms used herein, including technical and scientific terms, shall have the meaning commonly understood by one of ordinary skill in the art to which this invention pertains. Terms such as those defined in common dictionaries shall be interpreted as having a meaning consistent with their meaning in the context of the relevant technical field, and shall not be interpreted as having an idealized or overly formal meaning unless expressly defined herein.

[0035] In the following description, the embodiments disclosed herein will be described in detail with reference to the accompanying drawings, but the same or similar parts will be referred to by the same reference numerals and are not related to the figure symbols, and redundant descriptions will be omitted.

[0036] In the following description of the embodiments, the term "predetermined" means that the value of a parameter is predetermined when used in a process or algorithm. The value of the parameter can be set when the process or algorithm starts, or it can be set during time intervals when the process or algorithm is executed according to the embodiments.

[0037] For ease of writing instructions, the component suffixes “module” and “unit” used in the following description can be used interchangeably, and they do not have any distinguishing meaning or function.

[0038] In describing the embodiments disclosed in this specification, detailed descriptions of related known technologies are omitted if it is determined that such detailed descriptions might obscure the essential aspects of the embodiments disclosed in this specification. Furthermore, the accompanying drawings are provided only to facilitate understanding of the embodiments disclosed in this specification. The technical concepts disclosed in this specification are not limited to the drawings but should be understood to include all variations, equivalents, and alternatives encompassed within the spirit and scope of the invention.

[0039] Terms including first and second ordinal numbers can be used to describe various components, but these components are not limited by these terms. These terms are only used to distinguish one component from another.

[0040] When a component is said to "connect to" or "access" another component, it should be understood that it can directly connect to or access the other component, but there may be other components between them. On the other hand, when a component is said to "directly connect to" or "directly access" another component, it should be understood that there are no other components between them.

[0041] Unless the context clearly indicates otherwise, singular expressions include plural expressions.

[0042] In this specification, terms such as “comprising” or “having” are intended to indicate the presence of the features, quantities, steps, operations, components, parts or combinations thereof described in this specification, but should not be construed as excluding the possibility of the presence or addition of one or more other features, quantities, steps, operations, components, parts or combinations thereof.

[0043] The simultaneous localization and mapping apparatus and method according to embodiments of the present invention aim to improve the quality of feature-extracted images by adjusting the parameters of shooting conditions, and to make simultaneous localization and mapping robust under various environmental conditions by reflecting various parameters on the extracted features.

[0044] In the following description of the synchronous positioning and mapping method according to embodiments of the present invention, reference will first be made to... Figure 1 The configuration of the synchronous positioning and mapping device according to an embodiment of the present invention is described.

[0045] Figure 1 This is a diagram illustrating the configuration of a synchronous positioning and mapping device according to an embodiment of the present invention.

[0046] like Figure 1 As shown, the synchronous positioning and mapping device 10 may include a sensor unit 100, a parameter adjustment unit 200, a feature map generation unit 300, a position estimation unit 400, and a visual map generation unit 500. However, Figure 1 The components shown are primarily related to the description of embodiments of the present invention; the actual synchronous positioning and mapping device 10 may be implemented to include more or fewer components.

[0047] The components will be described below.

[0048] First, the sensor unit 100 is disposed on the moving object to capture images of the surrounding environment of the moving object during its movement, and includes at least one camera capable of adjusting parameters for the shooting conditions.

[0049] Here, the parameters of the shooting conditions may include at least one of focus, focal length, exposure, and gain, and the resolution, sharpness, brightness, magnification / reduction ratio, etc. of the captured image may vary depending on the parameters of the shooting conditions.

[0050] In addition, at least one camera can be positioned at different locations on the moving object, with each camera capturing images of the object's surroundings from different directions. As the number of camera shooting directions increases, the images used for position estimation and mapping can have a wider field of view, and more features can be extracted from these images.

[0051] For this purpose, the camera can be set at at least one of the front, side, back and front slope of the moving object, and the cameras set at different positions can capture images based on the non-overlapping field of view of the moving object.

[0052] On the other hand, when images are captured by multiple cameras in this way, time information based on the shooting time can be added to the captured images.

[0053] Time information may include, for example, the time when the image was captured or stored, and the order in which it was captured or stored.

[0054] This type of time information can be used to classify images captured by multiple cameras that have the same or corresponding time information added to them.

[0055] In particular, images classified according to time information can be combined to obtain the complete field of view of the environment around a moving object.

[0056] The parameter adjustment unit 200 can determine the satisfaction status of the field of view conditions based on the parameters of the captured image, and adjust the parameters to satisfy the field of view conditions.

[0057] Here, field of view conditions can be understood as a standard for evaluating the reliability of the acquired image, and the satisfaction of field of view conditions can depend on parameters.

[0058] In other words, the parameter adjustment unit 200 can determine the reliability of the acquired image by determining the state of the field of view conditions, and can adjust the shooting conditions of the camera to improve the reliability of the acquired image.

[0059] In this case, the field of view conditions can be determined based on at least one of the following: the occlusion range of the image, the number of objects contained in the image, and the brightness of the image.

[0060] For example, if the proportion of the occlusion area of ​​the acquired image to the entire image area exceeds a predetermined proportion, the number of objects in the image is less than a predetermined number, the image brightness deviates from the predetermined brightness range, or the clarity is lower than a predetermined value, the parameter adjustment unit 200 can determine that the field of view conditions are not met.

[0061] When the field of view conditions are not met in this way, the parameter adjustment unit 200 can adjust the parameters to meet the field of view conditions.

[0062] For example, the parameter adjustment unit 200 can adjust the focus of the camera to make the sharpness meet the field of view conditions, or it can adjust the gamma value to make the image brightness meet the field of view conditions.

[0063] In this way, the parameter adjustment unit 200 can adjust the parameters of the shooting conditions in response to changes in the surrounding environment, thereby effectively acquiring images for location estimation and mapping even in various environments.

[0064] On the other hand, the feature map generation unit 300 can associate the parameters corresponding to each of the at least one camera with the features extracted from the acquired image to generate a feature map for each of the at least one camera.

[0065] The number of generated feature maps can be determined based on the total number of cameras. Each feature map can contain features extracted from the images captured by the corresponding camera, and each feature can be associated with the parameters of the camera corresponding to the feature map.

[0066] The position estimation unit 400 can estimate the position of a moving object by matching features extracted from an image that satisfies the visual conditions with features generated by the visual map generation unit 500.

[0067] More specifically, the position estimation unit 400 can determine the relative position between the extracted features and the features of the visual map by matching the features extracted from the image that satisfies the visual conditions with the features of the visual map.

[0068] After determining the relative position between the features extracted from the image and the features in the visual map, the position estimation unit 400 can estimate the position of the moving object based on the determined relative position.

[0069] After estimating the position of the moving object, the position estimation unit 400 can output the estimated position of the moving object.

[0070] In addition, the position estimation unit 400 can determine the reliability of the estimated position of the moving object based on the number of extracted features and the number of features on the visual map that correspond to the extracted features.

[0071] The position estimation unit 400 can output the estimated position of the moving object based on a certain reliability, thereby improving the position estimation performance of the moving object.

[0072] In addition, the position estimation unit 400 can not only provide the estimated position of the moving object, but also output path guidance information based on the estimated position of the moving object.

[0073] This type of route guidance information can include waypoints to the destination, thus helping moving objects move efficiently to their destination even when GPS (Global Positioning System) or similar technologies are unavailable.

[0074] On the other hand, a visual graph can be configured to be composed of one or more features connected together, each feature having multiple models based on the position and parameters of at least one camera.

[0075] More specifically, each of the multiple models of a feature can be defined based on the location of the feature on the visual graph, the descriptor corresponding to the characteristics of the surrounding environment of the location information, and the parameters.

[0076] For example, for any given feature, any one of the multiple models (X1) can be represented as follows: X1=(P1,[(D1,C1)1,(D1,C1)2,…,(D1,C1) m ]) Where P can be the x, y, z axis coordinates of the corresponding model on the visual graph, D can be the descriptor corresponding to the surrounding environmental features of the location information, and C can be the parameter values ​​of the camera.

[0077] In addition, the visual image generation unit 500 can generate a visual image based on the feature map generated by the feature map generation unit 300 and the relative position between at least one camera.

[0078] For example, the visual map generation unit 500 can determine the relative position between each feature map based on the relative position between the cameras, and generate a visual map by combining the features contained in each feature map according to the determined relative position.

[0079] In addition, the visual graph generation unit can add a model to each feature on the visual graph based on the extracted features.

[0080] In this scenario, when the number of models reaches a predetermined value, the visual graph generation unit can replace the model for each feature in the visual graph based on the weight of each model, ensuring that the number of models does not exceed the predetermined value.

[0081] Each component of the synchronous positioning and mapping device 10 can be implemented as a communication device for communicating with sensors, a memory for storing operating system or logic commands and input / output information, and one or more processors for performing judgments, calculations, decisions, etc., required to perform control functions to execute the aforementioned functions.

[0082] In the following text, reference will be made to Figure 2 Describe the feature extraction process.

[0083] Figure 2 This is a diagram illustrating the feature extraction process according to an embodiment of the present invention.

[0084] Reference Figure 2 The camera can be set on the front, front bevel, side and back of the mobile device, and each camera can capture the surrounding environment from different directions based on the mobile device.

[0085] Features can be extracted from images captured by each camera; for example, features can be extracted based on objects contained in the captured images.

[0086] Feature maps (feature maps #1-#3) can be generated for each camera, and each feature map (feature maps #1-#3) can contain features extracted from the images captured by each camera.

[0087] In addition, each feature map (feature map #1-#3) can reflect the parameters of the corresponding camera (camera parameters #1-#3).

[0088] In the following text, reference will be made to Figure 3 Describe the above synchronous positioning and mapping process.

[0089] Figure 3 This is a flowchart of the synchronous positioning and mapping process according to an embodiment of the present invention.

[0090] Reference Figure 3 First, in step S301, the parameter adjustment unit 200 acquires the images captured by each camera of the sensor unit 100 and the parameters of each camera, and in step S302, it determines the state in which the field of view conditions of the acquired images are satisfied.

[0091] When the field of view conditions are not met ("No" in step S302), the parameter adjustment unit 200 adjusts the parameters in step S303 to meet the field of view conditions; when the field of view conditions are met ("Yes" in step S302), the parameter adjustment unit 200 stores the parameters in step S304.

[0092] In step S305, the feature map generation unit 300 generates feature maps containing features extracted from an image that meets the visual conditions. In step S306, the position estimation unit 400 matches the generated feature maps with the visual map to determine the relative positions between the features in step S307, and estimates the position of the moving object based on the determination result in step S308.

[0093] Subsequently, the visual graph generation unit 500 can update the features of the visual graph using the newly extracted features, and the updated visual graph can be used again for location estimation.

[0094] As described above, according to various embodiments of the present invention, a wider field of view can be obtained by adjusting the parameters of the shooting conditions, and the quality of the feature extraction image can be improved by mitigating the influence of changes in ambient lighting.

[0095] In addition, improving image quality can enhance feature extraction performance, thereby ensuring location estimation and mapping performance even in open spaces where feature extraction is difficult, and facilitating driving in unknown environments without coordinate or map information.

[0096] In addition, by incorporating various parameters into the features, the performance of location estimation and mapping under various environmental conditions can be improved.

[0097] Although the present invention has been shown and described above through specific embodiments, it will be apparent to those skilled in the art that various modifications and alterations can be made to the present invention without departing from the spirit of the invention as defined in the appended claims.

[0098] Explanation of reference numerals in the attached figures 100: Sensor Unit 200: Parameter Adjustment Unit 300: Feature Map Generation Unit 400: Location Estimation Unit 500: Visual graph generation unit

Claims

1. A simultaneous localization and mapping apparatus comprising: a sensor unit disposed on a mobile object, which photographs a surrounding environment of the mobile object during movement of the mobile object, and includes at least one camera capable of adjusting parameters of a photographing condition; a parameter adjustment unit which determines a satisfaction state of a field of view condition of an image collected by photographing based on at least one parameter, and adjusts the parameter to satisfy the field of view condition; a visual map generation unit for generating a visual map, the visual map being connected by one or more features, each feature having a plurality of models according to a position and a parameter of the at least one camera; and a position estimation unit for estimating a position of the mobile object by matching between a feature extracted from an image satisfying the field of view condition and a feature of the visual map, and outputting an estimated position of the mobile object. 2.The apparatus according to claim 1, further comprising: a feature map generation unit for generating a feature map for each of the at least one camera by associating a parameter corresponding to each of the at least one camera with an extracted feature, wherein the visual map generation unit generates the visual map based on a relative position between the feature map and the at least one camera.

3. The simultaneous localization and mapping device of claim 1, wherein, The at least one camera is disposed at different positions of the mobile object, each camera photographing the surrounding environment of the mobile object from a different direction.

4. The simultaneous localization and mapping device of claim 2, wherein, The at least one camera is disposed at at least one of a front, a side, a back, and a front slope of the mobile object.

5. The simultaneous localization and mapping device of claim 2, wherein, Time information according to a photographing time is added to an image.

6. The simultaneous localization and mapping device of claim 1, wherein, The parameter includes at least one of focus, focal length, exposure, and gain.

7. The simultaneous localization and mapping device of claim 1, wherein, The field of view condition is determined based on at least one of an occlusion range of an image, a number of objects included in an image, and a brightness of an image.

8. The simultaneous localization and mapping device of claim 1, wherein, The position estimation unit determines reliability of an estimated position of the mobile object based on a number of extracted features and a number of features on a visual map corresponding to the extracted features, and outputs the estimated position of the mobile object based on the determined reliability.

9. The simultaneous localization and mapping device of claim 1, wherein, The position estimation unit further outputs path guidance information based on the estimated position of the mobile object.

10. The simultaneous localization and mapping device of claim 9, wherein, The path guidance information includes a path point to a destination.

11. The simultaneous localization and mapping device of claim 1, wherein, The visual map generation unit adds a model to each feature on the visual map based on an extracted feature.

12. The simultaneous localization and mapping device of claim 10, wherein, When a number of the models reaches a predetermined value, the visual map generation unit replaces the model of each feature on the visual map based on a weight of each model so that the number of the models does not exceed the predetermined value.

13. The simultaneous localization and mapping device of claim 1, wherein, Each of the plurality of models is defined according to a position of a feature on the visual map, a descriptor corresponding to a surrounding environment characteristic of position information, and the parameter.

14. The simultaneous localization and mapping device of claim 1, wherein, The position estimation unit determines a relative position between an extracted feature and a feature of the visual map by matching between the extracted feature and the feature of the visual map, and estimates a position of the mobile object based on the determined relative position. 15.A simultaneous localization and mapping method comprising: acquiring images by a sensor unit, the sensor unit being arranged on a moving object, taking surroundings of the moving object during movement of the moving object, and comprising at least one camera capable of adjusting parameters of a shooting condition; determining a satisfaction state of a field of view condition of the images acquired by the taking based on at least one parameter, and adjusting the parameters to satisfy the field of view condition; generating a visual graph, the visual graph being connected by one or more features, each feature having a plurality of models according to positions and parameters of the at least one camera; and estimating a position of the moving object by matching between features extracted from the images satisfying the field of view condition and features of the visual graph, and outputting the estimated position of the moving object.

Images