A map construction method, device and storage medium

Through visual tracking, we judge the common-visual relationship between image frames and submaps and create a new submap, which solves the problem that maps cannot be expanded in the SLAM system, improves the accuracy and usability of map construction, and is suitable for mobile devices.

CN112581514BActive Publication Date: 2024-11-26ZHEJIANG SENSETIME TECH DEV CO LTD
View PDF 13 Cites 0 Cited by

Patent Information

Application Number
CN201910944288.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-09-30
Publication Date
2024-11-26
Estimated Expiration
2039-09-30

AI Technical Summary

Technical Problem

In the SLAM system, during the map construction process, due to poor camera image quality, fast user movement and low IMU accuracy, the collected image keyframe pose has a large deviation, resulting in the map being unable to expand normally.

Method used

By visually tracking the current image frame and the current submap, we can determine whether there is a common view relationship. If there is no common view relationship, create a new submap and use the current image frame as the initial frame in the new submap to avoid a single The problem that maps cannot be scaled due to difficulty in visual tracking.

Benefits of technology

It effectively solves the problem that maps cannot be scaled, improves the accuracy and usability of map construction, reduces dependence on IMU, and is suitable for a wider range of mobile devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112581514B_ABST
    Figure CN112581514B_ABST
Patent Text Reader

Abstract

The present disclosure provides a map construction method and device, and a storage medium, wherein the map construction method includes: visually tracking the current image frame in combination with the current sub-map to obtain a visual tracking result; determining whether there is a common-viewing relationship between the current image frame and the current sub-map based on the visual tracking result; and creating a new sub-map in response to the fact that there is no common-viewing relationship between the current image frame and the current sub-map. The present disclosure creates a new sub-map when there is no common-viewing relationship between the current image frame and the current sub-map, so as to solve the problem that the map cannot be expanded once the visual tracking in a single map becomes difficult to continue. At the same time, the method proposed in the present disclosure makes full use of visual information, does not rely on the assistance of other sensor information such as IMU, and can be applied to a wider range of fields.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of map processing, and in particular to a map construction method and device, and a storage medium. Background Art

[0002] In the SLAM (Simultaneous Localization And Mapping) system, map construction is an important component. The map records the scene information that has been visited in some form. Consistent maps can reduce errors in the map construction process.

[0003] However, maintaining a consistent map is difficult, especially on mobile devices such as mobile phones. Due to the poor quality of images collected by the camera, the blurry images caused by users moving the mobile phone too quickly, and the low accuracy of the IMU (Inertial Measurement Unit) installed on the mobile device, the poses of some key frames of the collected images will have large deviations, resulting in the inability to expand the map normally. Summary of the invention

[0004] In view of this, the present disclosure provides a map construction method and device, and a storage medium to address the deficiencies in the related art.

[0005] According to a first aspect of an embodiment of the present disclosure, a map construction method is provided, the method comprising: performing visual tracking on a current image frame in combination with a current submap to obtain a visual tracking result; determining whether there is a common viewing relationship between the current image frame and the current submap based on the visual tracking result; and creating a new submap in response to the fact that there is no common viewing relationship between the current image frame and the current submap.

[0006] In the above embodiment, when there is no co-viewing relationship between the current image frame and the current sub-map, a new sub-map is created, which can solve the problem that the map cannot be expanded once the visual tracking in a single map becomes difficult to continue.

[0007] Optionally, determining whether there is a common viewing relationship between the current image frame and the current submap based on the visual tracking result includes: in response to the visual tracking result indicating that there are matching feature points between the current image frame and the current submap, and the number of matching feature points is greater than or equal to a first preset threshold, determining that there is a common viewing relationship between the current image frame and the current submap; otherwise, determining that there is no common viewing relationship between the current image frame and the current submap.

[0008] In the above embodiment, according to the fact that the matching feature points between the current image frame and the current submap are greater than or equal to the first preset threshold, it is determined that the current image frame and the current submap have a common viewing relationship, otherwise it is determined that the two do not have a common viewing relationship. The implementation is simple and the usability is high.

[0009] Optionally, determining whether there is a common-view relationship between the current image frame and the current submap based on the visual tracking result includes: in response to the visual tracking result indicating that there are matching feature points between the current image frame and the current submap, removing incorrectly matched first feature points from the matched feature points to obtain validly matched second feature points; in response to the number of the second feature points being greater than or equal to a second preset threshold, determining that there is a common-view relationship between the current image frame and the current submap, otherwise determining that there is no common-view relationship between the current image frame and the current submap.

[0010] In the above embodiment, in addition to determining whether there are matching feature points between the current image frame and the current sub-map, whether there is a common view relationship between the current image frame and the current sub-map is determined based on the number of effectively matching second feature points between the two, thereby reducing the possibility of misjudgment.

[0011] Optionally, after removing the incorrectly matched first feature points from the matched feature points, obtaining the effectively matched second feature points includes: inputting the current image frame and the image frame included in the current submap into a pre-established feature point matching model, removing the incorrectly matched first feature points from the matched feature points through the feature point matching model, and obtaining the effectively matched second feature points.

[0012] In the above embodiment, a feature point matching model can be established in advance, and the current image frame and the current submap can be input into the feature point matching model. The feature point matching model can remove the first feature points that are incorrectly matched between the current image frame and the current submap, thereby obtaining the second feature points that are effectively matched. In this way, the second feature points that are effectively matched can be quickly determined, which is simple to implement.

[0013] Optionally, the method further includes: in response to a co-viewing relationship between the current image frame and the current sub-map, expanding the current sub-map using the current image frame.

[0014] In the above embodiment, when there is a co-viewing relationship between the current image frame and the current sub-map, the current sub-map can be expanded based on the current image frame to improve the map accuracy.

[0015] Optionally, after expanding the current submap using the current image frame, the method further includes: merging the current submap with at least one of the other submaps in response to a co-viewing relationship between the current image frame and at least one of the other submaps.

[0016] In the above embodiment, when there is a co-viewing relationship between the current image frame and at least one other sub-map, the current sub-map may be merged with the at least one other sub-map to improve the map accuracy.

[0017] Optionally, the creating a new sub-map includes: creating a new sub-map with empty content; and in the new sub-map with empty content, using the current image frame as an initial frame of the new sub-map.

[0018] The above embodiment can use the current image frame as the initial frame of the new submap in the new submap whose content is empty, so that the new submap can be expanded later according to the collected new current image frame, thereby avoiding the problem of using a single map in the related technology and the map being unable to be expanded once a situation where visual tracking is difficult occurs.

[0019] Optionally, after creating the new submap, the method further includes: performing visual tracking on the new current image frame in combination with the new submap to obtain a new visual tracking result; determining whether there is a common viewing relationship between the new current image frame and the new submap based on the new visual tracking result; in response to the new current image frame and the new submap not having a common viewing relationship, creating a new submap again.

[0020] In the above embodiment, the new current image frame can be combined with the new sub-map for visual tracking. Based on the new visual tracking results, it is determined whether there is a common viewing relationship between the new current image frame and the new sub-map. If there is no common viewing relationship, a new sub-map can still be created again, avoiding the problem of using a single map in related technologies and the inability to expand the map once visual tracking becomes difficult.

[0021] Optionally, the method further includes: merging all sub-maps that meet preset merging conditions to obtain a global map.

[0022] In the above embodiment, all sub-maps that meet the preset merging conditions can be merged to obtain a global map with higher usability.

[0023] Optionally, the submap that meets the preset merging condition is determined in the following manner: in response to at least two submaps having a co-viewing relationship, it is determined that the at least two submaps having the co-viewing relationship are the submaps that meet the preset merging condition.

[0024] In the above embodiment, when at least two sub-maps have a common view relationship, it can be determined that the at least two sub-maps satisfy a preset merging condition, thereby ensuring the accuracy of the global map finally obtained.

[0025] According to a second aspect of an embodiment of the present disclosure, a multi-map construction device is provided, the device comprising: a first visual tracking module, used to visually track a current image frame in combination with a current sub-map to obtain a visual tracking result; a first common-view relationship determination module, used to determine whether there is a common-view relationship between the current image frame and the current sub-map based on the visual tracking result; and a first map construction module, used to create a new sub-map in response to the lack of a common-view relationship between the current image frame and the current sub-map.

[0026] Optionally, the first co-viewing relationship determination module includes: a first determination submodule, for determining that there is a co-viewing relationship between the current image frame and the current submap in response to the visual tracking result indicating that there are matching feature points between the current image frame and the current submap, and the number of the matching feature points is greater than or equal to a first preset threshold, otherwise determining that there is no co-viewing relationship between the current image frame and the current submap.

[0027] Optionally, the first co-viewing relationship determination module includes: an acquisition submodule, used to remove the incorrectly matched first feature points from the matched feature points in response to the visual tracking result indicating that there are matching feature points between the current image frame and the current submap, and obtain validly matched second feature points; a second determination submodule, used to determine that there is a co-viewing relationship between the current image frame and the current submap in response to the number of the second feature points being greater than or equal to a second preset threshold, otherwise determine that there is no co-viewing relationship between the current image frame and the current submap.

[0028] Optionally, the acquisition submodule includes: an acquisition unit, used to input the current image frame and the image frame included in the current submap into a pre-established feature point matching model, and remove the first feature points that are incorrectly matched from the matched feature points through the feature point matching model to obtain the second feature points that are effectively matched.

[0029] Optionally, the device further includes: a second map construction module, configured to expand the current sub-map using the current image frame in response to a co-viewing relationship between the current image frame and the current sub-map.

[0030] Optionally, the device further comprises: a third map construction module, configured to merge the current submap with at least one of the other submaps in response to a co-viewing relationship between the current image frame and at least one of the other submaps.

[0031] Optionally, the first map construction module includes: a first map construction submodule, used to create a new submap with empty content; and a second map construction submodule, used to use the current image frame as an initial frame of the new submap in the new submap with empty content.

[0032] Optionally, the device also includes: a second visual tracking module, used to visually track the new current image frame in combination with the new sub-map to obtain a new visual tracking result; a second common-view relationship determination module, used to determine whether there is a common-view relationship between the new current image frame and the new sub-map based on the new visual tracking result; and a fourth map construction module, used to create a new sub-map again in response to the lack of a common-view relationship between the new current image frame and the new sub-map.

[0033] Optionally, the device further includes: a fifth map construction module, configured to merge all sub-maps that meet preset merging conditions to obtain a global map.

[0034] Optionally, the fifth map construction module includes: a third determination submodule, configured to, in response to at least two submaps having a co-viewing relationship, determine that the at least two submaps having the co-viewing relationship are submaps that meet a preset merging condition.

[0035] According to a third aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the map construction method described in the first aspect above.

[0036] According to a fourth aspect of an embodiment of the present disclosure, a map construction device is provided, the device comprising: a processor; a memory for storing processor executable instructions; wherein the processor is used to call the executable instructions stored in the memory to implement the map construction method described in the first aspect above.

[0037] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0039] Figure 1 is a flowchart of a map construction method according to an exemplary embodiment of the present disclosure;

[0040] Figure 2 is a flowchart of another map construction method according to an exemplary embodiment of the present disclosure;

[0041] Figure 3 is a flowchart of another map construction method according to an exemplary embodiment of the present disclosure;

[0042] Figure 4 is a flowchart of another map construction method according to an exemplary embodiment of the present disclosure;

[0043] Figure 5 is a flowchart of another map construction method according to an exemplary embodiment of the present disclosure;

[0044] Figure 6 is a flowchart of another map construction method according to an exemplary embodiment of the present disclosure;

[0045] Figure 7 is a flowchart of another map construction method according to an exemplary embodiment of the present disclosure;

[0046] Figure 8 is a block diagram of a map construction device according to an exemplary embodiment of the present disclosure;

[0047] Fig. 9 is a block diagram of another map construction device according to an exemplary embodiment of the present disclosure;

[0048] Fig.10 is a block diagram of another map construction device according to an exemplary embodiment of the present disclosure;

[0049] Fig.11 is a block diagram of another map construction device according to an exemplary embodiment of the present disclosure;

[0050] Fig.12 is a block diagram of another map construction device according to an exemplary embodiment of the present disclosure;

[0051] Fig.13 is a block diagram of another map construction device according to an exemplary embodiment of the present disclosure;

[0052] Fig.14 is a block diagram of another map construction device according to an exemplary embodiment of the present disclosure;

[0053] Fig.15 is a block diagram of another map construction device according to an exemplary embodiment of the present disclosure;

[0054] Fig.16 is a block diagram of another map construction device according to an exemplary embodiment of the present disclosure;

[0055] Fig.17is a block diagram of another map construction device according to an exemplary embodiment of the present disclosure;

[0056] Fig.18 It is a structural schematic diagram of a map construction device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0057] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0058] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms "a", "the" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0059] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0060] The embodiment of the present disclosure provides a map construction method, which can be used in a machine device, terminal device or platform that needs to perform real-time map construction. The following description will only be made using a terminal as an example. Figure 1 As shown, Figure 1 A map construction method according to an exemplary embodiment includes the following steps:

[0061] In step 101, visual tracking is performed on the current image frame in combination with the current sub-map to obtain a visual tracking result.

[0062] The terminal may collect the current image frame through an image collection device, such as a camera. The current sub-map is a collection of image information included in at least one image frame collected previously.

[0063] Optionally, in the disclosed embodiment, the terminal may also obtain corresponding sensor data through at least one other sensor, so that when visual tracking of the captured image frames is subsequently performed, these sensor data may be combined for analysis to improve the accuracy of visual tracking of the image frames. In the disclosed embodiment, the above-mentioned sensors may include but are not limited to IMU sensors, orientation sensors, acceleration sensors, gravity sensors, etc.

[0064] Furthermore, the terminal may perform visual tracking on the current image frame in combination with the current sub-map, thereby obtaining a visual tracking result.

[0065] In the disclosed embodiment, feature points are extracted for the current image frame and the previous image frame of the current image frame (i.e., the last image frame for constructing the current sub-map), respectively, to determine whether there are matching feature points between the current image frame and the previous image frame. If so, the visual tracking result is determined to be that there are matching feature points between the current image frame and the current sub-map, otherwise the visual tracking result is determined to be that there are no matching feature points between the current image frame and the current sub-map. In step 102, based on the visual tracking result, it is determined whether there is a common view relationship between the current image frame and the current sub-map.

[0066] The terminal may determine whether there is a common viewing relationship between the current image frame and the current submap based on whether there are matching feature points between the current image frame and the current submap indicated by the visual tracking result.

[0067] Optionally, if the visual tracking result indicates that there are matching feature points between the current image frame and the current submap, and the number of matching feature points is large, it is determined that there is a common viewing relationship between the current image frame and the current submap; otherwise, it is determined that there is no common viewing relationship between the current image frame and the current submap.

[0068] Alternatively, if the visual tracking result indicates that there are matching feature points between the current image frame and the current submap, and the number of valid matching (inlier) second feature points obtained after removing the wrongly matched (outlier) first feature points among the matching feature points is large, then it is determined that there is a common viewing relationship between the current image frame and the current submap; otherwise, it is determined that there is no common viewing relationship between the current image frame and the current submap.

[0069] In step 103, in response to the current image frame and the current sub-map not having a co-viewing relationship, a new sub-map is created.

[0070] In response to the fact that there is no common view relationship between the current image frame and the current submap, the terminal can first create a new submap with empty content, and then use the current image frame as the initial frame of the new submap, so as to continue to expand the new submap later. This can avoid the problem of using a single map in related technologies, and the map cannot be expanded once visual tracking becomes difficult. On the other hand, the terminal does not need to rely on IMU for map expansion, has higher availability, and can be applied to a wider range of fields.

[0071] In some optional embodiments, the terminal may use any one of the following methods to determine whether there is a co-viewing relationship between the current image frame and the current sub-map according to the visual tracking result.

[0072] <The first method> It is only necessary to determine whether there are matching feature points between the current image frame and the current sub-map, and whether the number of matching feature points between the two is greater than or equal to a first preset threshold.

[0073] In the first mode, if there are matching feature points between the current image frame and the current submap, and the number of the matching feature points is greater than or equal to a first preset threshold, it is determined that there is a common view relationship between the current image frame and the current submap.

[0074] For example, the feature points corresponding to the last image frame a of the current submap constructed by the current submap correspond to scene A, and the feature points corresponding to the current image frame also correspond to scene A. It is determined that the number of matching feature points between the current image frame and image frame a is greater than or equal to the first preset threshold, indicating that there are enough matching feature points between the current image frame and the current submap, and it is determined that there is a common viewing relationship between the current image frame and the current submap.

[0075] For another example, the last image frame a used to construct the current submap corresponds to scene A, and the corresponding feature points of the current image frame correspond to scene B. If the number of matched feature points determined is less than the first preset threshold, it means that there are not enough matching feature points between the two.

[0076] The first method can be used to quickly determine whether there is a common viewing relationship between the current image frame and the current sub-map. However, on the other hand, there may be misjudgment. In order to avoid this situation, in the embodiment of the present disclosure, the terminal can also use the second method to determine whether there is a common viewing relationship between the current image frame and the current sub-map.

[0077] <Second Method> It is simultaneously determined whether there are matching feature points between the current image frame and the current sub-map, and whether the number of effectively matching feature points between the two is greater than or equal to a second preset threshold.

[0078] If there are matching feature points between the current image frame and the current submap, it is necessary to further eliminate the first feature points of incorrect matching (outlier) through verification methods such as geometric consistency, obtain the second feature points of valid matching (inlier), and determine whether the number of valid matching second feature points is sufficient, for example, whether it is greater than or equal to the second preset threshold. Only when the number of valid matching second feature points is greater than or equal to the second preset threshold, the terminal considers that there is a common view relationship between the current image frame and the current submap. Among them, the second preset threshold and the first preset threshold are the same or different, and the present disclosure does not limit this.

[0079] In other cases, for example, if there are no matching feature points between the current image frame and the current submap, or if there are matching feature points between the two, but the number of effectively matched second feature points is small and less than the second preset threshold, it can also be considered that there is no common view relationship between the current image frame and the current submap, thereby reducing misjudgment.

[0080] In some optional embodiments, the current image frame and the current sub-map may be input into a pre-established feature point matching model, and the feature point matching model may remove the incorrectly matched first feature points from the matched feature points, thereby obtaining the effectively matched second feature points.

[0081] Among them, the feature point matching model can be established based on local geometric consistency or global geometric consistency methods. For example, the feature point matching model uses a global geometric consistency method to verify all feature points between at least two sample image frames, and the second feature points that are effectively matched between at least two sample image frames have been marked. By training the feature point matching model, the second feature points that are effectively matched between at least two sample image frames output by the feature point matching model meet the preset fault tolerance range or are even consistent with the second feature points marked in at least two sample image frames.

[0082] In the above embodiment, a feature point matching model can be established in advance, and the current image frame and the current submap can be input into the feature point matching model. The feature point matching model can remove the first feature points that are incorrectly matched between the current image frame and the current submap, thereby obtaining the second feature points that are effectively matched. In this way, the second feature points that are effectively matched can be quickly determined, which is simple to implement.

[0083] In some optional embodiments, for example Figure 2 As shown, the above method may further include the following steps:

[0084] In step 104, in response to the current image frame and the current sub-map having a co-viewing relationship, the current sub-map is expanded using the current image frame.

[0085] In the disclosed embodiment, if there is a co-viewing relationship between the current image frame and the current submap, then based on the feature point portion that matches the current image frame and the current submap, information on other feature points of the current image frame can be added to the current submap, thereby expanding the current submap.

[0086] In the above embodiment, when there is a co-viewing relationship between the current image frame and the current sub-map, the current sub-map can be expanded based on the current image frame to improve the map accuracy.

[0087] In some optional embodiments, for example Figure 3 As shown, after completing step 104, the method may further include:

[0088] In step 105, in response to the current image frame and at least one other sub-map having a co-viewing relationship, the current sub-map is merged with the at least one other sub-map.

[0089] In this step, the terminal can also determine whether the current submap has a co-viewing relationship with at least one other submap except the current submap in the same manner as determining whether the current image frame has a co-viewing relationship with the current submap in the above step 102, so that when the current image frame has a co-viewing relationship with at least one other submap, a closed loop algorithm is used to merge the current submap with at least one other submap, and errors are eliminated through global optimization to improve map accuracy.

[0090] In some optional embodiments, for example Figure 4 As shown, the above step 103 may include:

[0091] In step 103 - 1 , a new sub-map with empty content is created.

[0092] In the disclosed embodiment, in addition to the current sub-map, a new sub-map with empty content can be recreated.

[0093] In step 103 - 2 , in the new sub-map whose content is empty, the current image frame is used as the initial frame of the new sub-map.

[0094] In the new sub-map whose content is empty, the current image frame is used as the initial frame of the new sub-map, and the new sub-map may include information of feature points in the current image frame.

[0095] In the above embodiment, in the new sub-map whose content is empty, the current image frame can be used as the initial frame of the new sub-map, so that the new sub-map can be subsequently expanded according to the collected new current image frame, thereby avoiding the problem of using a single map in the related technology, and the map cannot be expanded once the visual tracking is difficult.

[0096] In some optional embodiments, for example Figure 5 As shown, after completing step 103, the method may further include:

[0097] In step 106, visual tracking is performed on the new current image frame in combination with the new sub-map to obtain a new visual tracking result.

[0098] The method of step 101 is repeated to perform visual tracking on the new current image frame collected by the terminal in combination with the new sub-map, thereby obtaining a new visual tracking result.

[0099] In step 107, it is determined whether there is a common viewing relationship between the new current image frame and the new sub-map according to the new visual tracking result.

[0100] In the embodiment of the present disclosure, the same method as the above step 102 may be used to determine whether the new current image frame and the new sub-map have a common viewing relationship, which will not be described in detail herein.

[0101] In step 108, in response to the new current image frame and the new sub-map not having a co-viewing relationship, a new sub-map is created again.

[0102] If there is no co-viewing relationship between the new current image frame and the new sub-map, a new sub-map may be created again; if there is a co-viewing relationship between the new current image frame and the new sub-map, the new sub-map may be extended through the new current image frame.

[0103] In the above embodiment, the new current image frame can be combined with the new sub-map for visual tracking, and according to the new visual tracking result, it is determined whether there is a common view relationship between the new current image frame and the new sub-map. If there is no common view relationship, a new sub-map can still be created again, avoiding the problem of using a single map in the related art, and the map cannot be expanded once the visual tracking is difficult. In some optional embodiments, for example Figure 6 As shown, the method may also include:

[0104] In step 109, all sub-maps that meet the preset merging conditions are merged to obtain a global map.

[0105] In the disclosed embodiment, all sub-maps may be stored in a map database.

[0106] The terminal can determine whether the two submaps meet the preset merging condition by judging whether there is a co-viewing relationship between any two submaps, and if the number of effectively matched second feature points between the two submaps is greater than or equal to a second preset threshold, it can be determined that the two submaps have a co-viewing relationship. Further, it can be determined that the two submaps meet the preset merging condition.

[0107] The terminal can use the second feature points that are effectively matched between the two submaps with a common view relationship to correct the possible relative position offset between the two submaps, merge the two submaps using a closed loop algorithm, and finally obtain a global map through global optimization.

[0108] In the above embodiment, when at least two sub-maps have a common view relationship, it can be determined that at least two sub-maps meet a preset merging condition, and the sub-maps with the common view relationship can be merged using a loop closure algorithm to ensure the accuracy of the final global map and higher usability.

[0109] In one embodiment, if Figure 7 As shown, Figure 7 Another map construction method according to an exemplary embodiment includes the following steps:

[0110] In step 201, visual tracking is performed on the current image frame in combination with the current sub-map to obtain a visual tracking result.

[0111] In step 202, it is determined whether there is a common viewing relationship between the current image frame and the current sub-map according to the visual tracking result.

[0112] In response to the visual tracking result indicating that there are matching feature points between the current image frame and the current submap, and the number of valid matching second feature points is greater than or equal to the second preset threshold, it is determined that there is a common view relationship between the current image frame and the current submap, and step 203 is executed. Otherwise, it is determined that there is no common view relationship between the current image frame and the current submap, and step 205 is executed.

[0113] In step 203, the current sub-map is expanded using the current image frame.

[0114] In step 204, in response to the current image frame and at least one other sub-map having a co-viewing relationship, the current sub-map is merged with the at least one other sub-map.

[0115] In step 205, a new sub-map with empty content is created.

[0116] In step 206, in the new sub-map whose content is empty, the current image frame is used as the initial frame of the new sub-map.

[0117] In step 207, visual tracking is performed on the new current image frame in combination with the new sub-map to obtain a new visual tracking result.

[0118] In step 208, it is determined whether there is a common viewing relationship between the new current image frame and the new sub-map according to the new visual tracking result.

[0119] In step 209, in response to the new current image frame and the new sub-map having a co-viewing relationship, the new current sub-map is expanded using the new current image frame, otherwise a new sub-map is created again.

[0120] In step 210, all sub-maps are merged to obtain a global map.

[0121] In response to at least two sub-maps having a common view relationship, a loop closure algorithm is used to merge the sub-maps having the common view relationship, and errors are eliminated through optimization to obtain a global map.

[0122] In the above embodiment, submap expansion and submap generation are performed based on whether there is a common view relationship between the current image frame and the current submap, so as to solve the problem that the map cannot be expanded once the visual tracking in a single map is difficult to continue. Corresponding to the above method embodiment, the present disclosure also provides an embodiment of the device.

[0123] like Figure 8 As shown, Figure 8 It is a block diagram of a map construction device shown in the present disclosure according to an exemplary embodiment, and the device includes: a first visual tracking module 310, which is used to visually track the current image frame in combination with the current sub-map to obtain a visual tracking result; a first co-viewing relationship determination module 320, which is used to determine whether there is a co-viewing relationship between the current image frame and the current sub-map based on the visual tracking result; a first map construction module 330, which is used to create a new sub-map in response to the lack of a co-viewing relationship between the current image frame and the current sub-map.

[0124] like Fig. 9 As shown, Fig. 9 The present disclosure shows another map expansion device block diagram according to an exemplary embodiment. Figure 8 On the basis of the embodiment, the first co-viewing relationship determination module 320 includes: a first determination submodule 321, which is used to determine that there is a co-viewing relationship between the current image frame and the current submap in response to the visual tracking result indicating that there are matching feature points between the current image frame and the current submap, and the number of matching feature points is greater than or equal to a first preset threshold, otherwise it is determined that there is no co-viewing relationship between the current image frame and the current submap.

[0125] like Fig.10As shown, Fig.10 The present disclosure shows another map construction device block diagram according to an exemplary embodiment. Fig. 9 On the basis of the embodiment, the first co-viewing relationship determination module 320 includes: an acquisition submodule 322, which is used to remove the incorrectly matched first feature points from the matched feature points in response to the visual tracking results indicating that there are matching feature points between the current image frame and the current submap, and obtain the effectively matched second feature points; a second determination submodule 323, which is used to determine that there is a co-viewing relationship between the current image frame and the current submap in response to the number of the second feature points being greater than or equal to a second preset threshold, otherwise determine that there is no co-viewing relationship between the current image frame and the current submap.

[0126] like Fig.11 As shown, Fig.11 The present disclosure shows another map construction device block diagram according to an exemplary embodiment. Fig.10 Based on the embodiment, the acquisition submodule 322 includes: an acquisition unit 3221, which is used to input the current image frame and the image frame included in the current submap into a pre-established feature point matching model, and remove the first feature points that are incorrectly matched from the matched feature points through the feature point matching model to obtain the second feature points that are effectively matched.

[0127] like Fig.12 As shown, Fig.12 The present disclosure shows another map construction device block diagram according to an exemplary embodiment. Figure 8 Based on the embodiment, the device further includes: a second map construction module 340, which is used to expand the current sub-map using the current image frame in response to the co-viewing relationship between the current image frame and the current sub-map.

[0128] like Fig.13 As shown, Fig.13 The present disclosure shows another map construction device block diagram according to an exemplary embodiment. Figure 8 Based on the embodiment, the device further includes: a third map construction module 350, which is used to merge the current submap with at least one of the other submaps in response to a co-viewing relationship between the current image frame and at least one of the other submaps.

[0129] like Fig.14 As shown, Fig.14 The present disclosure shows another map construction device block diagram according to an exemplary embodiment. Figure 8Based on the embodiment, the first map construction module 330 includes: a first map construction sub-module 331, used to create a new sub-map with empty content; a second map construction sub-module 332, used to use the current image frame as the initial frame of the new sub-map in the new sub-map with empty content.

[0130] like Fig.15 As shown, Fig.15 The present disclosure shows another map construction device block diagram according to an exemplary embodiment. Figure 8 On the basis of the embodiment, the device also includes: a second visual tracking module 360, which is used to visually track the new current image frame in combination with the new sub-map to obtain a new visual tracking result; a second common-view relationship determination module 370, which is used to determine whether there is a common-view relationship between the new current image frame and the new sub-map based on the new visual tracking result; and a fourth map construction module 380, which is used to create a new sub-map again in response to the lack of a common-view relationship between the new current image frame and the new sub-map.

[0131] like Fig.16 As shown, Fig.16 The present disclosure shows another map construction device block diagram according to an exemplary embodiment. Figure 8 Based on the embodiment, the device further includes: a fifth map construction module 390, which is used to merge all sub-maps that meet the preset merging conditions to obtain a global map.

[0132] like Fig.17 As shown, Fig.17 The present disclosure shows another map construction device block diagram according to an exemplary embodiment. Fig.16 Based on the embodiment, the fifth map construction module 390 includes: a third determination submodule 391, which is used to determine that the at least two submaps having a co-viewing relationship are submaps that meet a preset merging condition in response to at least two submaps having a co-viewing relationship.

[0133] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art may understand and implement it without creative work.

[0134] The embodiment of the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute any of the above-mentioned map construction methods.

[0135] In some optional embodiments, the present disclosure provides a computer program product, including computer-readable code. When the computer-readable code is run on a device, a processor in the device executes instructions for implementing the map construction method provided in any of the above embodiments.

[0136] In some optional embodiments, the embodiments of the present disclosure further provide another computer program product for storing computer-readable instructions, which, when executed, enable a computer to perform the operations of the map construction method provided in any of the above embodiments.

[0137] The computer program product may be implemented in hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium (including volatile and non-volatile storage media), and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0138] The embodiment of the present disclosure also provides a map construction device, which includes: a processor; a memory for storing processor executable instructions; wherein the processor is used to call the executable instructions stored in the memory to implement any of the above-mentioned map construction methods.

[0139] like Fig.18 As shown, Fig.18 1 is a schematic diagram of a map construction device 1800 according to an exemplary embodiment. For example, the device 1800 may be provided as a machine device, a terminal or a platform. Fig.18 , the device 1800 includes a processing component 1822, which further includes one or more processors, and a memory resource represented by a memory 1832 for storing instructions executable by the processing component 1822, such as an application. The application stored in the memory 1832 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1822 is used to execute instructions to perform any of the above-mentioned map construction methods.

[0140] The device 1800 may also include a power supply component 1826 for performing power management of the device 1800, a wired or wireless network interface 1850 for connecting the device 1800 to a network, and an input / output (I / O) interface 1858. The device 1800 may operate based on an operating system stored in the memory 1832, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeB SD™ or the like.

[0141] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0142] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the scope of protection of the present disclosure.

Claims

1. A map construction method, characterized in that: The method comprises: Perform visual tracking on the current image frame combined with the current sub-map to obtain a visual tracking result; Determining, according to the visual tracking result, whether there is a common viewing relationship between the current image frame and the current sub-map; In response to the current image frame and the current sub-map not having a co-viewing relationship, creating a new sub-map; The creation of a new sub-map includes: Create a new submap with empty content; In the new sub-map whose content is empty, taking the current image frame as the initial frame of the new sub-map; After creating the new sub-map, the method further includes: Performing visual tracking on the new current image frame in combination with the new sub-map to obtain a new visual tracking result; Determining, according to the new visual tracking result, whether there is a common viewing relationship between the new current image frame and the new sub-map; In response to the new current image frame and the new sub-map not having a co-viewing relationship, creating a new sub-map again; All submaps that meet the preset merging condition are merged to obtain a global map; wherein, in response to at least two submaps having a common view relationship, it is determined that the at least two submaps having the common view relationship are submaps that meet the preset merging condition.

2. The method according to claim 1, characterized in that The determining, according to the visual tracking result, whether there is a co-viewing relationship between the current image frame and the current sub-map includes: In response to the visual tracking result indicating that there are matching feature points between the current image frame and the current submap, and the number of the matching feature points is greater than or equal to a first preset threshold, it is determined that there is a common viewing relationship between the current image frame and the current submap, otherwise it is determined that there is no common viewing relationship between the current image frame and the current submap.

3. The method according to claim 1, characterized in that The determining, according to the visual tracking result, whether there is a co-viewing relationship between the current image frame and the current sub-map includes: In response to the visual tracking result indicating that there are matching feature points between the current image frame and the current sub-map, removing the first feature points that are incorrectly matched from the matched feature points to obtain the second feature points that are effectively matched; In response to the number of the second feature points being greater than or equal to a second preset threshold, it is determined that there is a common viewing relationship between the current image frame and the current submap; otherwise, it is determined that there is no common viewing relationship between the current image frame and the current submap.

4. The method according to claim 3, characterized in that: The step of removing the first feature points that are incorrectly matched from the matched feature points to obtain the second feature points that are effectively matched comprises: The current image frame and the current sub-map are input into a pre-established feature point matching model, and the first feature points that are incorrectly matched are removed from the matched feature points through the feature point matching model to obtain the second feature points that are effectively matched.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: In response to a co-viewing relationship between the current image frame and the current sub-map, the current sub-map is expanded using the current image frame.

6. The method according to claim 5, characterized in that After expanding the current sub-map by using the current image frame, the method further includes: In response to a co-viewing relationship between the current image frame and at least one other sub-map, the current sub-map is merged with the at least one other sub-map.

7. A map construction device, characterized in that: The device comprises: A first visual tracking module is used to perform visual tracking on the current image frame in combination with the current sub-map to obtain a visual tracking result; A first co-viewing relationship determination module, configured to determine whether there is a co-viewing relationship between the current image frame and the current sub-map according to the visual tracking result; A first map construction module, configured to create a new sub-map in response to the current image frame and the current sub-map not having the co-viewing relationship; The first map construction module includes: The first map construction submodule is used to create a new submap with empty content; A second map construction submodule is used to use the current image frame as an initial frame of the new submap in the new submap whose content is empty; The device also includes: A second visual tracking module, used to perform visual tracking on the new current image frame in combination with the new sub-map to obtain a new visual tracking result; A second common-view relationship determination module, used to determine whether there is a common-view relationship between the new current image frame and the new sub-map according to the new visual tracking result; a fourth map construction module, configured to create a new sub-map again in response to the new current image frame and the new sub-map not having a co-viewing relationship; A fifth map construction module is used to merge all sub-maps that meet the preset merging conditions to obtain a global map; the fifth map construction module includes: a third determination sub-module, used to determine that the at least two sub-maps with the common viewing relationship are sub-maps that meet the preset merging conditions in response to the at least two sub-maps having the common viewing relationship.

8. The device according to claim 7, characterized in that The first co-viewing relationship determining module includes: The first determination submodule is used to determine that there is a common viewing relationship between the current image frame and the current submap in response to the visual tracking result indicating that there are matching feature points between the current image frame and the current submap, and the number of the matching feature points is greater than or equal to a first preset threshold, otherwise determine that there is no common viewing relationship between the current image frame and the current submap.

9. The device according to claim 8, characterized in that The first co-viewing relationship determining module includes: an acquisition submodule, configured to remove the first feature points that are incorrectly matched from the matched feature points in response to the visual tracking result indicating that there are matching feature points between the current image frame and the current submap, and obtain the second feature points that are effectively matched; The second determination submodule is used to determine that there is a common viewing relationship between the current image frame and the current submap in response to the number of the second feature points being greater than or equal to a second preset threshold, otherwise determine that there is no common viewing relationship between the current image frame and the current submap.

10. The device according to claim 9, characterized in that The acquisition submodule includes: The acquisition unit is used to input the current image frame and the image frame included in the current sub-map into a pre-established feature point matching model, and remove the first feature points that are incorrectly matched from the matched feature points through the feature point matching model to obtain the second feature points that are effectively matched.

11. The device according to any one of claims 7 to 10, characterized in that: The device also includes: The second map construction module is configured to expand the current sub-map using the current image frame in response to a co-viewing relationship between the current image frame and the current sub-map.

12. The device according to claim 11, characterized in that The device also includes: The third map construction module is configured to merge the current submap with at least one other submap in response to a co-viewing relationship between the current image frame and at least one other submap.

13. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the map construction method described in any one of claims 1 to 6 above.

14. A multi-map maintenance device, characterized in that: The device comprises: processor; a memory for storing processor-executable instructions; The processor is used to call the executable instructions stored in the memory to implement the map construction method according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • Stable motion tracking method and stable motion tracking device based on integration of simple camera and IMU (inertial measurement unit) of smart cellphone

    CN105953796A

  • Indoor feature point and structural line combination-based indoor SLAM (Simultaneous Localization and Mapping) method

    CN107392964A

  • Mapping method and device

    CN108319976A

  • Robot positioning and map construction system based on binocular vision features and IMU information

    CN108665540A

  • Feature map construction method, visual positioning method and corresponding device

    CN109059941A