Mapping method and apparatus, electronic device, and storage medium

Abstract

The embodiment of the present disclosure discloses a mapping method, comprising: acquiring a visual image frame collected by a camera device; screening the visual image frame to obtain a key image frame and a supplementary image frame; constructing a map according to the key image frame to obtain a core map; and in a post-processing stage, post-processing the core map based on the supplementary image frame to obtain an optimized high-precision map. The technical solution of the present disclosure can improve the real-time performance of mapping and the robustness of the constructed map.

Description

Technical Field

[0001] This disclosure relates to the field of map technology, and more particularly to a map-building method, apparatus, electronic device, and storage medium. Background Technology

[0002] Mapping and localization are core technologies in the field of autonomous driving. For driverless vehicles, it is necessary to first build a map in the unknown environment, and then use the map for localization during subsequent autonomous driving. In the map building process, two performance aspects are usually considered: real-time map building and map robustness. To make the map points denser and thus improve its robustness, more visual image frames are usually needed. However, processing more visual image frames takes more time, thus affecting the real-time performance of map building. Summary of the Invention

[0003] To address the aforementioned technical problems, this disclosure provides a mapping method, apparatus, electronic device, and storage medium.

[0004] In a first aspect, embodiments of this disclosure provide a mapping method, the method comprising:

[0005] Acquire visual image frames captured by the camera device;

[0006] The visual image frames are filtered to obtain key image frames and supplementary image frames;

[0007] A core map is constructed based on the key image frames.

[0008] In the post-processing stage, the core map is post-processed based on the supplementary image frames to obtain an optimized high-precision map.

[0009] Secondly, embodiments of this disclosure also provide a mapping apparatus, the apparatus comprising:

[0010] The image frame acquisition module is used to acquire visual image frames captured by the camera device;

[0011] The image frame classification module is used to filter the visual image frames to obtain key image frames and supplementary image frames.

[0012] The map building module is used to build a core map based on the key image frames.

[0013] The post-processing module is used to perform post-processing on the core map based on the supplementary image frames in the post-processing stage to obtain an optimized high-precision map.

[0014] Thirdly, embodiments of this disclosure also provide an electronic device, the electronic device comprising: one or more processors; a storage device for storing one or more programs; and when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the mapping method as described above.

[0015] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the mapping method described above.

[0016] The mapping method provided in this disclosure divides mapping into two stages: a preprocessing stage and a post-processing stage. The preprocessing stage, also known as a lightweight screenshot stage, first acquires visual image frames captured by a camera device. Then, it filters these visual image frames to obtain key image frames and supplementary image frames. Based on the key image frames, a core map is constructed. This core map construction, due to the small number of visual image frames and map points involved, allows for reasonable control of the computational load for local map optimization, thus enabling rapid construction and improving the real-time performance of map building. Furthermore, the core map construction step can be performed online during the camera's acquisition of visual image frames. Finally, in the post-processing stage, the core map is post-processed based on the supplementary image frames obtained in the preprocessing stage to obtain an optimized high-precision map. This post-processing stage can be performed online or offline. The use of supplementary image frames makes the map points in the final high-precision map denser, further enriching the map elements and resulting in a more robust high-precision map. Attached Figure Description

[0017] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0018] Figure 1 This is a flowchart illustrating a mapping method according to an embodiment of this disclosure;

[0019] Figure 2 for Figure 1 A detailed flowchart of step 120 in the illustrated embodiment is shown.

[0020] Figure 3 for Figure 1 A detailed flowchart of step 130 in the illustrated embodiment is shown.

[0021] Figure 4 A schematic diagram illustrating the process of constructing a core map for an embodiment of this disclosure;

[0022] Figure 5 This is a flowchart illustrating the post-processing stage in an embodiment of this disclosure;

[0023] Figure 6 This is a technical framework diagram of an embodiment of the present disclosure;

[0024] Figure 7 This is a schematic diagram of a lightweight mapping process in an embodiment of this disclosure;

[0025] Figure 8 This is a flowchart illustrating a post-processing procedure in an embodiment of this disclosure;

[0026] Figure 9 This is a schematic diagram of the structure of a mapping device according to an embodiment of the present disclosure;

[0027] Figure 10 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0028] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0029] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0030] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0031] Currently, the mapping methods commonly used in autonomous driving struggle to balance real-time performance and robustness during map construction. This disclosure addresses these shortcomings by providing a new mapping method. This method divides mapping into two stages: a preprocessing stage and a post-processing stage. The preprocessing stage, also known as a lightweight mapping stage, first acquires visual image frames from a camera device. These frames are then filtered to obtain key and supplementary image frames. A core map is constructed based on these key image frames. Because the core map construction process uses fewer visual image frames and involves fewer map points, the computational load for local map optimization can be reasonably controlled, allowing for rapid construction and improved real-time performance. Furthermore, the core map construction step can be performed online during the acquisition of visual image frames by the camera device. Finally, in the post-processing stage, the core map is post-processed based on the supplementary image frames obtained in the pre-processing stage to obtain an optimized high-precision map. This post-processing stage can be executed online or offline. Furthermore, due to the use of supplementary image frames, the map points in the final high-precision map are denser, further enriching the map elements and giving the constructed high-precision map better robustness.

[0032] Figure 1 This is a flowchart illustrating a mapping method according to an embodiment of this disclosure. The method can be executed by a mapping device, which can be implemented in software and / or hardware, and can be configured in an electronic device. Figure 1 As shown, the method may specifically include the following steps:

[0033] Step 110: Acquire visual image frames captured by the camera device;

[0034] The technical solution of this disclosure is a vision-based mapping scheme, which involves installing a camera device on a vehicle to capture visual image frames, and then constructing a map based on the visual image frames. Typically, the vehicle travels in the area or on the road where the map needs to be constructed, and the camera device continuously captures images to obtain continuous visual image frames.

[0035] Step 120: Filter the above visual image frames to obtain key image frames and supplementary image frames;

[0036] Since this embodiment emphasizes that the mapping scheme is divided into two stages, the preprocessing stage only needs to use a small number of key image frames to build the core map, and then the postprocessing stage uses supplementary image frames to postprocess the core map to obtain an optimized high-precision map, some filtering conditions can be set in this step to filter the visual image frames continuously acquired by the camera device to obtain key image frames and supplementary image frames.

[0037] In the actual execution process, a first filtering condition and a second filtering condition can be preset to filter out key image frames and supplementary image frames from the visual image frames, respectively. For details, please refer to [link to relevant documentation]. Figure 2 ,like Figure 2 As shown, step 120 above can specifically include:

[0038] Step 1201: For each visual image frame, filter the visual image frame according to the preset first filtering conditions to obtain the key image frame;

[0039] The first filtering criteria in this step include at least one of the following: the number of map points matching the current core map, the acquisition distance difference with the previous key image frame, and the acquisition angle difference with the previous key image frame. In practical application scenarios, certain thresholds can be set for the number of map points, acquisition distance difference, and acquisition angle difference, and then the visual image frames can be filtered according to these thresholds to obtain the key image frames.

[0040] Step 1202: Filter the non-critical image frames other than the critical image frames according to the preset second filtering conditions to obtain the supplementary image frames.

[0041] Specifically, the second filtering condition mentioned above includes at least one of the following: the number of map points matching the current core map, the acquisition distance difference between the current key image frames and the acquisition angle ...

[0042] Step 130: Construct a map based on the above key image frames to obtain a core map;

[0043] This step, based on the key image frames selected in step 120 above, constructs the core map by acquiring the poses of the key image frames and the map points and their location information within them. The poses of the key image frames can be solved through pose transformation, and upon successful solution, triangulation is performed to recover the feature point depth to form map points and obtain their location information. The pose calculation and depth recovery steps can be performed pre-processed, i.e., during the preprocessing of each visual image frame, or they can be performed in this step on the selected key image frames. Alternatively, the poses and map point location information of some key image frames can be obtained during the preprocessing stage, while the poses and map point location information of others can be obtained in this step, depending on the specific circumstances.

[0044] Step 140: In the post-processing stage, the core map is post-processed based on the above-mentioned supplementary image frames to obtain an optimized high-precision map.

[0045] In this step, the poses of each supplementary image frame can be obtained, and map point triangulation can be performed based on the poses of each supplementary image frame to obtain the depth of the map points, thereby obtaining complete map point location information. The above poses and map point location information are then added to the core map to obtain an optimized high-precision map.

[0046] In some embodiments, the pose of the supplementary image frame in this step can be obtained by acquiring the initial pose of the supplementary image frame during the preprocessing stage of the visual image frame, and then optimizing the initial pose during the postprocessing stage of this step to obtain the optimized pose of the supplementary image frame. The specific method of obtaining the optimized pose will be described in subsequent embodiments. The technical solution provided by the embodiments of this disclosure, in the process of map construction using visual image frames, firstly filters the visual image frames to obtain key image frames and supplementary image frames, then constructs a core map based on the key image frames, and optimizes the core map based on the supplementary image frames to obtain an optimized high-precision map. In the process of constructing the core map, since the number of key image frames and the number of map points used are relatively small, rapid construction can be achieved, improving the real-time performance of map construction; while optimizing the core map based on the supplementary image frames to obtain a high-precision map, the map elements in the core map can be effectively supplemented to improve the robustness of the high-precision map in subsequent positioning, realizing the simultaneous consideration of real-time performance and robustness in map construction, and obtaining a higher quality visual map.

[0047] In some embodiments, in addition to obtaining key map frames through the first filtering condition described above, a secondary filtering can be performed to obtain optimized key image frames and map points. Then, a core map can be constructed based on the optimized key image frames and their map points. Specifically, such as... Figure 3 As shown, step 130 above can specifically include:

[0048] Step 1301: Filter the key image frames according to the third filtering condition to obtain optimized key image frames and map points on the optimized key image frames;

[0049] In this step, we aim to select high-quality image frames and map points from the aforementioned key image frames as much as possible. In order to better control the real-time performance of core map construction, we can set certain constraints to limit the computational scale. At this time, the third selection condition mentioned above includes optimization target conditions and constraint conditions. The optimization target conditions can help select optimization key image frames, while the constraint conditions can limit the computational scale.

[0050] Specifically, the optimization conditions mentioned above may include at least one of the following: the number of times map points on key image frames are observed, the spatial uniformity of map points on key image frames, and the center distance between different key image frames. By setting certain thresholds for the number of observations, spatial uniformity, and center distance, key image frames can be filtered to obtain optimized key image frames. Based on these optimized key image frames, corresponding map points can be obtained. Thus, although fewer optimized key image frames are used for core map construction, the quality of the key image frames used can be guaranteed, and the accuracy of the constructed core map can be improved.

[0051] The constraints mentioned above can include the number of residual terms to be constructed. By limiting the number of residual terms, the number of key image frames and map points to be optimized can be effectively limited, thereby controlling the computational scale in the process of constructing the core map. Ultimately, this enables control over the time required to construct the core map, thereby improving real-time performance.

[0052] Step 1302: Construct a core map based on the optimized key image frames and the map points on the optimized key image frames.

[0053] In this step, based on the further filtering of key image frames in step 1302 above to obtain optimized key image frames and map points on the optimized key image frames, the pose and map point location information of the optimized key image frames are obtained, and then the core map is constructed, which can further improve the real-time performance of map construction and the accuracy of the constructed core map.

[0054] In some embodiments, during map construction using optimized key image frames and map points on those frames, local optimization processing of the obtained pose and map point location information of the optimized key image frames can be considered to improve the accuracy of the constructed core map. In this case, the specific steps for constructing the core map can be as follows: Figure 4 As shown, it includes the following steps:

[0055] Step 410: Obtain the pose and map point location information of the optimized key image frames;

[0056] In this step, the method for optimizing the pose of key image frames and the acquisition of map point location information can be carried out according to the method described in the above embodiments.

[0057] Step 420: Construct the first local optimization residual term;

[0058] In this step, a first locally optimized residual term can be constructed based on the visual observation error, and at least one of the IMU data, wheel speed sensor data, and GPS data. The visual observation error can be the reprojection error obtained from error analysis of visual image frames. The IMU data, wheel speed sensor data, and GPS data are obtained by the mapping method provided in this embodiment of the disclosure. In addition to acquiring visual image frames, the above data can be further acquired through other sensors. Then, the first locally optimized residual term is constructed based on all the above data.

[0059] Step 430: Perform local optimization processing on the pose and map point location information of the key image frame based on the first local optimization residual term;

[0060] This step, based on step 420 above, uses the first local optimization residual term obtained in the previous step as a constraint for local optimization. The specific objective function can be expressed as follows:

[0061]

[0062] In the above objective function formula, I represents the total number of image frames in the set of key image frames to be optimized, and K represents the total number of map points in the set of map points to be optimized. i Let P be the pose of the i-th keyframe image frame. k This represents the location information of the k-th map point. To constrain visual observation errors, For IMU data constraints, For wheel speed gauge data constraints, GPS data constraints. This is a residual term constructed based on visual observation errors, relating map point location information and keyframe image pose. This refers to the residual term for the pose of key image frames constructed based on IMU data. This is the residual term for the pose of key image frames, constructed based on wheel speed meter data. This is the residual term for the pose of key image frames, constructed based on GPS data.

[0063] Based on the first local optimization residual term as a constraint, local optimization processing can be performed to optimize the position information of key image frames and map points.

[0064] Step 440: Construct the core map based on the pose and map point location information of the optimized key image frames after local optimization processing.

[0065] Based on the local optimization of the pose and map point location information of the key image frames, the above pose and location information can be added to the existing core map to construct an updated core map.

[0066] In some embodiments, in addition to performing local optimization of the core map through the steps described above, the core map can also be optimized by calculating weight coefficients, as detailed in [reference needed]. Figure 4 It also includes:

[0067] Step 450: Calculate the weight coefficients for the poses of each optimized key image frame in the core map, and / or calculate the weight coefficients for the position information of each map point in the core map.

[0068] Specifically, this step may begin by recording the number of times each map point is observed in the optimization key image frames, the optimized position information of the map point, and the initial position information of the map point after triangulation of two consecutive image frames. Then, the confidence level of the map point is calculated by combining the above information to determine the weight coefficient of the map point's position information. For the weight coefficient of the pose in the optimization key image frames, it can be calculated based on information such as the number of times the map point is observed in the optimization key image frames. The weight coefficient calculation here is more inclined to distinguish the confidence level of the map point's position information because the optimization key image frames have more constraints and are generally more accurate, while the position information of map points is more prone to large fluctuations due to mismatches or small disparities.

[0069] The technical solution for constructing a core map according to the embodiments of this disclosure has been described in detail above. Through the technical measures such as obtaining optimized key image frames and performing local optimization on the core map, a core map with a certain degree of accuracy can be obtained, and the efficiency of constructing the core map is high, ensuring its real-time performance. In the above embodiments, supplementary image frames are obtained through a second filtering condition. These supplementary image frames can be applied to post-processing. In the post-processing stage, high-precision maps can be constructed using these supplementary image frames. The specific process can be found in the following embodiments.

[0070] Figure 5 This is a flowchart illustrating the post-processing stage in an embodiment of this disclosure. It can be the specific execution steps of step 140 in the above embodiments, as shown in Figure 5, and includes the following steps:

[0071] Step 510: Determine the optimized pose of the supplementary image frame and determine the position information of the map points in the supplementary image frame;

[0072] In this step, on the one hand, the supplementary image frames are triangulated to obtain the depth of the new map points, thereby obtaining the location information of the new map points. Specifically, each supplementary image frame can be triangulated sequentially according to the acquisition timestamp of each supplementary image frame to obtain the location information of the map points. During the triangulation process, if a feature point of the current supplementary image frame does not have a reasonably matching feature point in the subsequent N supplementary image frames, it can be considered that its credibility as a map point is low, and thus such feature points are removed without triangulation.

[0073] The pose of the supplementary image frame can be obtained during the preprocessing stage. After receiving the visual image frame sent by the camera device, the pose of each visual image frame is obtained through a preprocessing process and used as the initial pose. Therefore, the supplementary image frame in this step also has an initial pose. The constraints used when calculating the initial pose include, but are not limited to, visual observation errors, and constraints formed by at least one of IMU data, wheel speed meter data and GPS data.

[0074] This step is in the post-processing stage, where a core map has already been constructed using optimized key image frames. The poses of these optimized key image frames in the core map have undergone local optimization and weight coefficient calculation. Therefore, these optimized key image frames and supplementary image frames can be used together for optimization to obtain the optimized poses of the supplementary image frames. This optimization process can include two cases: first, optimization can be performed using only the optimized key image frames and supplementary image frames; second, optimization can be performed using all three—optimized key image frames, supplementary image frames, and non-optimized key image frames. When using the optimized poses of the image frames to perform triangulation to calculate the depth of map points and further obtain map point location information, more accurate map point location information can be obtained.

[0075] The specific execution process of this step may include:

[0076] First, a second local optimization residual term is constructed. This second local optimization residual term can be constructed based on visual observation errors, and at least one of IMU data, wheel velocity sensor data, and GPS data. This second local optimization residual term can serve as a constraint condition for pose optimization. Optimizing the pose based on this second local optimization residual term can improve the pose accuracy of the supplementary image frame. After constructing the second local optimization residual term, the pose of the optimized key image frame and the supplementary image frame can be optimized based on the second local optimization residual term, or the pose of the optimized key image frame, the non-optimized key image frame, and the supplementary image frame can be optimized based on the second local optimization residual term. The non-optimized key image frame is any image frame other than the optimized key image frame. The difference between the two is that the latter also optimizes the pose of the non-optimized key image frame. Specifically, the optimized key image frame with the best pose accuracy can be used as the starting and ending image frames, and a supplementary image frame can be inserted between them, or a non-optimized key image frame can also be inserted. The specific process can employ a weighted approach. Since the key image frames have already been optimized, they are considered to have higher reliability and can be assigned a higher weight coefficient, allowing for smaller adjustments in this local optimization. However, the inserted supplementary image frames, or both the non-optimized key image frames and supplementary image frames inserted simultaneously, can be given larger adjustments. The choice between optimizing supplementary image frames or simultaneously optimizing both non-optimized key image frames and supplementary image frames can be made based on actual needs. The objective function for this optimization can be expressed mathematically as follows:

[0077]

[0078] In the above objective function formula, I represents the total number of image frames in the set of image frames participating in the optimization. This set includes key image frames for optimization and supplementary image frames, and may also include non-key image frames for optimization. To constrain visual observation errors, This is a residual term for optimizing the pose of key image frames, constructed based on visual observation error constraints. This is a residual term constructed based on IMU data to optimize the pose of key image frames. This is a residual term for the pose of optimized key image frames, constructed based on wheel speed meter data. v is a residual term constructed based on GPS data to optimize the pose of key image frames. vp v ip v wp v gp The weight coefficient matrix can distinguish the confidence of pose of different types of image frames. The weight coefficients of the key image frames can be set to be larger.

[0079] Step 520: Add the optimized pose of the supplementary image frame and the map point location information in the supplementary image frame to the core map.

[0080] After optimizing the pose of the supplementary image frame determined in step 510, the optimized pose of the supplementary image frame and the map point location information in the supplementary image frame can be added to the core map to form a high-precision map. If the pose of the non-critical optimized image frame is also optimized in the above steps, the optimized pose of the non-critical optimized image frame and its corresponding map point location information can also be added to the core map to enrich the map elements in the high-precision map.

[0081] The local optimization process in the above embodiments of this disclosure is to optimize a local map or a portion of image frames. By optimizing, the pose of the image frames in the map becomes more accurate, thereby enabling the calculation of more accurate map point location information and providing the accuracy of the final high-precision map.

[0082] In other embodiments, based on the aforementioned local optimizations, the post-processing stage can further perform global optimizations on the core map; see references for details. Figure 5 It also includes:

[0083] Step 530: Perform global map optimization on the core map to obtain a high-precision map after optimization.

[0084] This step involves global map optimization of the core map, which means optimizing the residual constraints based on the poses of all image frames and the location information of map points. Specific steps may include:

[0085] A global optimization residual term is constructed, which can be based on visual observation errors and at least one of IMU data, wheel velocity sensor data, and GPS data. This global optimization residual term can serve as a constraint for optimizing the pose and map point position information of all image frames. Optimizing the pose based on this global optimization residual term can improve the pose accuracy and map point position information accuracy of each image frame. After constructing the global optimization residual term, the poses of optimized key image frames, supplementary image frames, and non-optimized key image frames can be optimized based on the global optimization residual term. The non-optimized key image frames are the other image frames among the key image frames excluding the optimized key image frames.

[0086] The pose optimization process can be achieved by using the optimized key image frame with the best pose accuracy as the starting and ending image frames, and inserting non-optimized key image frames and supplementary image frames in between. Specifically, a weighted approach can be used. Optimized key image frames, having already been optimized, are considered to have higher reliability and can be assigned a higher weight coefficient, allowing for smaller adjustments in this local optimization. In contrast, the inserted non-optimized key image frames and supplementary image frames can be given larger adjustments. The optimization can be performed on the supplementary image frames, or simultaneously on both, depending on the specific needs. Additionally, the map point location information involved in each image frame can also be optimized concurrently. The objective function for this optimization can be expressed mathematically as follows:

[0087]

[0088] In the above objective function formula, I represents the total number of image frames in the image frame set, and K represents the total number of map points in the map point set. i Let P be the pose of the i-th image frame. k This represents the location information of the k-th map point. To constrain visual observation errors, For IMU data constraints, For wheel speed gauge data constraints, GPS data constraints. This is a residual term constructed based on visual observation errors, relating map point location information and image frame pose. This refers to the residual term for the pose of an image frame, constructed based on IMU data. This is the residual term for the pose of the image frame, constructed based on wheel speed meter data. v is the residual term for the pose of the image frame constructed based on GPS data; vp v ip v wp v gpThe weight coefficient matrix can distinguish the confidence of pose of different types of image frames. The weight coefficients of the key image frames can be set to be larger.

[0089] Figure 6 This is a technical framework diagram of an embodiment of the present disclosure, such as... Figure 6 As shown, in the technical solution provided by the above embodiments of this disclosure, map construction is divided into a preprocessing stage and a post-processing stage. The preprocessing stage, also known as lightweight mapping, takes visual image frames acquired by a camera device as input. In some cases, at least one of IMU data, wheel speedometer data, and GPS data can also be input. These data mainly provide some auxiliary functions to improve the accuracy of visual mapping. In the lightweight mapping stage, a core map is constructed based on a small number of high-quality optimized key image frames, and its real-time performance and accuracy are well controlled. In addition, the lightweight mapping stage also filters and outputs supplementary image frames. In the post-processing stage, map elements are supplemented to the core map using the filtered supplementary image frames to obtain a high-precision map, which significantly improves the robustness of the final constructed map.

[0090] Figure 7 This is a schematic diagram of a lightweight mapping process in an embodiment of this disclosure. Figure 7 The lightweighting stage is shown in the figure. Input information includes visual image frames, and may also include at least one of IMU data, wheel speedometer data, or GPS data. Output includes a core map and supplementary image frames, such as... Figure 7 As shown, it includes the following steps:

[0091] Step 701, Feature Point Extraction and Descriptor Calculation: This step extracts feature points from the visual image frames captured by the camera device, obtaining a feature point set. This feature point set may include the location information of the feature points in the image, and descriptor calculation is performed to obtain the descriptive information of each feature point, i.e., descriptor vector information. The feature point extraction algorithm can be a traditional feature point extraction algorithm such as SURF, SIFT, ORB, or a deep learning algorithm such as SuperPoint.

[0092] Step 702: Map Initialization, which involves the initial construction of the core map to be built. In this step, pose transformation is solved based on the feature point matching relationship between two consecutive visual image frames. If the calculation is successful, the poses of the two visual image frames can be obtained, and triangulation can be performed based on these poses to obtain the feature point depths in the two frames, thereby obtaining the position information of map points and constructing the initial core map. If the construction fails, the initialization process continues. The initialized core map can be stored in the core map storage module. This core map is continuously optimized in subsequent steps 706 and 707.

[0093] Step 703, Feature Point Tracking: If map initialization has been successfully performed and the core map has been obtained, the core map can be used to continuously match feature points in subsequent input visual image frames (which have already undergone feature point extraction and descriptor calculation). The pose of subsequent input visual image frames can be solved using 2D to 3D conversion algorithms. The pose of each visual image frame can be solved. The 2D to 3D conversion algorithms mentioned above include, but are not limited to, DLT, P3P, and EPnP algorithms.

[0094] Step 704: Key Image Frame and Map Point Filtering. Key frame filtering in this step includes two stages: preliminary filtering and secondary filtering. The preliminary filtering selects key image frames from all visual image frames. Non-key image frames can be used to obtain supplementary image frames, and map points can be extracted from the key image frames to obtain new map point location information. The secondary filtering selects optimized key image frames from the key image frames; the remaining key image frames are considered non-optimized. Secondary filtering controls the number of optimized key image frames obtained, increasing the likelihood of obtaining optimized key image frames, thus enabling the construction of a core map with good real-time performance and accuracy. The specific execution process of this step can be found in the relevant descriptions in the above embodiments. In this step, non-optimized key image frames and their map point location information can also be stored in the core map.

[0095] Step 705, supplementary image frame screening, mainly involves supplementing the screening of non-critical image frames from step 704 above. The specific execution process can be found in the relevant descriptions in the above embodiments.

[0096] Step 706, Local Map Optimization, mainly involves optimizing the local map based on the optimized key image frames obtained in Step 704. The optimization object includes the core map already stored in the core map storage module. In this step, a first local optimization residual term can be constructed based on visual observation errors and at least one of IMU data, wheel speedometer data, and GPS data. This first local optimization residual term serves as a constraint condition for local map optimization. After local map optimization of the previously established core map, it can achieve higher accuracy. The specific execution process of this step can be found in the relevant descriptions in the above embodiments.

[0097] Step 707: Weight Coefficient Allocation. This step involves calculating weight coefficients for the poses of each optimized key image frame in the core map, and / or calculating weight coefficients for the location information of each map point in the core map. The core map after local map optimization and weight coefficient allocation will be re-stored in the core map storage module. The specific execution process of this step can be found in the relevant descriptions in the above embodiments.

[0098] Figure 8 This is a flowchart illustrating a post-processing procedure in one embodiment of the present disclosure. In the post-processing stage, the inputs are supplementary image frames and a core map, and the output is a high-precision map, such as... Figure 8 As shown, it includes the following steps:

[0099] Step 801, inserting supplementary image frames, mainly includes optimizing the pose of the supplementary image frames to obtain optimized poses, triangulating the supplementary image frames to obtain new map points, and adding the optimized poses of the supplementary image frames and the location information of the newly added map points to the input core map. This step enriches the map elements in the core map, thereby improving the robustness of the map. The specific execution process of this step can be found in the relevant descriptions in the above embodiments.

[0100] Step 802: Global map optimization. After adding the optimized pose of the supplementary image frames and the location information of the newly added map points to the input core map in the above steps, global map optimization can be performed to output the final high-precision map. Global map optimization can improve the accuracy of the map. The specific execution process of this step can be found in the relevant descriptions in the above embodiments.

[0101] Figure 9 This is a schematic diagram of a mapping device according to an embodiment of this disclosure. Figure 9 As shown, the device includes: an image frame acquisition module 910, an image frame filtering module 920, a map construction module 930, and a post-processing module 940.

[0102] Among them, the image frame acquisition module 910 is used to acquire visual image frames captured by the camera device;

[0103] The image frame filtering module 920 is used to filter the visual image frames to obtain key image frames and supplementary image frames.

[0104] The map building module 930 is used to build a core map based on the key image frames.

[0105] The post-processing module 940 is used to perform post-processing on the core map based on the supplementary image frames in the post-processing stage to obtain an optimized high-precision map.

[0106] In some embodiments, the image frame filtering module 920 described above is specifically used to filter each visual image frame according to a preset first filtering condition to obtain the key image frame; and to filter non-key image frames other than the key image frame according to a preset second filtering condition to obtain the supplementary image frame.

[0107] In some embodiments, the first filtering condition includes at least one of the following: the number of map points matching the current core map, the acquisition distance difference with the previous key image frame, and the acquisition angle difference with the previous key image frame.

[0108] The second filtering criteria include at least one of the following: the number of map points matching the current core map, the difference in acquisition distance between the current and two consecutive key image frames, and the difference in acquisition angle between the current and two consecutive key image frames.

[0109] In some embodiments, the map building module 930 described above is specifically used to filter the key image frames according to a third filtering condition to obtain optimized key image frames and map points on the optimized key image frames; and to build a map based on the optimized key image frames and map points on the optimized key image frames to construct a core map.

[0110] In some embodiments, the third screening condition includes optimization target conditions and constraints.

[0111] In some embodiments, the optimization objective conditions include at least one of the following: the number of times map points on key image frames are observed, the spatial uniformity of map points on key image frames, and the center distance between different key image frames.

[0112] The constraints include the number of residual terms to be constructed.

[0113] In some embodiments, the map building module 930 described above is specifically used to obtain the pose and map point location information of the optimized key image frame;

[0114] Construct the first local optimization residual term;

[0115] Based on the first local optimization residual term, the pose and map point location information of the optimized key image frame are locally optimized;

[0116] A core map is constructed based on the pose and map point location information of the optimized key image frames after the local optimization process.

[0117] In some embodiments, constructing the first locally optimized residual term includes:

[0118] The first local optimization residual term is constructed based on visual observation error and at least one of IMU data, wheel speedometer data, and GPS data.

[0119] In some embodiments, the map building module 930 described above is further specifically used to calculate the weight coefficients of the poses of each optimized key image frame in the core map, and / or to calculate the weight coefficients of the position information of each map point in the core map.

[0120] In some embodiments, the post-processing module 940 described above is specifically used to determine the optimized pose of the supplementary image frame and the position information of the map points in the supplementary image frame; and to add the optimized pose of the supplementary image frame and the position information of the map points in the supplementary image frame to the core map.

[0121] In some embodiments, determining the optimized pose of the supplementary image frame includes:

[0122] Construct the second local optimization residual term;

[0123] The poses of the optimized key image frames and supplementary image frames are optimized based on the second local optimization residual term.

[0124] In some embodiments, determining the optimized pose of the supplementary image frame includes:

[0125] Construct the second local optimization residual term;

[0126] The poses of the optimized key image frames, non-optimized key image frames, and supplementary image frames are optimized according to the second local optimization residual term. The non-optimized key image frames are the other image frames among the key image frames besides the optimized key image frames.

[0127] In some embodiments, the post-processing module 940 described above is further used to perform global map optimization on the core map to obtain a high-precision map after global map optimization.

[0128] In some embodiments, the above-mentioned optimization of the core map to obtain a high-precision map after global map optimization includes:

[0129] Construct globally optimized residual terms;

[0130] Based on the global optimization residual term, the pose and map point location information of all image frames contained in the core map are optimized and calculated to obtain the optimized high-precision map.

[0131] The mapping apparatus provided in this embodiment is capable of performing... Figures 1 to 8 The method embodiments shown are identical in content and have the same technical effects, and will not be repeated in this disclosure. Figure 10 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. See below for details. Figure 10 It shows a schematic diagram of a structure suitable for implementing the electronic device 500 in the embodiments of this disclosure. Figure 10 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0132] like Figure 10 As shown, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 501, which can perform various appropriate actions and processes to implement the methods of the embodiments described herein, based on a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device 500. The processing device 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0133] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts, thereby implementing the mapping method as described above. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the methods of embodiments of this disclosure.

[0134] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0135] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the aforementioned mapping method.

[0136] Optionally, when one or more of the above-described procedures are executed by the electronic device, the electronic device may also perform other steps described in the above embodiments.

[0137] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0138] Option 1: A mapping method, the method comprising:

[0139] Acquire visual image frames captured by the camera device;

[0140] The visual image frames are filtered to obtain key image frames and supplementary image frames;

[0141] A core map is constructed based on the key image frames.

[0142] In the post-processing stage, the core map is post-processed based on the supplementary image frames to obtain an optimized high-precision map.

[0143] Option 2, according to the method described in Option 1, the step of filtering visual image frames to obtain key image frames and supplementary image frames includes:

[0144] For each visual image frame, the visual image frame is filtered according to a preset first filtering condition to obtain the key image frame;

[0145] The non-critical image frames, excluding the critical image frames, are filtered according to the preset second filtering conditions to obtain the supplementary image frames.

[0146] Scheme 3: According to the method described in Scheme 2, the first filtering condition includes at least one of the following: the number of map points matching the current core map, the acquisition distance difference with the previous key image frame, and the acquisition angle difference with the previous key image frame.

[0147] The second filtering criteria include at least one of the following: the number of map points matching the current core map, the difference in acquisition distance between the current and two consecutive key image frames, and the difference in acquisition angle between the current and two consecutive key image frames.

[0148] Option 4: According to the method described in Option 1, the step of constructing a core map based on key image frames includes:

[0149] The key image frames are filtered according to the third filtering condition to obtain optimized key image frames and map points on the optimized key image frames.

[0150] A core map is constructed based on the optimized key image frames and the map points on the optimized key image frames.

[0151] Option 5: According to the method described in Option 4, the third screening condition includes optimization target conditions and constraint conditions.

[0152] Scheme 6: According to the method described in Scheme 5, the optimization objective conditions include at least one of the following: the number of times map points on key image frames are observed, the spatial distribution uniformity of map points on key image frames, and the center distance between different key image frames.

[0153] The constraints include the number of residual terms to be constructed.

[0154] Solution 7: According to the method described in Solution 4, the step of constructing a map based on the optimized key image frame and the map points on the optimized key image frame includes:

[0155] Obtain the pose and map point location information of the optimized key image frame;

[0156] Construct the first local optimization residual term;

[0157] Based on the first local optimization residual term, the pose and map point location information of the optimized key image frame are locally optimized;

[0158] A core map is constructed based on the pose and map point location information of the optimized key image frames after the local optimization process.

[0159] Option 8: According to the method described in Option 7, the construction of the first local optimization residual term includes:

[0160] The first local optimization residual term is constructed based on visual observation error and at least one of IMU data, wheel speedometer data, and GPS data.

[0161] Solution 9: According to the method described in Solution 8, the step of constructing a map based on the optimized key image frame and the map points on the optimized key image frame further includes:

[0162] The pose of each optimized key image frame in the core map is weighted and / or the position information of each map point in the core map is weighted.

[0163] Option 10: According to any one of Options 1-9, the post-processing of the core map based on supplementary image frames includes:

[0164] Determine the optimized pose of the supplementary image frame, and determine the position information of the map points in the supplementary image frame;

[0165] The optimized pose of the supplementary image frame and the map point location information in the supplementary image frame are added to the core map.

[0166] Solution 11: According to the method described in Solution 10, determining the optimized pose of the supplementary image frame includes:

[0167] Construct the second local optimization residual term;

[0168] The poses of the optimized key image frames and supplementary image frames are optimized based on the second local optimization residual term.

[0169] Solution 12: According to the method described in Solution 10, determining the optimized pose of the supplementary image frame includes:

[0170] Construct the second local optimization residual term;

[0171] The poses of the optimized key image frames, non-optimized key image frames, and supplementary image frames are optimized according to the second local optimization residual term. The non-optimized key image frames are the other image frames among the key image frames besides the optimized key image frames.

[0172] Solution 13: According to the method described in Solution 11 or Solution 12, the post-processing of the core map based on the supplementary image frames further includes:

[0173] The core map is then optimized globally to obtain a high-precision map.

[0174] Option 14: According to the method described in Option 13, the step of optimizing the core map to obtain a high-precision map after global map optimization includes:

[0175] Construct globally optimized residual terms;

[0176] Based on the global optimization residual term, the pose and map point location information of all image frames contained in the core map are optimized and calculated to obtain the optimized high-precision map.

[0177] Option 15: A mapping device, comprising:

[0178] The image frame acquisition module is used to acquire visual image frames captured by the camera device;

[0179] The image frame filtering module is used to filter the visual image frames to obtain key image frames and supplementary image frames.

[0180] The map building module is used to build a core map based on the key image frames.

[0181] The post-processing module is used to perform post-processing on the core map based on the supplementary image frames in the post-processing stage to obtain an optimized high-precision map.

[0182] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

Claims

1. A mapping method, characterized in that, include: Acquire visual image frames captured by the camera device; The visual image frames are filtered to obtain key image frames and supplementary image frames; A core map is constructed based on the key image frames. In the post-processing stage, the core map is post-processed based on the supplementary image frames to obtain an optimized high-precision map. The process of constructing a core map based on key image frames includes: The key image frames are filtered according to the third filtering condition to obtain optimized key image frames and map points on the optimized key image frames. A core map is constructed based on the optimized key image frames and the map points on the optimized key image frames.

2. The method according to claim 1, characterized in that, The process of filtering visual image frames to obtain key image frames and supplementary image frames includes: For each visual image frame, the visual image frame is filtered according to a preset first filtering condition to obtain the key image frame; The non-critical image frames, excluding the critical image frames, are filtered according to the preset second filtering conditions to obtain the supplementary image frames.

3. The method according to claim 2, characterized in that, The first filtering condition includes at least one of the following: the number of map points matching the current core map, the difference in acquisition distance with the previous key image frame, and the difference in acquisition angle with the previous key image frame. The second filtering criteria include at least one of the following: the number of map points matching the current core map, the difference in acquisition distance between the current and two consecutive key image frames, and the difference in acquisition angle between the current and two consecutive key image frames.

4. The method according to claim 3, characterized in that, The third screening criteria include optimization objective conditions and constraint conditions.

5. The method according to claim 4, characterized in that, The optimization objective conditions include at least one of the following: the number of times map points on key image frames are observed, the spatial uniformity of map points on key image frames, and the center distance between different key image frames. The constraints include the number of residual terms to be constructed.

6. The method according to claim 3, characterized in that, The map construction based on optimized key image frames and map points on the optimized key image frames includes: Obtain the pose and map point location information of the optimized key image frame; Construct the first local optimization residual term; Based on the first local optimization residual term, the pose and map point location information of the optimized key image frame are locally optimized; A core map is constructed based on the pose and map point location information of the optimized key image frames after the local optimization process.

7. The method according to claim 6, characterized in that, The construction of the first local optimization residual term includes: The first local optimization residual term is constructed based on visual observation error and at least one of IMU data, wheel speedometer data, and GPS data.

8. The method according to claim 7, characterized in that, The map construction based on the optimized key image frame and the map points on the optimized key image frame also includes: The pose of each optimized key image frame in the core map is weighted and / or the position information of each map point in the core map is weighted.

9. The method according to any one of claims 1-8, characterized in that, The post-processing of the core map based on supplementary image frames includes: Determine the optimized pose of the supplementary image frame, and determine the position information of the map points in the supplementary image frame; The optimized pose of the supplementary image frame and the map point location information in the supplementary image frame are added to the core map.

10. The method according to claim 9, characterized in that, Determining the optimized pose of the supplementary image frame includes: Construct the second local optimization residual term; The poses of the optimized key image frames and supplementary image frames are optimized based on the second local optimization residual term.

11. The method according to claim 9, characterized in that, Determining the optimized pose of the supplementary image frame includes: Construct the second local optimization residual term; The poses of the optimized key image frames, non-optimized key image frames, and supplementary image frames are optimized according to the second local optimization residual term. The non-optimized key image frames are the other image frames among the key image frames besides the optimized key image frames.

12. The method according to claim 10 or 11, characterized in that, The post-processing of the core map based on the supplementary image frames further includes: The core map is then optimized globally to obtain a high-precision map.

13. The method according to claim 12, characterized in that, The process of performing global map optimization on the core map to obtain a high-precision map after global map optimization includes: Construct globally optimized residual terms; Based on the global optimization residual term, the pose and map point location information of all image frames contained in the core map are optimized and calculated to obtain the optimized high-precision map.

14. A mapping device, characterized in that, include: The image frame acquisition module is used to acquire visual image frames captured by the camera device; The image frame filtering module is used to filter the visual image frames to obtain key image frames and supplementary image frames. The map building module is used to build a core map based on the key image frames. The post-processing module is used to perform post-processing on the core map based on the supplementary image frames in the post-processing stage to obtain an optimized high-precision map. The step of constructing a core map based on key image frames includes: The key image frames are filtered according to the third filtering condition to obtain optimized key image frames and map points on the optimized key image frames. A core map is constructed based on the optimized key image frames and the map points on the optimized key image frames.

15. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-13.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-13.

Images