Map building and repositioning method in complex dynamic scene, medium and product

Abstract

The invention discloses a map construction and relocation method in a complex dynamic scene, a medium and a product, and relates to the field of map construction and relocation, and the method comprises the steps: obtaining an original image and a to-be-located image; based on the original image, a semantic segmentation result of the original image is obtained by using a deep learning model; determining angular points based on the original image; based on the angular points and the original image, extracting descriptors; constructing a map based on the semantic segmentation result, the angular points and the descriptors; based on the map, adopting a map optimization method to obtain an optimized map; obtaining clustering result information and key frame information of the image features based on the optimized map by adopting a DBoW3 library; constructing a dictionary based on clustering result information of the image features, and storing key frame information in a database; and based on the optimized map, dictionary and database, the to-be-positioned image is repositioned by adopting a bag-of-word model matching method, and accurate map construction and accurate and rapid positioning can be realized in a complex dynamic scene.

Description

Technical Field

[0001] This application relates to the field of map construction and relocalization, and particularly to a method, medium, and product for map construction and relocalization in complex dynamic scenarios. Background Art

[0002] The Simultaneous Localization and Mapping (SLAM) technology enables robots and autonomous vehicles to determine their own positions while mapping the environment in an unknown environment. To efficiently complete tasks such as navigation, mobile robots highly rely on the accuracy of the map. In a system using visual SLAM, the robot mainly uses a camera to achieve self-localization and construct a map of the surrounding environment. Since visual SLAM can obtain a large amount of rich visual data, it is widely favored. However, a basic assumption of traditional visual SLAM systems is that the environment is static, which makes it extremely challenging to maintain accurate localization and mapping in an environment with many dynamic objects. Summary of the Invention

[0003] The purpose of this application is to provide a method, medium, and product for map construction and relocalization in complex dynamic scenarios, which can achieve accurate map construction and precise and rapid localization in complex dynamic scenarios.

[0004] To achieve the above purpose, this application provides the following solutions:

[0005] In the first aspect, this application provides a method for map construction and relocalization in complex dynamic scenarios, including:

[0006] Obtain the original image and the image to be localized;

[0007] Based on the original image, use a deep learning model to obtain the semantic segmentation result of the original image;

[0008] Based on the original image, determine the corner points;

[0009] Based on the corner points and the original image, extract descriptors;

[0010] Based on the semantic segmentation result, the corner points, and the descriptors of the original image, construct a map;

[0011] Based on the map, use a graph optimization method to obtain an optimized map;

[0012] Use the DBoW3 library to obtain the clustering result information of image features and the key frame information based on the optimized map;

[0013] Construct a dictionary based on the clustering result information of image features and store the key frame information in a database;

[0014] Based on the optimized map, the dictionary, and the database, a bag-of-words model matching method is used to relocalize the image to be located.

[0015] Optionally, based on the original image, a deep learning model is used to obtain the semantic segmentation result of the original image, including:

[0016] Using a deep learning model, based on the original image, obtain the core, category, and label of the semantic object; the core of the semantic object is the central pixel coordinates of the semantic object;

[0017] Using the SAM model, based on the core, category, label of the object, and the original image, obtain the semantic segmentation result of the original image.

[0018] Optionally, the deep learning model is the SparseInst model.

[0019] Optionally, based on the original image, determine the corner points, including:

[0020] Based on the original image, use a corner detection algorithm to obtain the initial corner points;

[0021] Based on the initial corner points, use a quadtree algorithm to obtain the corner points.

[0022] Optionally, the corner detection algorithm is the FAST corner detection algorithm with an adaptive threshold.

[0023] Optionally, the expression of the adaptive threshold is:

[0024]

[0025] where s is the adaptive threshold, I a is the average grayscale value of the image, and I i is the grayscale value of the i-th pixel point.

[0026] Optionally, based on the corner points and the original image, extract descriptors, including:

[0027] Based on the corner points and the original image, use a binary descriptor algorithm to extract descriptors.

[0028] Optionally, based on the optimized map, the dictionary, and the database, using the bag-of-words model matching method to relocalize the image to be located includes:

[0029] Determine the bag-of-words vectors of all frames of the image to be located based on the dictionary;

[0030] Based on the bag-of-words vectors and the database, determine the candidate relocalization key frames;

[0031] Based on all frames of the image to be located and the alternative relocalization key frames, the bag-of-words model matching method is used to determine the relocalization key frames;

[0032] Based on the relocalization key frames, a PnP solver is used to determine the pose of the image to be located in the optimized map.

[0033] In a second aspect, the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the map construction and relocalization method in any one of the above are implemented.

[0034] In a third aspect, the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, the steps of the map construction and relocalization method in any one of the above are implemented.

[0035] According to the specific embodiments provided by the present application, the present application has the following technical effects:

[0036] The present application provides a map construction and relocalization method, medium and product in a complex dynamic scenario. By using a deep learning model to perform semantic segmentation on the original image, semantic segmentation in a complex dynamic scenario can be realized, so that dynamic targets in the scenario can be marked, and accurate construction of the map can be achieved. By using the DBoW3 library to construct a dictionary and a database, the clustering result information of image features and key frame information are stored in the optimized map, and further, the bag-of-words model matching method is used to achieve precise and fast positioning of the image to be located. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0038] Figure 1 It is a schematic flowchart of a map construction and relocalization method in a complex dynamic scenario according to an embodiment of the present application;

[0039] Figure 2 It is a schematic diagram of the SparseInst model provided by an embodiment of the present application;

[0040] Figure 3 It is a schematic diagram of semantic segmentation results in different scenarios provided by an embodiment of the present application;

[0041] Figure 4Schematic diagram of the constructed map provided by an embodiment of the present application;

[0042] Figure 5 Schematic diagram of the dictionary construction process provided by an embodiment of the present application;

[0043] Figure 6 Schematic diagram of the relocalization result provided by an embodiment of the present application;

[0044] Figure 7 Schematic diagram of the algorithm framework of the map construction and relocalization method under complex dynamic scenarios provided by an embodiment of the present application;

[0045] Figure 8 Schematic diagram of the algorithm ROS node relationship provided by an embodiment of the present application;

[0046] Figure 9 Schematic diagram of the structure of a computer device provided by an embodiment of the present application. Detailed implementation manners

[0047] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts shall fall within the protection scope of the present application.

[0048] To make the above objects, features, and advantages of the present application more obvious and understandable, the present application will be further described in detail below in conjunction with the accompanying drawings and specific implementation manners.

[0049] In an exemplary embodiment, as Figure 1 shown, a map construction and relocalization method under complex dynamic scenarios is provided, including the following steps 100 to step 107. Among them:

[0050] Step 100, obtain the original image and the image to be located.

[0051] Step 101, based on the original image, use a deep learning model to obtain the semantic segmentation result of the original image.

[0052] Step 102, based on the original image, determine the corner points.

[0053] Step 103, based on the corner points and the original image, extract the descriptors.

[0054] Step 104, based on the semantic segmentation result, corner points, and descriptors of the original image, construct a map.

[0055] Step 105: Based on the map, use the graph optimization method to obtain an optimized map.

[0056] Step 106: Use the DBoW3 library to obtain the clustering result information of image features and key frame information based on the optimized map; construct a dictionary based on the clustering result information of image features and store the key frame information in the database.

[0057] Step 107: Based on the optimized map, dictionary, and database, use the bag-of-words model matching method to relocalize the image to be localized.

[0058] Implementing the above Steps 100 to 107 can achieve accurate map construction and precise and fast localization in complex dynamic scenarios.

[0059] In another exemplary embodiment of this application, in order to accurately obtain semantic feature information in complex dynamic scenarios, the original image can be used as the input of a deep learning model to obtain the semantic segmentation result of the original image. Among them:

[0060] The deep learning model used in this application can be the SparseInst model. As Figure 2 shown, the backbone network of the deep learning model uses a residual network (Residual Network, ResNet). Taking the original image as the input, a 3×3 convolutional layer is used to perform preliminary feature extraction on the original image, and the preliminarily extracted features are passed to different layers of the feature pyramid network. The feature pyramid network includes three feature layers with resolutions of C3, C4, and C5. Among them, the pyramid pooling module is introduced in the C5 layer to capture multi-scale context information by expanding the receptive field. After being processed by the pyramid pooling module, the features of the C5 layer are fused with the features of the C4 and C3 layers layer by layer, and the features after each layer of fusion are compressed in channels through a convolutional layer, thereby generating a single-level feature, where Figure 2 4x or 2x in

[0061] represents an upsampling factor of 4 or 2. The fused single-level feature is passed to the instance context encoder. The instance context encoder further enhances the context expression ability of the features through convolutional operations, compresses the input features to 1 / 4 resolution, and at the same time ensures the compactness and multi-scale fusion effect of the features. The encoder finally outputs a single-level feature with a resolution of 1 / 18 of the original input image and enters the Instance Activation Maps (IAM) module.

[0062] The IAM module is the core part of the decoder. It generates instance activation maps through weighted mapping of perceptual instances, highlighting the key regions of each instance object. Through a secondary matching mechanism, it ensures that each instance activation map targets only a single object, thus significantly enhancing the instance perception ability of the deep learning model. Under the processing of the decoder, the SparseInst model finally outputs three sets of data, namely the core, category, and score of the object. Among them, the core represents the pixel coordinates of the center of each semantic object; the category is the semantic category of the object, indicating which predefined category the object recognized by the deep learning model belongs to, such as pedestrians, cars, bicycles, traffic signs, trees, or buildings, etc.; the score is the confidence score of the deep learning model for each semantic object, indicating the probability that the semantic object belongs to the predicted category.

[0063] The output data of the SparseInst model is combined with the original image as the input of the SAM model to generate the segmentation mask of the object. The core task of the SAM model is to identify the target regions in the image and generate activation maps of foreground objects based on the core coordinates. In this process, the SAM model relies on the deep neural network to extract global features of the image, combines the output data of the SparseInst model as sparse cues, highlights and segments the foreground regions. Finally, the output of the SAM model is the segmentation results of each target in the image, as Figure 3 shown in the different scene segmentation results. Among them, Figure 3 part (a) shows the segmentation result of the scene containing buses, part (b) shows the segmentation result of the scene containing pedestrians, and part (c) shows the segmentation result of the scene containing cars. These segmentation results can identify the key information regions in the image. Optimize the boundaries of each region to obtain more accurate segmentation results. The SparseInst model and the SAM model work together to achieve more accurate semantic segmentation.

[0064] In another exemplary embodiment of this application, the process of map reconstruction based on the semantic segmentation results is given as follows:

[0065] First, corner detection: The traditional corner detection algorithm (Features From Accelerated Segment Test, FAST) calculates the difference in grayscale values between a corner point and 16 points on its circumference. If the difference in grayscale values of 9 or 12 consecutive pixel points meets the threshold, then P is determined to be a corner point. The size of the threshold determines the number of feature points extracted, and the size of the threshold is inversely proportional to the extracted corner points. However, the threshold in the FAST algorithm is set manually and has poor anti-interference ability. Moreover, as the light changes, the brightness of the images captured by the vision sensor also changes. If a fixed threshold is continued to be used, the probability of false extraction by the system will increase. Furthermore, the fixed threshold makes it impossible to adapt to the extraction of feature points in different environments, and the problem that the extracted feature points are unevenly distributed on the image and easily form feature point blocks. In view of this situation, this application uses a FAST corner detection algorithm with an adaptive threshold to solve the above problems. The adaptive threshold is calculated through the grayscale information of the image, so that the threshold can change according to the change of the grayscale information of the image. The calculation formula is expressed as:

[0066]

[0067] where s is the adaptive threshold, I a is the average grayscale value of the image, and I i is the grayscale value of the i-th pixel point. Compared with the traditional corner detection algorithm, the average value after removing the maximum and minimum values is used for calculation. By dynamically adjusting the threshold, it is ensured that enough corner points can be stably extracted under different lighting and texture conditions.

[0068] Since there are still a large number of redundancies in the extracted FAST corner points, a quadtree algorithm is used in this application to further screen these corner points to avoid the over-concentration of feature points in the high-frequency region and the shortage in the low-frequency region. The quadtree algorithm in this application also has the ability to filter dynamic objects (such as people and vehicles), ensuring that the extracted feature points mainly come from static objects, thereby improving the stability of the map. The steps are as follows:

[0069] Step 1: Input the unassigned feature points in the image.

[0070] Step 2: Construct an initial root node containing a piece of image area according to the image area, and each root node also includes four child nodes.

[0071] Step 3: Assign the unassigned feature points to the root node in Step 2, and all its feature points are included in this root node;

[0072] Step 4: Determine whether the key points of the current root node can be further allocated to its four sub-nodes; if they can be allocated, the separated sub-nodes are used as the new root nodes. If all nodes cannot be allocated, or the number of nodes has exceeded the required number, stop splitting, as Figure 4 shown in the optimized composition result based on the quadtree algorithm.

[0073] Second, descriptor extraction: After completing FAST corner detection, the feature points are represented by a binary descriptor in vector form, simplifying the calculation of the descriptor to directly comparing the gray values of image points, without the need for a training process. It is not only fast in calculation but also saves memory, effectively reducing the storage space required for the point cloud map. Descriptor extraction is a key step after corner extraction, providing an efficient and reliable feature representation for subsequent feature matching and map construction.

[0074] In practical applications, with the help of the pixel density function, a more accurate three-dimensional gray center point can be designed to determine the direction of the descriptor and maintain rotational invariance in the spherical perspective model. The descriptor similarity is calculated using the Hamming distance, which is easier to calculate compared to the L2 norm. Randomly select a pair of pixel points x and y within the defined neighborhood around the key point. This defined neighborhood is called p, which is a pixel square area surrounding the feature point. If the gray value of the first pixel point is higher than that of the second pixel point, the corresponding bit is assigned 1, otherwise 0. We select n pairs of pixel points to obtain an n-bit descriptor.

[0075] Define the test τ (a binary test defined on the image patch) on the image neighborhood p of size S*S as:

[0076]

[0077] where p(x) is the gray value of the smoothed image neighborhood p at point x, and p(y) is the gray value of the smoothed image neighborhood p at point y. Select n pairs of pixel points at different positions and uniquely describe the pixel pairs with a binary string. The generated n-dimensional binary descriptor string f n (p) is:

[0078] f n (p) = ∑ 1≤j≤n 2 j-1 τ(p; x j , y j ).

[0079] where x j is the pixel point x of the jth pair of pixel points, and y jFor the pixel point y of the j-th pixel pair, where j = 1,..., n is the number of pixel pairs. This application considers the cases where n equals 128, 256, and 512. Experimental results show that these data can achieve a good trade-off among speed, storage efficiency, and recognition rate.

[0080] Third, map construction: The extracted corner points and their descriptors are used to construct the topological structure of the map. Among them, the corner points serve as nodes in the map, and the descriptors are used to calculate the weights of the edges to achieve efficient matching between nodes.

[0081] In practical applications, unreliable feature points and edges can be further filtered out to ensure the robustness and sparsity of the map, and finally a global semantic point cloud map is generated. Among them, the filtering process is as follows: 1. Adopt embedded selection or other feature methods. According to the importance or correlation analysis of features, clearly eliminate those feature points that contribute little to map construction. This process can be achieved by calculating the importance index of features (such as the number of segmentation times, information gain, etc. of features in map construction). 2. Screen the edges of the already constructed map. By analyzing factors such as the connection strength of the edges and their contribution to map correlation, further eliminate those edges that contribute little to the structure of the map.

[0082] Fourth, camera pose optimization: Use graph optimization technology to jointly optimize the camera pose and the positions of 3D map points to improve the accuracy of the map. By tracking the feature point matching between the current frame and the reference key frame, the PnP method is used to estimate the camera pose. Then, combined with the local map for relocalization and loop closure detection, and the sliding window optimization method is used to optimize the sliding window variables of the key frames and map points to improve the accuracy and robustness of the map construction method provided by this application.

[0083] In practical applications, the determination of the camera pose is as follows:

[0084] According to the three-dimensional coordinate transformation law and the coordinates in the camera coordinate system, the expression of the coordinate mapping relationship P″ is:

[0085] p” = ∏(P c )).

[0086] Among them, P c ” is the projection point.

[0087] The expression of the projection point P c ” in the map is:

[0088] P c ” = R cb P b ”.

[0089] Among them, R cbIt is the pose matrix of feature points.

[0090] The coordinate mapping relationship can be further expressed as:

[0091] p” = ∏(R cb P b ”).

[0092] In an exemplary embodiment, the process of relocating the image to be located based on the optimized map is given as follows:

[0093] First, construct a dictionary and a database: In this application, the DBoW3 library is used to complete the construction of the dictionary and the database. The process of constructing the dictionary is as Figure 5 shown. First, it is necessary to extract feature points from the optimized map, then use the kmeans++ method to initialize the map feature points, randomly select the first feature point as the first clustering center from these feature points, and then calculate the distance D(x) between all feature points and the first clustering center. Next, follow the principle that the larger the probability of D(x), the more likely to select a new clustering center. Finally, repeat the above steps continuously until k clustering centers are generated, and construct a dictionary based on this clustering result information. The database in this application stores the key frame information of the optimized map, where the key frame is a representative image frame in the map.

[0094] According to the prior art, the dictionary is the basis for constructing the bag-of-words model, which provides a bridge for mapping image features to visual words. The bag-of-words model then uses the dictionary to convert the image into a concise and easy-to-process representation form, thus facilitating subsequent image processing and analysis tasks. The bag-of-words model is used to describe a frame of image, which is composed of the words in the dictionary that appear in the image and is stored in vector form. The dictionary is generally generated from the feature descriptors of each frame in the dataset. Each word corresponds to a similar type of feature descriptor, and its weight format and feature descriptor can be used to distinguish different features. Calculating the word weights can make the distinction between different frames better. Specifically, the TF-IDF method can be used to calculate the word weights.

[0095] Specifically, for a certain frame of image, let the number of word categories that have appeared be m, the number of times a certain word appears be mi, the number of dataset features be k, and the number of features belonging to a certain word category be ki. Then:

[0096]

[0097] Among them, TFi is the frequency of occurrence of a certain word in the document, and IDFi is the inverse document frequency.

[0098] In practical applications, the process of accelerating matching can be achieved by combining a dictionary with a bag-of-words model. The following uses an example to specifically illustrate the matching process. First, define an index table from words to corresponding frame indices. After finding the corresponding word based on the features of the current frame, all frames in which the word has appeared can be found from the index table. If two frames are similar, they will have more identical words. Therefore, the search range of potentially similar frames can be narrowed down through this index table, and then the BoW is used to calculate the similarity. Define an index table from nodes in a certain layer of the kd-tree to key frames: when calculating the similarity between two frames, only the words that have appeared in both frames under the nodes in this layer need to be compared, which can accelerate the finding of identical words that have appeared.

[0099] Second, the relocalization process: Determine the bag-of-words vectors of all frames of the image to be localized based on the dictionary; Determine the candidate relocalization key frames based on the bag-of-words vectors and the database; Determine the relocalization key frames using the bag-of-words model matching method based on all frames of the image to be localized and the candidate relocalization key frames; Based on the relocalization key frames, use the PnP solver to determine the pose of the image to be localized in the optimized map.

[0100] The bag-of-words vector is a vector used to describe an image, mainly composed of the dictionary word index and the weight of the dictionary word. Calculating the similarity between images is actually calculating the distance between the bag-of-words vectors of the images. The feature vector is used to accelerate the matching of images. In this application, multiple feature points in an image are divided into many clusters according to the descriptors. Since the standard for dividing the descriptors is provided by the dictionary, it is unified among different images. For a feature point in an image, it stores the node ID and the index corresponding to the feature point in the image. The images can be classified according to the descriptors. When this method needs to match two images, only the feature points in each image need to be matched in turn. Compared with direct matching, clustering the feature points in advance can effectively reduce the amount of matching calculation and improve the matching efficiency.

[0101] In practical applications, when a new frame of image comes in, it is necessary to calculate the bag-of-words vector for this image and then compare it with the bag-of-words vectors of the previously existing frame images. Specifically: convert the descriptors of the binary features in the current frame into vector form to obtain the bag-of-words vector of the current image frame, and search for key frames in the key frame database based on the similarity of the bag-of-words vectors, so as to obtain potential relocalization key frames. Among them, the steps of searching for key frames in the key frame database include: obtaining key frames with co-visibility relationships, filtering according to BoW similarity, and filtering according to co-visibility relationships. After obtaining a bunch of potential relocalization key frames, it is necessary to traverse them one by one and try to match the potential relocalization key frames with the current image frame using the bag-of-words model matching method. The number of matched map points is used as the return value, and further judge the number of matched map points. If the number of matched map points is less than the threshold, skip this loop; otherwise, proceed to the next step of solving the pose. Finally, construct a PnP solver to solve the pose for each relocalization key frame. If a solved pose filters a relatively large number of corresponding map points after optimization, it is considered that relocalization is successful; otherwise, it is determined to be a failure. For each potential relocalization key frame, perform 5 iterations to obtain the transformation matrix, so as to determine the pose based on the transformation matrix; if it does not converge after 5 iterations, discard this relocalization key frame. Assume that the PnP solver successfully calculates the pose within 5 times, and then verify the pose. Further use the call function to try to optimize the solved pose and return the number of optimized map points. If the number of optimized map points is less than the threshold, skip this loop and consider that this relocalization key frame is unreliable; if the number of optimized map points exceeds the threshold but is relatively close, calculate the pose again with a relatively loose range. If the final number of map points is much greater than the threshold, it is considered that this relocalization key frame is relatively reliable.

[0102] When the area covered by the map increases, the number of key frames will also increase accordingly. Therefore, there will also be scenes with similar description vectors, especially in areas where the scene is relatively simple and the repetition rate is high. At this time, there will be several key frames with a high degree of similarity to the current scene. Therefore, in order to distinguish several similar scenes and improve the matching accuracy, the ICP method can be added for secondary matching on the basis of the original bag-of-words model matching. First, select the top several key frames with the highest similarity as candidate point clouds by the bag-of-words model, and then perform ICP matching on each group of candidate point clouds with the zero matrix as the initial transformation matrix. After ICP matching, a fitness score will be returned as the similarity. Finally, the matching score of each group of candidate point clouds is obtained by multiplying the bag-of-words model similarity and the ICP matching similarity, and the one with the highest score is selected as the final matching point cloud, so as to obtain a more accurate initial pose as shown in Figure 6 the map.

[0103] In an exemplary embodiment, based on asFigure 7 The algorithm framework of the map construction and relocalization method in the complex dynamic scenario shown. In this embodiment, the implementation process can be as follows: First, the original data (image in this application) is semantically recognized and then input into a lightweight and robust feature point extractor for feature point extraction. Then, sparse map reconstruction is performed, including: a tracking thread, key frames, and a mapping thread, and a key frame database is constructed based on the reconstructed map. Finally, loop detection and relocalization are performed. Based on this, the present application can also provide a system for implementing the algorithm of the map construction and relocalization method in a complex dynamic scenario. The system can include: an environment semantic perception module based on deep learning (including: a semantic recognition sub-module and a lightweight and robust feature point extractor sub-module in this application) and a sparse representation and reconstruction module based on environmental features (including: a sparse map reconstruction sub-module, a loop detection and relocalization sub-module, and a key frame database sub-module in this application), as follows:

[0104] First, the environment semantic perception module based on deep learning undertakes the key task of semantic segmentation processing on the input visual image. Specifically, various objects in the scene are recognized and classified through a deep learning network to generate a mask containing dynamic objects. This mask can not only effectively mark dynamic targets such as pedestrians, vehicles, and non-motor vehicles in the scene, but also has a high accuracy rate to ensure that it matches the contour of the actual object.

[0105] To ensure the real-time performance and accuracy of data processing, the environment semantic perception module based on deep learning strictly aligns the generated mask with the original image through a time synchronization mechanism. Using the messagefilter function in ROS, unified timestamp processing is performed on data from different sources to ensure that image and semantic information can be processed in the same time frame for subsequent processing. The node relationship of the algorithm ROS is as Figure 8 shown.

[0106] After completing semantic segmentation and time synchronization, all processing results are packaged into messages and finally published through the ROS topic " / yolo_results". These messages will be transmitted to the subsequent sparse representation and reconstruction module based on environmental features for further processing of environmental information, such as the extraction of static objects in the scene and map reconstruction. This modular design can achieve efficient division of labor and cooperation, a complete closed-loop from environmental perception to sparse representation, and improve the real-time performance and stability of the overall algorithm.

[0107] The semantic perception module based on deep learning in this application is implemented using the semantic perception algorithm based on deep learning. The following table 1 shows the specific process of the specific implementation method of the semantic perception algorithm based on deep learning.

[0108] Table 1 Flowchart of Semantic Perception Algorithm Based on Deep Learning

[0109]

[0110]

[0111]

[0112] Second, the sparsification expression and reconstruction module based on environmental features needs to process sensor information and calculate the pose of the current image frame in the activation map in real time, and also determines whether to use the current frame as a key frame. When tracking is lost, this thread will try to re - locate the current frame in the Atlas map. If the re - location is successful, tracking will be restored, and the activation map will be switched when necessary. If activation is not successful after a period of time, the activation map will be stored as an unactivated map, and a new activation map will be re - initialized. Therefore, there will be an activation map set and an unactivated map set in the map pool. When a new key frame and map points are added, this information will be added to the current activation map to enrich the available data of the map. At the same time, redundant key frames will be deleted to avoid the redundant impact of too much invalid data on the map, thus improving efficiency. To optimize the map, the sparsification expression and reconstruction module based on environmental features will operate on the surrounding key frames of the current frame, using Bundle Adjustment (BA) or Visual - Inertial BA technology to ensure the accuracy of map and pose estimation.

[0113] In inertial mode, the mapping thread will combine IMU data and use the maximum a posteriori estimation technology to initialize and optimize the relevant parameters of the IMU, ensuring good fusion and consistency between visual and inertial data.

[0114] Whenever a new key frame is added to the system, the mapping thread will detect whether there is a common area overlapping with other key frames in the activation map and the entire Atlas map. If the common area belongs to the current activation map, the system will perform loop closure correction to adjust and correct the local map; if the common area belongs to other maps, the system will fuse these maps into a whole, and the new fused map will be used as the new activation map.

[0115] After performing loop closure correction, the system also starts an independent thread to perform global BA operation. Global BA can further optimize the entire map to ensure the consistency and accuracy of the map. At the same time, since this operation is executed asynchronously in the thread, it will not affect the real-time performance of the system. A similar design also exists in the ORB-SLAM2 module structure. The sparse representation and reconstruction technology and the fast relocalization module design in this application are shown in Table 2. This modular design ensures that the system in this embodiment can maintain high efficiency in real-time processing of large-scale scenes and provide accurate map construction and optimization at the same time.

[0116] Table 2 Sparse Representation and Reconstruction Technology, Fast Relocalization Module and Module Function Description Table

[0117]

[0118]

[0119] The algorithm implementation process of this application has the following advantages compared with the prior art:

[0120] (1) The system is relatively lightweight: mainly because a large number of computational processes of 8 iterations are cancelled in the network and replaced by convolution, normalization and activation in a certain way, thus reducing the computational amount.

[0121] (2) The results are relatively better: adopting a residual-like structure can retain the original features to a greater extent, and using smoothness loss as part of the loss function retains the authenticity of the results and can deepen the details.

[0122] In terms of the deployment of the software environment, aiming at the special architecture characteristics of the Jetson Orin board, a programming strategy based on the ROS framework is clearly adopted at the initial stage of algorithm design to give full play to the computing power and parallel processing advantages of the Jetson Orin. In the system, the environment semantic perception module driven by deep learning is mainly written in Python to ensure high flexibility and scalability when processing large-scale data. The core algorithms for environmental semantic map reconstruction and semantic feature-based relocalization and loop closure detection are written in C++ to ensure the real-time performance and accuracy of key tasks with the help of the high computing power of C++.

[0123] To further accelerate the processing of semantic segmentation, the system installs the OpenCV library with CUDA acceleration support. This technology can utilize the GPU resources of Jetson Orin to significantly improve the speed of image processing and feature extraction. In the overall software environment, Jetpack 5.1.1 is a key component, which provides comprehensive driver support and development tools for the system, especially optimized for deep learning and computer vision tasks. Through the integration of Jetpack, the system can make full use of libraries such as CUDA and cuDNN to accelerate the inference and training of various deep learning models.

[0124] At the algorithm implementation level, the system specifically utilizes the multi-threading features in C++14, including: "std::thread" for creating parallel threads and std::chrono for high-precision time control. Through these two features, the system realizes the synchronous operation of three threads for tasks such as semantic recognition, object tracking, loop detection, and relocalization. This multi-threading architecture not only improves the coordination between tasks but also significantly enhances the overall computing efficiency of the system, especially performing excellently in scenarios with high real-time requirements.

[0125] In summary, through deep customization and optimization of the architectural features of the Jetson Orin board, high computing efficiency and real-time processing capabilities are achieved in software design and deployment. The specific environmental information is shown in Table 3 below, which presents the configuration of each module in the system and its corresponding performance optimization strategies.

[0126] Table 3 Deployment of Environment and Dependent Libraries

[0127]

[0128] In an exemplary embodiment, a computer device is provided. The computer device can be a server or a terminal, and its internal structure diagram can be as Figure 9As shown in the figure. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store map construction and relocalization data in complex dynamic scenarios. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through a network connection. When the computer program is executed by the processor, it implements a method for map construction and relocalization in complex dynamic scenarios.

[0129] Those skilled in the art can understand that Figure 9 the structure shown in the figure is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above method embodiments are implemented.

[0130] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0131] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0132] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.

[0133] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memory can include Read-Only Memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0134] The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.

[0135] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0136] Specific examples are used in this article to elaborate on the principles and implementation manners of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A method for map construction and relocalization in complex dynamic scenarios, characterized in that The method for map construction and relocalization in the complex dynamic scenario includes: Obtain the original image and the image to be localized; Based on the original image, use a deep learning model to obtain the semantic segmentation result of the original image; Based on the original image, determine the corner points; Based on the corner points and the original image, extract descriptors; Based on the semantic segmentation result, the corner points, and the descriptors of the original image, construct a map; Based on the map, adopt a graph optimization method to obtain an optimized map; Use the DBoW3 library to obtain the clustering result information of image features and the key frame information based on the optimized map; Construct a dictionary based on the clustering result information of image features and store the key frame information in a database; Based on the optimized map, the dictionary, and the database, adopt a bag-of-words model matching method to relocalize the image to be localized.

2. The method for map construction and relocalization in a complex dynamic scenario according to claim 1, wherein, Based on the original image, using a deep learning model to obtain the semantic segmentation result of the original image includes: Adopt a deep learning model to obtain the core, category, and label of semantic objects based on the original image; the core of the semantic object is the central pixel coordinates of the semantic object; Adopt the SAM model to obtain the semantic segmentation result of the original image based on the core, category, label of the object, and the original image.

3. The method for map construction and relocalization in a complex dynamic scenario according to claim 2, wherein The deep learning model is the SparseInst model.

4. The method for map construction and relocalization in a complex dynamic scenario according to claim 1, wherein Based on the original image, determining the corner points includes: Based on the original image, adopt a corner point detection algorithm to obtain initial corner points; Based on the initial corner points, adopt a quadtree algorithm to obtain corner points.

5. The method for map construction and relocalization in a complex dynamic scenario according to claim 4, wherein The corner point detection algorithm is the FAST corner point detection algorithm with an adaptive threshold.

6. The method for map construction and relocalization in a complex dynamic scenario according to claim 5, wherein The expression of the adaptive threshold is: where s is the adaptive threshold, I a is the average grayscale value of the image, and I i is the grayscale value of the i-th pixel point.

7. The method for map construction and relocalization in a complex dynamic scenario according to claim 1, characterized in that Based on the corner points and the original image, extracting descriptors includes: Based on the corner points and the original image, adopt a binary descriptor algorithm to extract descriptors.

8. The method for map construction and relocalization in a complex dynamic scenario according to claim 1, wherein Based on the optimized map, the dictionary, and the database, adopting a bag-of-words model matching method to relocalize the image to be localized includes: Based on the dictionary, determine the bag-of-words vectors of all frames of the image to be localized; Based on the bag-of-words vectors and the database, determine the candidate relocalization key frames; Based on all frames of the image to be localized and the candidate relocalization key frames, adopt a bag-of-words model matching method to determine the relocalization key frames; Based on the relocalization key frames, use a PnP solver to determine the pose of the image to be localized in the optimized map.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method for map construction and relocalization in the complex dynamic scenario according to any one of claims 1-8.

10. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method for map construction and relocalization in the complex dynamic scenario according to any one of claims 1-8.

Images