Map data processing method and device, storage medium and electronic equipment

Abstract

The present disclosure relates to a map data processing method, device, storage medium and electronic equipment. The map data processing method comprises: obtaining target feature map data comprising a current key frame, the target feature map data comprising a non-seeing key frame sequence corresponding to the current key frame and a current frame sequence to which the current key frame belongs; in a case where the current key frame satisfies a loop detection condition, determining a difference degree matrix based on the non-seeing key frame sequence and the current frame sequence, the difference degree matrix being used to represent feature difference degrees between each non-seeing key frame in the non-seeing key frame sequence and each key frame in the current frame sequence; and determining a loop frame corresponding to the current key frame from the non-seeing key frame sequence based on the difference degree matrix, the loop frame being determined based on a target path in the difference degree matrix. The present disclosure can improve the robustness and accuracy of matching a loop frame when a scene changes by constructing a difference degree matrix and adding time and space constraints.

Description

Technical Field

[0001] This disclosure relates to the field of visual positioning technology, and in particular to map data processing methods, apparatus, storage media and electronic devices. Background Technology

[0002] Visual feature mapping is one of the fundamental and core technologies for robotic applications such as autonomous driving. Visual feature mapping typically utilizes loop closure detection to optimize the map and eliminate accumulated errors during the mapping process. However, current loop closure detection technologies suffer from low accuracy and feature discrimination. When environments are extremely similar, false positives may occur, and when features change within the same scene, loop closures may not be detected, thus reducing the accuracy of map construction. Summary of the Invention

[0003] To address at least one of the aforementioned technical problems, this disclosure provides a map data processing method, apparatus, storage medium, and electronic device.

[0004] According to one aspect of this disclosure, a map data processing method is provided, comprising:

[0005] Acquire target feature map data including the current keyframe, wherein the target feature map data includes the non-common keyframe sequence corresponding to the current keyframe and the current frame sequence to which the current keyframe belongs;

[0006] If the current keyframe satisfies the loop closure detection condition, a difference matrix is ​​determined based on the non-common keyframe sequence and the current frame sequence. The difference matrix is ​​used to characterize the feature difference between each non-common keyframe in the non-common keyframe sequence and each keyframe in the current frame sequence.

[0007] Based on the difference matrix, the loopback frame corresponding to the current keyframe is determined from the non-co-view keyframe sequence. The loopback frame is determined based on the target path in the difference matrix. The target path is the shortest path that traverses the matrix elements corresponding to each non-co-view keyframe and has the smallest path feature difference.

[0008] In some possible implementations, determining the loopback frame corresponding to the current keyframe from the non-co-visual keyframe sequence based on the difference matrix includes:

[0009] The target path is obtained by performing shortest path planning to search for the minimum path feature difference in the difference matrix.

[0010] The non-co-visual keyframe corresponding to the current keyframe in the target path is identified as a candidate keyframe;

[0011] The credibility of the candidate keyframes is verified to obtain a first verification result;

[0012] If the first verification result is successful, the candidate keyframe is subjected to matching degree verification to obtain a second verification result.

[0013] If the second verification result is successful, the candidate keyframe is determined to be the loopback frame.

[0014] In some possible implementations, obtaining a first verification result based on the credibility verification of the candidate keyframes includes:

[0015] Determine the target difference degree corresponding to the candidate keyframe and the mean difference degree corresponding to the current keyframe in the difference degree matrix;

[0016] A probability prediction model is generated based on the mean difference.

[0017] Based on the probability prediction model, the target difference degree and the mean difference degree are predicted probabilistically to obtain a first probability and a second probability;

[0018] The confidence level of the candidate keyframe is determined based on the first probability and the second probability, and the confidence level represents the degree of confidence that the candidate keyframe is the loopback frame;

[0019] If the confidence level meets the preset confidence level conditions, the first verification result is determined to be a successful verification.

[0020] In some possible implementations, the matching degree verification of the candidate keyframes to obtain a second verification result includes:

[0021] The local features of the current keyframe and the local features of the candidate keyframe are matched to obtain matching point pairs.

[0022] Determine the feature points in the matching point pair, wherein the attribute of the feature points is either an interior point or an exterior point;

[0023] The matching value of the candidate keyframe is determined based on the attributes of the feature points, and the matching value is used to characterize the degree of matching between the candidate keyframe and the current keyframe.

[0024] If the matching value meets the preset ratio condition, the second verification result is determined to be a successful verification.

[0025] In some possible implementations, acquiring the target feature map data, including the current keyframe, includes:

[0026] Acquire the current image frame and historical feature map data, wherein the historical feature map data includes global pose information corresponding to each key frame;

[0027] If the image sequence position of the current image frame satisfies the image frame distance condition, the current image frame is determined to be the current keyframe;

[0028] The historical feature map data is updated based on the feature data corresponding to the current keyframe to obtain the target feature map data;

[0029] Based on the global pose information, the pose of the current keyframe is estimated to obtain image pose information;

[0030] Obtain the global features and local features corresponding to the current keyframe;

[0031] Based on the current keyframe, the image pose information, the global features, and the local features, the historical feature map data is updated to obtain the target feature map data.

[0032] In some possible implementations, after determining the loopback frame corresponding to the current keyframe from the non-co-visual keyframe sequence based on the difference matrix, the method further includes:

[0033] Obtain semantic map data corresponding to the target feature map data, wherein the semantic map data includes the pose information of key frames;

[0034] Based on the timestamp of the current keyframe and the timestamp of the loopback frame, determine the first local point cloud data corresponding to the current keyframe and the second local point cloud data corresponding to the loopback frame in the semantic map data;

[0035] The first local point cloud data and the second local point cloud data are registered to obtain the trajectory relative pose information.

[0036] Determine the inter-frame relative pose information between every two keyframes in the semantic map data;

[0037] Based on the trajectory relative pose information and the inter-frame relative pose information, the semantic map data is optimized to obtain optimized trajectory data;

[0038] The target feature map data is updated based on the optimized trajectory data to obtain updated target feature map data.

[0039] In some possible implementations, updating the target feature map data based on the optimized trajectory data to obtain updated target feature map data includes:

[0040] Perform time-series matching on the optimized trajectory data and the target feature map data to obtain the starting keyframe corresponding to the first node of the optimized trajectory data in the target feature map data;

[0041] If the timestamp corresponding to the first node is less than the timestamp corresponding to the starting keyframe, determine the timestamp pair corresponding to the timestamp of the starting keyframe in the optimized trajectory data.

[0042] Based on the timestamp pairs in the optimized trajectory data, the target feature map data is interpolated to obtain the updated target feature map data.

[0043] According to a second aspect of this disclosure, a map data processing apparatus is provided, the apparatus comprising:

[0044] The current feature map acquisition module is used to acquire target feature map data including the current keyframe, wherein the target feature map data includes the non-common keyframe sequence corresponding to the current keyframe and the current frame sequence to which the current keyframe belongs;

[0045] The difference matrix determination module is used to determine a difference matrix based on the non-common keyframe sequence and the current frame sequence when the current keyframe satisfies the loop closure detection condition. The difference matrix is ​​used to characterize the feature difference between each non-common keyframe in the non-common keyframe sequence and each keyframe in the current frame sequence.

[0046] The loopback frame determination module is used to determine the loopback frame corresponding to the current keyframe from the non-co-visual keyframe sequence based on the difference matrix. The loopback frame is determined based on the target path in the difference matrix. The target path is the shortest path that traverses the matrix elements corresponding to each non-co-visual keyframe and has the smallest path feature difference.

[0047] According to a third aspect of this disclosure, an electronic device is provided, including at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor implements the map data processing method as described in any one of the first aspects by executing the instructions stored in the memory.

[0048] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided that stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the map data processing method as described in any of the first aspects.

[0049] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure.

[0050] Implementing this disclosure will have the following beneficial effects:

[0051] The system acquires target feature map data, including the current keyframe. This data comprises the sequence of non-common-view keyframes corresponding to the current keyframe and the current frame sequence to which the current keyframe belongs. Loop closure detection is performed using this feature map data. Feature information cached in a real-time topological feature map is used to determine whether the carrier has visited the current location, independent of edge positioning information. If the current keyframe meets the loop closure detection criteria, a difference matrix is ​​determined based on the non-common-view keyframe sequence and the current frame sequence. This difference matrix characterizes the feature differences between each non-common-view keyframe in the non-common-view keyframe sequence and each keyframe in the current frame sequence. Incorporating temporal and spatial information makes the sequence map matching results more accurate in special scenarios with environmental changes or high scene similarity, avoiding false positives. Based on the difference matrix, the loop closure frame corresponding to the current keyframe is determined from the non-common-view keyframe sequence. The loop closure frame is determined based on the target path in the difference matrix, which is the shortest path that traverses the matrix elements corresponding to each non-common-view keyframe and has the smallest path feature difference. This improves the accuracy and robustness of loop closure detection and the accuracy of map construction.

[0052] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0053] To more clearly illustrate the technical solutions and advantages in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 A schematic diagram illustrating an application environment according to an embodiment of the present disclosure is shown;

[0055] Figure 2 A schematic flowchart of a map data processing method according to an embodiment of the present disclosure is shown;

[0056] Figure 3 A flowchart illustrating a method for determining candidate keyframes according to an embodiment of the present disclosure is shown.

[0057] Figure 4 A flowchart illustrating a method for determining a first verification result according to an embodiment of the present disclosure is shown.

[0058] Figure 5 A flowchart illustrating a second verification result determination method according to an embodiment of the present disclosure is shown.

[0059] Figure 6 A flowchart illustrating a method for determining target feature map data according to an embodiment of the present disclosure is shown.

[0060] Figure 7 A flowchart illustrating an optimized trajectory data optimization method according to an embodiment of the present disclosure is shown.

[0061] Figure 8 A flowchart illustrating a target feature map data optimization method according to an embodiment of the present disclosure is shown.

[0062] Figure 9 A flowchart illustrating a method for constructing target feature map data according to an embodiment of the present disclosure is shown.

[0063] Figure 10 A schematic flowchart of a loop closure detection method for target feature map data according to an embodiment of the present disclosure is shown.

[0064] Figure 11 A schematic diagram illustrating the structure of keyframes in target feature map data according to an embodiment of the present disclosure is shown.

[0065] Figure 12 A schematic diagram of the structure of the difference matrix according to an embodiment of the present disclosure is shown;

[0066] Figure 13 A flowchart illustrating a method for optimizing target feature map data based on semantic maps according to an embodiment of the present disclosure is shown.

[0067] Figure 14 A schematic diagram of the structure of a map data processing apparatus according to an embodiment of the present disclosure is shown;

[0068] Figure 15 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0069] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0070] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0071] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0072] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0073] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0074] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0075] Visual feature mapping provides effective prior information for feature matching in robot visual relocalization. Feature map types include topological feature maps and point cloud feature maps. Based on feature type, image features can be divided into two categories: local image features and global image features.

[0076] Local image features are local representations of image features, typically local extrema. Local features exhibit low correlation, meaning that the disappearance of some local features does not affect the detection and matching of others, even under occlusion. Some local features, such as Scale-Invariant Feature Transform (SIFT) and Speeded-Up Robust Features (SURF), offer high detection repetition rates, and their descriptors possess rotation, illumination, and scale invariance. However, current mass-produced autonomous driving systems have limited computational and storage resources, and the aforementioned feature extraction algorithms with rotation, illumination, and scale invariance consume significant amounts of CPU resources. Other well-known local features, such as ORB features extracted in Gaussian space, address the scale invariance issue to some extent, and the BIREF rotation descriptor also addresses the rotation invariance issue to some degree. However, ORB feature maps are not effective for relocalization in scenarios with significant illumination changes, large viewpoint changes, and extremely similar environments, such as underground parking garages.

[0077] In addition, relocalization based on purely visual local features typically requires pre-training a bag-of-words model to aggregate local features into global features. Then, the global features aggregated by the model are used to retrieve relocalization frames on the feature map. Finally, multi-view geometric relationships are calculated based on the 2D feature points of the current frame and the scale-less "3D" point cloud in the retrieved relocalization frames to obtain the robot's current pose. This method requires loading a pre-trained bag-of-words model, consuming significant storage resources, and requires training corresponding bag-of-words models for different scenarios. Multi-view geometric relative pose calculations based on a pure monocular camera lack scale, making it difficult to align the scale of monocular mappings online, let alone perform relocalization. Global image features refer to features across the entire image, including texture, color, and shape features. Global features are low-level visual features at the pixel level and have geometric correlations between pixels, thus exhibiting good environmental invariance. Global features are also intuitive and computationally simple; however, most global features are computationally intensive and have high dimensionality. Compared to local features, they are not suitable for situations involving image occlusion or large changes in viewpoint. Visual feature mapping requires odometry to correlate inter-frame data; however, odometry has accumulated errors, necessitating loop closure detection to constrain and optimize the trajectory and map. Loop closure detection refers to the robot's scene recognition during mapping, determining whether the current location has been visited before and outputting the current pose and the pose of the same location previously. Visual mapping typically relies on depth recovery based on binocular visual odometry, inertial visual odometry, and dead reckoning (inertial navigation + wheel velocity measurement). These methods calculate the relative pose between pairs of related data to obtain the global pose. However, relative pose calculations are not always accurate because observations inevitably contain various noises, such as noise from camera feature point matching, wheel velocity measurement noise, and the zero-bias instability and random walk noise of the Inertial Measurement Unit (IMU). Global pose is usually calculated based on integration; due to the noise in relative pose, the calculated global pose carries accumulated errors. Therefore, a loop closure detection module is needed to optimize the map for loop closure and eliminate these accumulated errors.

[0078] In large-scale outdoor mapping, real-time kinematic (RTK) pose estimation is typically used for positional identification. However, in indoor scenes, such as underground parking garages where RTK signals are unavailable, vision-based scene recognition and loop closure detection are necessary. Well-known real-time mapping and localization systems, such as ORB-SLAM, train a bag-of-words model based on ORB features. While this model accelerates feature retrieval, it requires additional storage and computational resources. Furthermore, bag-of-words-based loop closure detection suffers from low accuracy and feature discrimination, necessitating numerous loop closure candidate frames for geometric filtering using similarity transformations. Accuracy becomes extremely low when scene similarity is high. Moreover, environmental changes alter the features of the same scene, potentially leading to loop closure-free recall.

[0079] Currently, there are two main methods for loop closure detection. One is based on geometric positioning, which involves measuring distances or angles from a reference device at a known location and then calculating the robot's pose using geometric relationships, such as GPS or RTK. This method is primarily used in high-precision map construction. A loop closure is considered found when the robot completes a closed loop and the difference in pose is less than a certain threshold. The other method is based on visual feature matching, which matches the currently observed features with the feature map being constructed to obtain the robot's loop closure pairs. This method is typically used in indoor mapping without high-precision RTK, such as the memory parking function in driver assistance systems.

[0080] Currently, mainstream low-level feature mapping systems, such as ORB_SLAM3 and VINS-MONO, are all SLAM systems based on feature point methods. The FAST features and BRIEF descriptors in low-level feature mapping are limited by the scene and environment. In scenarios with similar features and easily changing environments, such as underground parking garages, the relocalization robustness of these low-level feature maps is poor, and the accuracy and recall are low. Furthermore, loading the bag-of-words model, calculating features, and storing multiple maps require significant resources. Therefore, this disclosure provides a map data processing method to solve the problems of inaccurate loop closure detection, poor robustness, and high resource consumption.

[0081] Please see Figure 1 , Figure 1 This is a schematic diagram of an application environment provided in an embodiment of this application, such as... Figure 1 As shown, the application environment may include at least terminal 01 and server 02. In practical applications, terminal 01 and server 02 can be directly or indirectly connected via wired or wireless communication to achieve interaction between server 02 and terminal 01. This application does not impose any restrictions on this.

[0082] In this embodiment, server 02 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Specifically, server 02 can include physical devices, such as network communication units, processors, and memory, and software running on the physical devices, such as applications. In this embodiment, server 01 can be used to provide network services and data storage services to terminal 01.

[0083] In this embodiment, terminal 01 may include physical devices such as smartphones, desktop computers, tablets, laptops, digital assistants, augmented reality (AR) / virtual reality (VR) devices, smart TVs, smart speakers, smart wearable devices, and in-vehicle terminal devices, and may also include software running on the physical device, such as applications. Specifically, terminal 01 can use the feature difference matrix to perform loop closure detection and optimize the map based on the loop closure pairs during the map construction process.

[0084] Figure 2 This diagram illustrates a flowchart of a map data processing method according to an embodiment of the present disclosure, as shown below. Figure 2 As shown, the above method includes:

[0085] S101. Obtain target feature map data including the current keyframe. The target feature map data includes the non-common keyframe sequence corresponding to the current keyframe and the current frame sequence to which the current keyframe belongs.

[0086] Each node in the target feature map data represents a keyframe. A keyframe includes crucial information such as global image features, local features, timestamps, and the current image pose. The target feature map data is continuously updated as keyframes are added during the mapping process. The current keyframe represents the newly added keyframe in the target feature map data. Based on the timestamp of the current keyframe, the system queries the corresponding non-common-view keyframe sequence, common-view keyframe sequence, and current keyframe sequence in the target feature map data. The current keyframe sequence represents a first preset number of consecutive timestamps preceding the current keyframe. The common-view keyframe sequence represents a second preset number of consecutive timestamps preceding the current keyframe, where the second preset number is greater than the first preset number. Common-view keyframes include the current keyframe sequence. The non-common-view keyframe sequence represents a third preset number of consecutive timestamps preceding the common-view keyframe sequence. The timestamps of non-common-view keyframes are earlier than those of common-view keyframes.

[0087] In some embodiments, the target feature map data includes 300 keyframes. These 300 keyframes are divided into a shared keyframe sequence and a non-shared keyframe sequence. The shared keyframe sequence includes the current keyframe sequence. The current keyframe sequence includes the current keyframe and the 19 keyframes preceding it, i.e., the current keyframe sequence has 20 keyframes, and includes the local topology map of these 20 keyframes. The shared keyframe sequence includes the current keyframe and the 44 consecutive keyframes preceding it, i.e., the shared keyframe sequence includes 45 consecutive keyframes. The non-shared keyframe sequence includes the 255 consecutive keyframes following the shared keyframe sequence.

[0088] In some embodiments, the keyframe representation of the target feature map data CurrentMapBuffer up to the current keyframe is as follows:

[0089]

[0090] Where n is the total number of keyframes in the target feature map data. The last keyframe of the target feature map data is the current keyframe.

[0091] The keyframe representation in the NonCovisMapBuffer, which excludes shared keyframes (i.e., non-shared keyframe sequences), is as follows:

[0092]

[0093] Where id∈[1,n-α], n∈N + α represents the number of common keyframes in the target feature map data.

[0094] The keyframes in the current frame sequence CurSequence are represented as follows:

[0095]

[0096] Where β is the number of current keyframe sequences in the target feature map data.

[0097] S102. When the current keyframe meets the loop closure detection condition, a difference matrix is ​​determined based on the non-common keyframe sequence and the current frame sequence. The difference matrix is ​​used to characterize the feature difference between each non-common keyframe in the non-common keyframe sequence and each keyframe in the current frame sequence.

[0098] Each element in the difference matrix represents the pairwise feature difference between non-co-perceived keyframes in the non-co-perceived keyframe sequence and keyframes in the current frame sequence. The system determines whether a keyframe in the target feature map data has detected a loop closure. If no loop closure has been detected, and the cumulative distance between the current keyframe and the starting keyframe in the target feature map data is greater than or equal to a first preset distance threshold, or the cache size of keyframes in the target feature map data is greater than or equal to a first preset cache threshold, and the ID of the current keyframe satisfies a first preset identification code condition, then the current keyframe satisfies the loop closure detection condition. If a loop closure has been detected in the target feature map data, and the cumulative distance between the current keyframe and the loop closure position of the previous frame is greater than or equal to a first preset distance threshold, or the difference between the ID of the current keyframe and the ID of the loop closure position of the previous frame is greater than or equal to a first preset identification code threshold, and the ID of the current keyframe satisfies a first preset identification code condition, then the current keyframe satisfies the loop closure detection condition. A difference matrix is ​​constructed based on the non-common keyframe sequence and the current keyframe sequence. When constructing the difference matrix, it is not necessary to cache the non-common keyframe sequence and the current keyframe sequence separately. The difference is calculated based on the index of the keyframe in the non-common keyframe sequence and the index of the keyframe in the current keyframe sequence to obtain the difference matrix.

[0099] Theoretically, the longer the current frame sequence, the better the loop closure detection performance. However, the computational cost of constructing the difference matrix increases significantly, resulting in a time complexity of log n. 2If the current keyframe sequence is small, meaning the number of keyframes is limited, then the temporal and spatial constraints are insufficient, making misidentification easy in scenarios with high similarity and changing environments. In summary, the number of rows in the dissimilarity matrix can be the size of the target feature map data minus the number of co-visible keyframes, and the number of columns can be the size of the current keyframe sequence. Therefore, the size of the dissimilarity matrix is ​​(n-α)×β. The dissimilarity matrix for loop closure detection needs to be reconstructed each time because the map cache grows in real time as the mapping process progresses.

[0100] For example, the identification code ID of the current keyframe can satisfy the first preset identification code condition by being an odd number.

[0101] In some embodiments, the difference between the 20 keyframes of the current keyframe sequence and the 255 keyframes of the non-co-view keyframe sequence is calculated using the keyframe index, resulting in a 20*255 difference matrix. The feature difference in the difference matrix ranges from (0, 1). The difference calculation method includes, but is not limited to, cosine value calculation and counting the number of non-zero differences.

[0102] S103. Based on the difference matrix, determine the loopback frame corresponding to the current keyframe from the non-common keyframe sequence. The loopback frame is determined based on the target path in the difference matrix. The target path is the shortest path with the smallest path feature difference that traverses the matrix elements corresponding to each non-common keyframe.

[0103] The target path is obtained by searching the matrix elements corresponding to each non-common keyframe in the dissimilarity matrix and finding the shortest path with the minimum dissimilarity. The loopback frame corresponding to the current keyframe is then determined based on the target path.

[0104] In some embodiments, a dynamic programming algorithm is used to search for the shortest path with the minimum difference in the difference matrix, wherein the matrix elements in the shortest path are not in the same row or column.

[0105] The above technical solution, by constructing a difference matrix, incorporates temporal and spatial constraints, thereby improving the robustness and accuracy of loopback frame matching during scene changes.

[0106] Please see Figure 3 In some embodiments, based on the difference matrix, the loopback frame corresponding to the current keyframe is determined from the non-common keyframe sequence, including:

[0107] S1031. Perform shortest path planning by searching for the minimum path feature difference of the difference matrix to obtain the target path;

[0108] S1032. Determine the non-common keyframes corresponding to the current keyframes in the target path as candidate keyframes;

[0109] S1033. Verify the credibility of the candidate keyframes to obtain the first verification result;

[0110] S1034. If the first verification result is successful, the matching degree of the candidate keyframe is verified to obtain the second verification result.

[0111] S1035. If the second verification result is successful, the candidate keyframe is determined to be a loopback frame.

[0112] The first verification result is used to characterize the credibility of the candidate keyframe being a loopback frame, which meets the preset credibility condition. The second verification result is used to characterize the matching degree between the current keyframe and the candidate keyframe, which meets the preset ratio condition.

[0113] In some embodiments, the shortest path with the smallest sum of feature differences in the difference matrix is ​​determined as the target path, and the non-co-visual keyframes corresponding to the current keyframe in the target path are taken as candidate keyframes. If the candidate keyframes pass the credibility verification and matching degree verification, the candidate keyframes are determined as the loopback frames corresponding to the current keyframes.

[0114] The above technical solution performs two checks on the candidate keyframes, further ensuring the accuracy of loop closure detection.

[0115] Please see Figure 4 In some embodiments, a first verification result is obtained based on the credibility verification of candidate keyframes, including:

[0116] S10331. Determine the target difference degree corresponding to the candidate keyframe and the average difference degree corresponding to the current keyframe in the difference degree matrix;

[0117] S10332, Probability prediction model based on mean difference;

[0118] S10333. Based on the probability prediction model, the target difference degree and the mean difference degree are predicted probabilistically to obtain the first probability and the second probability.

[0119] S10334. Determine the confidence level of the candidate keyframe based on the first probability and the second probability. The confidence level represents the degree of confidence that the candidate keyframe is a loopback frame.

[0120] S10335. If the confidence level meets the preset confidence level conditions, the first verification result is determined to be a successful verification.

[0121] In some embodiments, the difference between the current keyframe and all non-co-view keyframes in the difference matrix is ​​calculated, i.e., the average difference vector corresponding to the current keyframe, to obtain the mean difference value. A probability prediction model is generated based on a preset distribution algorithm and the mean difference value. Based on the probability prediction model, the target difference value and the mean difference value corresponding to the candidate keyframe are predicted probabilistically to obtain a first probability and a second probability. The first probability and the second probability are compared to obtain the confidence level of the candidate keyframe. When the confidence level meets the preset confidence level condition, the first verification result is determined to be verified as passed.

[0122] In some embodiments, the preset distribution algorithm can be a Laplace distribution algorithm, using the mean difference as the scaling factor and the position coefficient as a preset value (exemplarily, the position coefficient is 0) to obtain a probabilistic prediction model. The target difference and mean difference corresponding to the candidate keyframe are substituted into the probabilistic prediction model to obtain a first probability p1 and a second probability p2. The first probability p1 is divided by the second probability p2 to obtain the confidence level corresponding to the candidate keyframe. If the confidence level is greater than the confidence level threshold, it is determined that the confidence level corresponding to the candidate keyframe meets the preset confidence condition, and the first verification result is considered to be verified successfully. The confidence level threshold needs to be set according to different features. For example, the range of the confidence level threshold for small image features can be [1.3, 1.5].

[0123] The above technical solution performs confidence verification on candidate keyframes based on a probabilistic prediction model before determining that the candidate keyframe is the loop closure frame corresponding to the current keyframe. This improves the accuracy of loop closure detection and avoids errors in loop closure frame detection that could lead to errors in the optimized map.

[0124] Please see Figure 5 In some embodiments, the candidate keyframes are verified for matching degree to obtain a second verification result, including:

[0125] S10341. Perform feature matching between the local features of the current keyframe and the local features of the candidate keyframes to obtain matching point pairs.

[0126] S10342. Determine the feature points in the matching point pair, wherein the attribute of the feature points is either interior point or exterior point;

[0127] S10343. Determine the matching value of the candidate keyframe based on the attributes of the feature points. The matching value is used to characterize the degree of matching between the candidate keyframe and the current keyframe.

[0128] S10344. If the matching value meets the preset ratio condition, the second verification result is determined to be verified as passed.

[0129] In some embodiments, if the first verification result is successful, geometric verification based on feature points is performed on the candidate keyframes. The target feature map data includes local features of the keyframes; local features of the current keyframe and candidate keyframes are obtained. Feature matching is performed between the local features of the current keyframe and candidate keyframes to obtain matching point pairs. A preset sampling algorithm is used to process the matching point pairs to obtain feature points within the matching point pairs, including inliers and outliers. Inliers represent points where the local features of the current keyframe and candidate keyframes in the matching point pair successfully match, while outliers represent points where the local features of the current keyframe and candidate keyframes fail to match. The matching value of the candidate keyframe is determined based on the number of inliers and outliers. If the matching value is greater than a matching degree threshold, it is determined that the matching value meets a preset ratio condition, and the second verification result is successful.

[0130] In some embodiments, the local features of the current keyframe and the local features of the candidate keyframe are brute-force matched to obtain at least 8 pairs of matching points. Random Sample Consensus (RANSAC) is used to calculate the fundamental matrix of the matching points to obtain interior points and exterior points. The number of interior points and the number of exterior points are counted. The ratio of the number of interior points to the number of exterior points is used as the matching value. When the matching value is greater than the matching degree threshold of 80%, the second verification result is determined to be verified as passed.

[0131] The above technical solution performs geometric verification on the local features of the current keyframe and the local features of the candidate keyframes to filter out false positives again, thereby improving the accuracy and robustness of loop closure detection.

[0132] Please see Figure 6 In some embodiments, obtaining target feature map data, including the current keyframe, includes:

[0133] S1011. Obtain the current image frame and historical feature map data. The historical feature map data includes global pose information corresponding to each key frame.

[0134] S1012. If the image sequence position of the current image frame satisfies the image frame distance condition, determine the current image frame as the current keyframe;

[0135] S1013. Update the historical feature map data based on the feature data corresponding to the current keyframe to obtain the target feature map data;

[0136] S1014. Based on global pose information, estimate the pose of the current keyframe to obtain image pose information;

[0137] S1015. Obtain the global features and local features corresponding to the current keyframe;

[0138] S1016. Update the historical feature map data based on the current keyframe, image pose information, global features, and local features to obtain the target feature map data.

[0139] In some embodiments, historical feature map data is used to represent a historically constructed topological feature map. Nodes in the map are represented by a keyframe in the historical feature map data, and the current image frame is acquired by an image acquisition device. If the offset distance between the current image frame's image sequence position and the previous image frame is greater than a preset offset distance threshold, the current image frame is determined to be the current keyframe. Otherwise, feature extraction is not performed to improve computing power. Global pose estimation is performed on the current image frame based on a preset measurement tool to obtain global pose information. The global pose information is stored in a pose cache. Linear interpolation is performed based on the timestamp of the current keyframe and the pose cache to obtain the current frame image pose. The grayscale image corresponding to the current image frame is reduced to a preset scale using a preset transformation method, and the reduced image is subjected to illumination normalization to obtain the global features corresponding to the current keyframe. A preset number of feature points are extracted from the reduced image, and the preset number of feature points are dimensionality-reduced using a trained matrix to retain effective information, obtaining the local features corresponding to the current keyframe. The historical feature map data is updated based on the current keyframe, image pose information, global features, and local features to obtain the target feature map data.

[0140] In some embodiments, when the current image frame F t Offset t c and the previous image frame KF t-1 When the offset distance is greater than a preset offset distance threshold, the current image frame is inserted as the current feature frame into the historical feature map data [KF1,KF2,…,KF]. t-1 KF t In this study, a networked inertial measurement unit (IMU) and wheel velocity sensor are used to calculate the dead reckoning (DR) of the current image frame to perform global pose estimation and obtain global pose information. This global pose information is stored in the DR pose buffer and then used with the timestamp t of the current keyframe. c Retrieve the left pose T from the DR pose buffer, based on the timestamp closest to the current keyframe. DR_L With right pose T DR_R And perform linear interpolation. Offset t C The difference is calculated using the following formula:

[0141] t C =t C +scale(t DR_R -t DR_L )

[0142]

[0143] Quaternion q c The linear interpolation representing rotation is expressed by the following formula:

[0144] q c =scale0*q DR_L +scale1*q DR_R

[0145] Among them, t DR_L Left pose T DR_L The corresponding offset, t DR_R Right pose T DR_R The corresponding offset, scale is the time ratio, t S_C For offset t C The corresponding timestamp, t s_L Left position T DR_L The corresponding timestamp, t S_R Right pose T DR_R The corresponding timestamps are 1-scale for scale0, 1-scale0 for scale1, and q. DR_L For timestamp t S_L The corresponding quaternion, q DR_R For timestamp t S_R The corresponding quaternion.

[0146] The grayscale image corresponding to the current image frame is reduced in size using the Lanczos interpolation method at a preset scale. For example, the grayscale image is reduced to a small image of 64x32 pixels. The reduced image is then divided into n*n small grids, and each small grid undergoes illumination normalization (patch normalization) to obtain the global features corresponding to the current feature frame. The formula for patch normalization of the small grids is as follows:

[0147]

[0148] in, These are the patch-normalized values ​​of each pixel in the small grid. I represents the illumination intensity value in the small grid. σ is the average illumination intensity of all pixels in a small grid. k This represents the sample bias of the illumination intensity for all pixels in a small grid.

[0149] The reduced image is processed by extracting N feature points, for example, N is 300. Then, a trained matrix is ​​used to perform a dimensionality reduction mapping on the BRIEF descriptors of the 300 feature points, retaining the effective information, and obtaining the local feature information corresponding to the current keyframe.

[0150] The aforementioned technical features, when constructing target feature map data, only include keyframes that meet the image frame distance conditions, saving computing power and resulting in a feature map that occupies little memory resources, thus solving the problem of extremely high computational resource consumption for loop closure detection in the global scope of semantic maps.

[0151] Please see Figure 7 In some embodiments, after determining the loopback frame corresponding to the current keyframe from the non-common keyframe sequence based on the difference matrix, the method further includes:

[0152] S201. Obtain semantic map data corresponding to the target feature map data. The semantic map data includes the pose information of key frames.

[0153] S202. Based on the timestamp of the current keyframe and the timestamp of the loopback frame, determine the first local point cloud data corresponding to the current keyframe and the second local point cloud data corresponding to the loopback frame in the semantic map data.

[0154] S203. Register the first local point cloud data and the second local point cloud data to obtain the trajectory relative pose information;

[0155] S204. Determine the inter-frame relative pose information between every two keyframes in the semantic map data;

[0156] S205. Based on the trajectory relative pose information and the inter-frame relative pose information, the semantic map data is optimized to obtain optimized trajectory data;

[0157] S206. Update the target feature map data based on the optimized trajectory data to obtain updated target feature map data.

[0158] In some embodiments, semantic map data corresponding to target feature map data is obtained, and loop closure optimization is performed on the semantic map data based on the information of the current key frame and the loop closure frame corresponding to the current key frame. Specifically, the first local point cloud data corresponding to the current key frame and the second local point cloud data corresponding to the loop closure frame are determined according to the timestamp. The first local point cloud data and the second local point cloud data are registered using a preset point cloud matching algorithm to obtain the trajectory relative pose information. The inter-frame relative pose information of each pair of key frames in the semantic map is determined to obtain the optimized trajectory data.

[0159] In some embodiments, the first local point cloud data and the second local point cloud data of the target feature map data are subjected to a normal distribution transformation (NDT) to obtain the trajectory relative pose information of the trajectory drift at the loop point, i.e., the loop constraint.

[0160] The above technical solution optimizes the trajectory of the target semantic map by first determining the loop closure pairs through loop closure detection, thereby obtaining optimized trajectory data and improving the accuracy and robustness of the mapping.

[0161] Please see Figure 8 In some embodiments, the target feature map data is updated based on the optimized trajectory data to obtain updated target feature map data, including:

[0162] S2061. Perform time-series matching on the optimized trajectory data and the target feature map data to obtain the starting keyframe corresponding to the first node of the optimized trajectory data in the target feature map data.

[0163] S2062. If the timestamp corresponding to the first node is less than the timestamp corresponding to the starting keyframe, determine the timestamp pair corresponding to the timestamp of the starting keyframe in the optimized trajectory data.

[0164] S2063. Based on the timestamps in the optimized trajectory data, interpolate the target feature map data to obtain the updated target feature map data.

[0165] In some embodiments, during the construction of the target feature map data, after determining the loopback frame corresponding to the current keyframe from the non-common keyframe sequence based on the difference matrix, the current keyframe and its corresponding loopback frame are stored in a loopback pair cache. Once the target feature map data is constructed and the loopback pair cache is not empty, the starting keyframe corresponding to the first node of the optimized trajectory data in the target feature map data is determined. If the timestamp corresponding to the first node of the optimized trajectory data is greater than or equal to the timestamp of the starting keyframe in the target feature map data, the pose of the starting keyframe in the target feature map data remains unchanged and is not updated. If the timestamp corresponding to the first node of the optimized trajectory data is less than the timestamp of the starting keyframe in the target feature map data, a neighboring timestamp pair is searched in the optimized trajectory data based on the timestamp of the starting keyframe. The timestamp pair represents the timestamps of two adjacent nodes before, after, or to the left and right of the first node. Linear interpolation is performed on the target feature map data based on the trajectory data corresponding to the timestamp pairs in the optimized trajectory data to obtain the updated target feature map data.

[0166] In some embodiments, once the target feature map data is constructed and the loopback cache is not empty, the target feature map data is optimized by traversing and optimizing the trajectory data. The first keyframe of the target map feature data, i.e., the starting keyframe, is determined, with the corresponding identification code 'i'. Optimized trajectory data is obtained. If the timestamp of the first node in the optimized trajectory data is not earlier than the timestamp of the i-th keyframe, the pose of the i-th keyframe remains unchanged and no optimization is performed. The identification code of the keyframe is incremented, and the next keyframe of the target map feature data is determined. If the timestamp of the first node in the optimized trajectory data is earlier than the timestamp of the i-th keyframe, then neighboring timestamp pairs are searched in the optimized trajectory data based on the timestamp of the i-th keyframe. Based on the optimized pose of the target feature map data corresponding to the trajectory data of the timestamp pairs in the optimized trajectory data for the i-th keyframe, nodes with timestamps earlier than the i-th keyframe are deleted from the optimized trajectory data. If the optimized trajectory data is not empty, 'i' is incremented, and the pose of the next keyframe of the target feature map data is optimized. If the optimized trajectory data is empty, then all map frames starting from the i-th keyframe of the target feature map data are deleted to obtain the updated target feature map data.

[0167] The above technical solution optimizes the target feature map data by optimizing the trajectory data of the semantic map data, so that the feature map and the semantic map are consistent, maintaining the accuracy and robustness of the mapping.

[0168] In some embodiments, the loop closure optimization process mainly involves the construction of target feature map data, such as... Figure 9 Perform loop closure detection on the current keyframe constructed from the target feature map, such as... Figure 10 After the target feature map data is constructed, the semantic map corresponding to the target feature map is optimized based on the detected loop closures. The target feature map is then further optimized based on the trajectory data optimized from the semantic map. Figure 13 For details, please refer to Figure 9Global pose estimation is performed using IMU and wheel velocity sensor for dead reckoning (DR), obtaining global pose information which is stored in the DR pose cache. The current image frame is acquired using a fisheye camera, and linear interpolation is performed based on the timestamp of the current image frame and the DR pose cache to obtain the image pose information corresponding to the current keyframe. If the offset of the current image frame from the previous keyframe is greater than a preset offset distance threshold, the current image frame is identified as the current keyframe. The grayscale image corresponding to the current keyframe is acquired, and a preset number of ORB features are extracted from the grayscale image to obtain the local features corresponding to the current keyframe. The grayscale image is then downsized using LANCZOS interpolation to obtain a smaller image of a preset size. This smaller image is then histogram-averaged, divided into n*n grids, and patch normalization is performed on each grid to obtain the global features corresponding to the current keyframe. The target feature map data is updated based on the current keyframe, its local features, global features, and image pose information.

[0169] Please see Figure 10 Load the currently updated target feature map data, determine whether the target feature map data has detected loop closures. If no loop closures have been detected, determine whether the current keyframe is greater than the position of the starting keyframe than the first preset distance threshold, and whether the identification code ID of the current keyframe is odd. If not, loop closure detection is not performed; if so, loop closure detection is performed. Figure 11 The target feature map data is structured as follows: it includes non-common-view keyframe sequences and common-view keyframe sequences, with the common-view keyframe sequences further including the current keyframe sequence. A difference matrix is ​​obtained by calculating the difference between the non-common-view keyframe sequences and the current keyframe sequence. Figure 12 A dynamic programming algorithm is used to search the difference matrix to determine candidate keyframes. The confidence level of each candidate keyframe is determined based on a probabilistic prediction model. If the confidence level is not greater than a confidence threshold, the current keyframe is considered to have no loopback results. If the confidence level is greater than the confidence threshold, a brute-force matching process is performed based on the local features of the current keyframe and the local features of the loopback frames to obtain matching point pairs. RANSAC is used to perform a fundamental matrix on the matching point pairs to obtain inliers and outliers. If the ratio of the number of inliers to the number of outliers is greater than a matching threshold, the candidate keyframe is determined to be the loopback frame corresponding to the current keyframe. The current keyframe and its corresponding loopback frame are stored in a loopback pair cache. If the ratio of the number of inliers to the number of outliers is not greater than a matching threshold, the current keyframe is considered to have no loopback results.

[0170] Please see Figure 13Once the target feature map data is constructed and the loop closure cache is not empty, trajectory optimization is performed on the semantic map data corresponding to the target feature map data based on the loop closure cache to obtain optimized trajectory data. The first keyframe of the target map feature data, i.e., the starting keyframe, is determined, with the corresponding identification code 'i'. Optimized trajectory data is obtained. If the timestamp of the first node in the optimized trajectory data is not earlier than the timestamp of the i-th keyframe, the pose of the i-th keyframe remains unchanged and no optimization is performed. The identification code of the keyframe is incremented, and the next keyframe of the target map feature data is determined. If the timestamp of the first node in the optimized trajectory data is earlier than the timestamp of the i-th keyframe, neighboring timestamp pairs are searched in the optimized trajectory data based on the timestamp of the i-th keyframe. Based on the optimized pose of the i-th keyframe of the target feature map data corresponding to the trajectory data of the timestamp pairs in the optimized trajectory data, nodes with timestamps earlier than the i-th keyframe are deleted from the optimized trajectory data. If the optimized trajectory data is not empty, 'i' is incremented, and the pose of the next keyframe of the target feature map data is optimized. If the optimized trajectory data is empty, then all map frames starting from the i-th keyframe of the target feature map data are deleted to obtain the updated target feature map data.

[0171] Please see Figure 14 According to a second aspect of this disclosure, a map data processing apparatus is provided, the apparatus comprising:

[0172] The current feature map acquisition module 10 is used to acquire target feature map data including the current key frame. The target feature map data includes the non-common key frame sequence corresponding to the current key frame and the current frame sequence to which the current key frame belongs.

[0173] The difference matrix determination module 20 is used to determine the difference matrix based on the non-common key frame sequence and the current frame sequence when the current key frame meets the loop closure detection condition. The difference matrix is ​​used to characterize the feature difference between each non-common key frame in the non-common key frame sequence and each key frame in the current frame sequence.

[0174] The loopback frame determination module 30 is used to determine the loopback frame corresponding to the current keyframe from the non-common keyframe sequence based on the difference matrix. The loopback frame is determined based on the target path in the difference matrix. The target path is the shortest path with the smallest path feature difference that traverses the matrix elements corresponding to each non-common keyframe.

[0175] In some embodiments, the loopback frame determination module 30 includes:

[0176] The target path determination unit 31 is used to perform shortest path planning by searching for the minimum value of the path feature difference in the difference matrix to obtain the target path.

[0177] Candidate keyframe determination unit 32 is used to determine the non-common keyframe corresponding to the current keyframe in the target path as a candidate keyframe;

[0178] The first verification unit 33 is used to verify the credibility of candidate keyframes and obtain the first verification result.

[0179] The second verification unit 34 is used to perform matching degree verification on the candidate keyframes when the first verification result is that the verification is passed, and to obtain the second verification result.

[0180] The loopback frame determination unit 35 is used to determine the candidate keyframe as the loopback frame if the second verification result is a successful verification.

[0181] In some embodiments, the first verification unit 33 includes:

[0182] The difference processing unit 331 is used to determine the target difference degree corresponding to the candidate keyframe and the average difference degree corresponding to the current keyframe in the difference degree matrix.

[0183] Probability prediction model generation unit 332 is used to generate a probability prediction model based on the mean difference.

[0184] The probability prediction unit 333 is used to make probability predictions on the target difference degree and the mean difference value based on the probability prediction model, so as to obtain the first probability and the second probability.

[0185] The confidence determination unit 334 is used to determine the confidence of the candidate keyframe based on the first probability and the second probability. The confidence level represents the degree of confidence that the candidate keyframe is a loopback frame.

[0186] The first verification result determination unit 335 is used to determine the first verification result as verification passed if the confidence level meets the preset confidence level conditions.

[0187] In some embodiments, the second verification unit 34 includes:

[0188] The feature matching unit 341 is used to perform feature matching between the local features of the current keyframe and the local features of the candidate keyframe to obtain matching point pairs;

[0189] The feature point determination unit 342 is used to determine the feature points in the matching point pair, wherein the attribute of the feature point is an interior point or an exterior point;

[0190] The matching value determination unit 343 is used to determine the matching value of the candidate keyframe based on the attributes of the feature points. The matching value is used to characterize the degree of matching between the candidate keyframe and the current keyframe.

[0191] The second verification result determination unit 344 is used to determine the second verification result as verification passed when the matching value meets the preset ratio condition.

[0192] In some embodiments, the current feature map acquisition module 10 includes:

[0193] The map data acquisition unit 11 is used to acquire the current image frame and historical feature map data. The historical feature map data includes global pose information corresponding to each key frame.

[0194] The current keyframe determination unit 12 is used to determine the current image frame as the current keyframe when the image sequence position of the current image frame meets the image frame distance condition;

[0195] The historical feature map data update unit 13 is used to update the historical feature map data based on the feature data corresponding to the current keyframe to obtain the target feature map data.

[0196] The image pose information is determined, but it is used to estimate the pose of the current keyframe based on the global pose information, thus obtaining the image pose information.

[0197] Feature acquisition unit 15 is used to acquire the global features and local features corresponding to the current key frame;

[0198] The target feature map data confirmation unit 16 is used to update the historical feature map data based on the current keyframe, image pose information, global features and local features to obtain the target feature map data.

[0199] In some embodiments, the apparatus further includes:

[0200] The semantic map data acquisition module 40 is used to acquire semantic map data corresponding to the target feature map data. The semantic map data includes the pose information of key frames.

[0201] The point cloud data determination module 50 is used to determine the first local point cloud data corresponding to the current key frame and the second local point cloud data corresponding to the loopback frame in the semantic map data based on the timestamp of the current key frame and the timestamp of the loopback frame.

[0202] The trajectory relative pose information determination module 60 is used to perform registration processing on the first local point cloud data and the second local point cloud data to obtain trajectory relative pose information.

[0203] The inter-frame relative pose information determination module 70 is used to determine the inter-frame relative pose information between every two keyframes in the semantic map data;

[0204] The trajectory data determination module 80 is used to optimize the trajectory of semantic map data based on trajectory relative pose information and inter-frame relative pose information to obtain optimized trajectory data.

[0205] The target feature map data update module 90 is used to update the target feature map data based on the optimized trajectory data to obtain updated target feature map data.

[0206] In some embodiments, the target feature map data update module 90 includes:

[0207] The temporal matching unit 91 is used to perform temporal matching between the optimized trajectory data and the target feature map data to obtain the starting keyframe corresponding to the first node of the optimized trajectory data in the target feature map data.

[0208] The timestamp pair determination unit 92 is used to determine the timestamp pair corresponding to the timestamp of the starting key frame in the optimized trajectory data if the timestamp corresponding to the first node is less than the timestamp corresponding to the starting key frame.

[0209] The interpolation processing unit 93 is used to interpolate the target feature map data based on the timestamps in the optimized trajectory data to obtain the updated target feature map data.

[0210] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0211] This application provides a map data processing device, which can be a terminal or a server. The map data processing device includes a processor and a memory. The memory stores at least one instruction or at least one program. The at least one instruction or at least one program is loaded and executed by the processor to implement the map data processing method provided in the above method embodiments.

[0212] Memory is used to store software programs and modules. The processor executes these stored software programs and modules to perform various functional applications and data processing. Memory can primarily consist of a program storage area and a data storage area. The program storage area stores the operating system, application programs required for functionality, etc.; the data storage area stores data created based on device usage, etc. Furthermore, memory can include high-speed random access memory (RAM) and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, memory can also include a memory controller to provide the processor with access to the memory.

[0213] The methods and embodiments provided in this application can be executed in electronic devices such as mobile terminals, computer terminals, servers, or similar computing devices. Figure 15 This is a hardware structure block diagram of an electronic device for a map data processing method provided in an embodiment of this application. For example... Figure 15 As shown, the electronic device 900 can vary significantly due to differences in configuration or performance. It may include one or more central processing units (CPUs) 910 (CPUs 910 may include, but are not limited to, microprocessors such as MCUs or programmable logic devices such as FPGAs), a memory 930 for storing data, and one or more storage media 920 (e.g., one or more mass storage devices) for storing application programs 923 or data 922. The memory 930 and storage media 920 may be temporary or persistent storage. The program stored in the storage media 920 may include one or more modules, each module may include a series of instruction operations on the electronic device. Furthermore, the CPU 910 may be configured to communicate with the storage media 920 and execute the series of instruction operations in the storage media 920 on the electronic device 900. Electronic device 900 may also include one or more power supplies 960, one or more wired or wireless network interfaces 950, one or more input / output interfaces 940, and / or one or more operating systems 921, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0214] The input / output interface 940 can be used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the electronic device 900. In one example, the input / output interface 940 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the input / output interface 940 may be a radio frequency (RF) module for wireless communication with the Internet.

[0215] Those skilled in the art will understand that Figure 15 The structure shown is for illustrative purposes only and does not limit the structure of the electronic device described above. For example, the electronic device 900 may also include... Figure 15 The more or fewer components shown, or having the same Figure 15 The different configurations shown.

[0216] Embodiments of this application also provide a computer-readable storage medium, which can be disposed in an electronic device to store at least one instruction or at least one program related to implementing a map data processing method in the method embodiment. The at least one instruction or the at least one program is loaded and executed by the processor to implement the map data processing method provided in the above method embodiment.

[0217] Optionally, in this embodiment, the storage medium may be located at at least one of the multiple network servers in a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0218] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various alternative implementations described above.

[0219] As can be seen from the embodiments of the map data processing method, apparatus, device, terminal, server, storage medium, or computer program provided in this application, this application acquires target feature map data including the current keyframe. The target feature map data includes the non-common-view keyframe sequence corresponding to the current keyframe and the current frame sequence to which the current keyframe belongs. Loop closure detection is performed using the feature map data, and the feature information cached by the real-time established topological feature map is used to determine whether the carrier has been to the current location, without relying on the outer edge positioning information. When the current keyframe meets the loop closure detection conditions, a difference matrix is ​​determined based on the non-common-view keyframe sequence and the current frame sequence. The difference matrix is ​​used to characterize the feature difference between each non-common-view keyframe in the non-common-view keyframe sequence and each keyframe in the current frame sequence. The addition of temporal and spatial information makes the sequence map matching results more accurate in special scenarios with environmental changes or high scene similarity, avoiding false positive detection. Based on the dissimilarity matrix, the loop closure frame corresponding to the current keyframe is determined from the sequence of non-common keyframes. The loop closure frame is determined based on the target path in the dissimilarity matrix, which is the shortest path that minimizes the path feature dissimilarity among the matrix elements corresponding to each non-common keyframe. This improves the accuracy and robustness of loop closure detection and enhances the accuracy of map construction.

[0220] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired results. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing are also possible or may be advantageous.

[0221] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device, equipment, and storage medium embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0222] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware, or by a program instructing the relevant hardware to implement them. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0223] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A map data processing method, characterized in that, The method includes: Acquire target feature map data including the current keyframe, wherein the target feature map data includes the non-common keyframe sequence corresponding to the current keyframe and the current frame sequence to which the current keyframe belongs; If the current keyframe satisfies the loop closure detection condition, a difference matrix is ​​determined based on the non-common keyframe sequence and the current frame sequence. The difference matrix is ​​used to characterize the feature difference between each non-common keyframe in the non-common keyframe sequence and each keyframe in the current frame sequence. Based on the difference matrix, the loopback frame corresponding to the current keyframe is determined from the non-co-visual keyframe sequence. The loopback frame is determined based on the target path in the difference matrix. The target path is the shortest path that traverses the matrix elements corresponding to each non-co-visual keyframe and has the smallest path feature difference. The target path is obtained by performing shortest path planning to search for the minimum path feature difference in the difference matrix. The non-co-visual keyframe corresponding to the current keyframe in the target path is identified as a candidate keyframe; The credibility of the candidate keyframes is verified to obtain a first verification result; If the first verification result is successful, the candidate keyframe is subjected to matching degree verification to obtain a second verification result. If the second verification result is successful, the candidate keyframe is determined to be the loopback frame.

2. The method according to claim 1, characterized in that, The first verification result obtained based on the credibility verification of the candidate keyframes includes: Determine the target difference degree corresponding to the candidate keyframe and the mean difference degree corresponding to the current keyframe in the difference degree matrix; A probability prediction model is generated based on the mean difference. Based on the probability prediction model, the target difference degree and the mean difference degree are probabilistically predicted to obtain a first probability and a second probability; The confidence level of the candidate keyframe is determined based on the first probability and the second probability, and the confidence level represents the degree to which the candidate keyframe is the loopback frame; If the confidence level meets the preset confidence level conditions, the first verification result is determined to be a successful verification.

3. The method according to claim 1, characterized in that, The matching degree verification of the candidate keyframes to obtain a second verification result includes: The local features of the current keyframe and the local features of the candidate keyframe are matched to obtain matching point pairs. Determine the feature points in the matching point pair, wherein the attribute of the feature points is either an interior point or an exterior point; The matching value of the candidate keyframe is determined based on the attributes of the feature points, and the matching value is used to characterize the degree of matching between the candidate keyframe and the current keyframe. If the matching value meets the preset ratio condition, the second verification result is determined to be a successful verification.

4. The method according to any one of claims 1-3, characterized in that, The acquisition of target feature map data, including the current keyframe, includes: Acquire the current image frame and historical feature map data, wherein the historical feature map data includes global pose information corresponding to each key frame; If the image sequence position of the current image frame satisfies the image frame distance condition, the current image frame is determined to be the current keyframe; The historical feature map data is updated based on the feature data corresponding to the current keyframe to obtain the target feature map data; Based on the global pose information, the pose of the current keyframe is estimated to obtain image pose information; Obtain the global features and local features corresponding to the current keyframe; Based on the current keyframe, the image pose information, the global features, and the local features, the historical feature map data is updated to obtain the target feature map data.

5. The method according to any one of claims 1-3, characterized in that, Based on the difference matrix, After determining the loopback frame corresponding to the current keyframe from the non-common keyframe sequence, the method further includes: Obtain semantic map data corresponding to the target feature map data, wherein the semantic map data includes the pose information of key frames; Based on the timestamp of the current keyframe and the timestamp of the loopback frame, determine the first local point cloud data corresponding to the current keyframe and the second local point cloud data corresponding to the loopback frame in the semantic map data; The first local point cloud data and the second local point cloud data are registered to obtain the trajectory relative pose information. Determine the inter-frame relative pose information between every two keyframes in the semantic map data; Based on the trajectory relative pose information and the inter-frame relative pose information, the semantic map data is optimized to obtain optimized trajectory data; The target feature map data is updated based on the optimized trajectory data to obtain updated target feature map data.

6. The method according to claim 5, characterized in that, The step of updating the target feature map data based on the optimized trajectory data to obtain updated target feature map data includes: Perform time-series matching on the optimized trajectory data and the target feature map data to obtain the starting keyframe corresponding to the first node of the optimized trajectory data in the target feature map data; If the timestamp corresponding to the first node is less than the timestamp corresponding to the starting keyframe, determine the timestamp pair corresponding to the timestamp of the starting keyframe in the optimized trajectory data. Based on the timestamp pairs in the optimized trajectory data, the target feature map data is interpolated to obtain the updated target feature map data.

7. A map data processing device, characterized in that, The device includes: The current feature map acquisition module is used to acquire target feature map data including the current keyframe, wherein the target feature map data includes the non-common keyframe sequence corresponding to the current keyframe and the current frame sequence to which the current keyframe belongs; The difference matrix determination module is used to determine a difference matrix based on the non-common keyframe sequence and the current frame sequence when the current keyframe satisfies the loop closure detection condition. The difference matrix is ​​used to characterize the feature difference between each non-common keyframe in the non-common keyframe sequence and each keyframe in the current frame sequence. The loopback frame determination module is used to determine the loopback frame corresponding to the current keyframe from the non-co-visual keyframe sequence based on the difference matrix. The loopback frame is determined based on the target path in the difference matrix, where the target path is the shortest path that traverses the matrix elements corresponding to each non-co-visual keyframe and has the smallest path feature difference. The module performs shortest path planning by searching for the minimum path feature difference in the difference matrix to obtain the target path. It then determines the non-co-visual keyframe corresponding to the current keyframe in the target path as a candidate keyframe. The module performs credibility verification on the candidate keyframe to obtain a first verification result. If the first verification result is successful, it performs matching degree verification on the candidate keyframe to obtain a second verification result. If the second verification result is successful, it determines the candidate keyframe as the loopback frame.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the map data processing method as described in any one of claims 1-6.

9. An electronic device, characterized in that, The method includes at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor implements the map data processing method as described in any one of claims 1-6 by executing the instructions stored in the memory.

Images