Image pose optimization method, device and equipment based on control point loop detection

Abstract

The application provides an image pose optimization method and device based on control point loop detection, and equipment, obtains an initial pose of at least one image to be optimized and control point observation records; combines observation time stamps in the same control point label group based on control point labels, constructs at least one candidate time stamp pair; filters a candidate image set based on the at least one candidate time stamp pair, and performs image pair filtering matching processing on the candidate image set in each candidate time stamp pair, and constructs a control point loop edge; adds the control point loop edge, a pre-constructed odometer edge, a common visual loop edge and a manual loop edge to the image to be optimized, optimizes the initial pose of the image to be optimized, and obtains an optimized image pose. In this way, the efficiency and accuracy of image loop detection pose optimization can be improved.

Description

Technical Field

[0001] This application relates to the field of computer vision image processing technology, and in particular to image pose optimization methods, apparatus and equipment based on control point loop closure detection. Background Technology

[0002] In visual 3D reconstruction or visual SLAM systems, the acquisition device continuously acquires images along the scene and estimates the camera pose based on the feature matching relationships between the images. Since odometry-based adjacent frame constraints accumulate errors as the acquisition distance increases, the system typically needs to use loop closure detection to identify repeated observations of the same physical area at different times or in different acquisition rounds, thereby establishing cross-temporal constraints and eliminating drift through pose graph optimization.

[0003] Existing loop closure detection methods mainly rely on global visual retrieval or co-view relationship filtering. For example, the system retrieves historical images with similar appearances to the current image from an image database, and then establishes loop closure edges through local feature matching and geometric verification. This type of method is effective in general scenarios, but in large-scale engineering acquisition tasks, the excessively large candidate search range, appearance similarity leading to false loop closures, and appearance changes leading to missed loop closures affect the efficiency and accuracy of image loop closure detection pose optimization. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide an image pose optimization method, apparatus, and device based on control point loop closure detection. By generating candidate timestamp pairs through control point labels and observation timestamps, the loop closure search is narrowed from the global image library to a local time window near the control points, reducing the computational load of retrieval and matching. At the same time, in the process of constructing control point loop closure edges, objectively existing control points are used as the benchmark, reducing the problem of loop closure errors caused by appearance similarity and appearance changes. By increasing the constraints of control point loop closure edges, it helps to improve the efficiency and accuracy of image loop closure detection pose optimization.

[0005] In a first aspect, embodiments of this application provide an image pose optimization method based on control point loop closure detection, the image pose optimization method comprising: Acquire the initial pose and control point observation records of at least one image to be optimized; wherein, the control point observation records include at least control point labels and observation timestamps; Based on the control point labels, the observation timestamps within the same control point label group are combined to construct at least one candidate timestamp pair; For each candidate timestamp pair, taking the timestamps contained in the candidate timestamp pair as the center, select the image set corresponding to each timestamp based on a preset time window, and generate a candidate image pair set from the determined image sets; for each candidate timestamp pair, filter out initial candidate image pairs with similarity matching; filter at least one initial candidate image pair according to preset filtering conditions to obtain at least one filtered candidate image pair; perform local feature matching and geometric filtering on each filtered candidate image pair to determine at least one target candidate image pair; and construct control point loop edges based on the similarity transformation results of each target candidate image pair. The control point loop edges, pre-constructed odometer edges, ordinary visual loop edges, and manual loop edges are added to the image to be optimized, and the initial pose of the image to be optimized is optimized to obtain the optimized image pose.

[0006] In one possible implementation, for each candidate timestamp pair, selecting the image set corresponding to each timestamp based on a preset time window, and generating a candidate image pair set from the determined image sets, includes: For each candidate timestamp pair, images within a preset time window are selected as the first image set, centered on the observation timestamp of the first time in the candidate timestamp pair; images within a preset time window are selected as the second image set, centered on the observation timestamp of the second time. Based on the first image set and the second image set, determine the set of candidate image pairs corresponding to the candidate timestamp pair.

[0007] In one possible implementation, for each candidate timestamp pair, initial candidate image pairs with similarity matching are selected; at least one initial candidate image pair is filtered according to preset filtering conditions to obtain at least one filtered candidate image pair; local feature matching and geometric filtering are performed on each filtered candidate image pair to determine at least one target candidate image pair; and control point closure edges are constructed based on the similarity transformation results of each target candidate image pair, including: For each candidate timestamp pair, the candidate images in the first image set corresponding to the candidate timestamp pair are matched with the candidate images in the second image set to determine at least one initial candidate image pair; At least one initial candidate image pair is filtered according to preset filtering conditions to determine at least one filtered candidate image pair; For each candidate image pair, local feature matching and geometric filtering are performed to determine at least one target candidate image pair; For each of the target candidate image pairs, a similarity transformation is performed on the two candidate images in the target candidate image pair to obtain the similarity transformation result corresponding to the target candidate image pair; Based on the similarity transformation results of each target candidate image pair, control point loop edges are constructed.

[0008] In one possible implementation, the step of performing local feature matching and geometric filtering for each candidate image pair to determine at least one target candidate image pair includes: For each candidate image pair, local feature matching is performed on the candidate image pair to obtain the candidate image matching result; based on the candidate image matching result, filtering is performed according to preset geometric filtering conditions to determine at least one target candidate image pair; The geometric filtering conditions include at least one of the following: The following conditions must be met: the number of inliers in the candidate image is not less than the minimum inlier count threshold; the ratio of the number of inliers in the candidate image to the original number of matches is not less than the minimum inlier ratio threshold; the triangulation angle in the candidate image is not less than the minimum triangulation angle threshold; the configuration of the candidate image is not invalid; and the number of loop connections in the candidate image is less than the preset connection count threshold.

[0009] In one possible implementation, constructing control point lap edges based on the similarity transformation results of each target candidate image pair includes: For target candidate image pairs whose similarity transformation results are successful, construct complete relative pose constraint edges that include rotation, translation, and scaling; For target candidate image pairs whose similarity transformation results are unsuccessful, a unit translation direction constraint edge is constructed when the estimated translation direction is valid; Based on the complete relative pose constraint edges and / or unit translation direction constraint edges, construct control point loop edges.

[0010] In one possible implementation, optimizing the initial pose of the image to be optimized to obtain the optimized image pose includes: The number of odometer edges and loop closure edges in the image to be optimized are determined, and the weights of the odometer edges and loop closure edges are adjusted according to the ratio between the number of odometer edges and the number of loop closure edges. A switch variable is set for the control point loop closure edge, and the initial pose and the switch variable are used as optimization variables. The loop closure residual is adjusted based on the switch variable. The first stage of optimization is performed on the image to be optimized according to the adjusted odometry edge weight, the loop closure edge weight, the control point loop closure edge, the odometry edge, the ordinary visual loop closure edge, and the manual loop closure edge. The switch variable represents the confidence level of the loop closure edge. The credibility of each loop edge is determined based on the switch variable, and at least one loop edge with a credibility lower than a preset credibility threshold is filtered out. After restoring the image pose of the image to be optimized to the initial pose before the first stage of optimization, the second stage of optimization is performed on the image to be optimized based on the filtered loop closure edges and odometer edges according to the adjusted odometer edge weights and loop closure edge weights, to obtain the optimized image pose.

[0011] In one possible implementation, the image pose optimization method further includes: Obtain the observation records of the first control point in the first image to be optimized and the observation records of the second control point in the second image to be optimized; The observation timestamps of the first image to be optimized and the second image to be optimized are grouped according to the control point labels of the first control point observation record and the second control point observation record. When the same target label exists in the first image to be optimized and the second image to be optimized, select the set of images within a preset time window corresponding to the observation time of the target label in the two first images to be optimized and the second image to be optimized respectively; Cross-image retrieval and feature matching are performed on the first image to be optimized and the selected image set of the second image to be optimized to establish cross-image loop edges; Based on the cross-image loop edges, the first image to be optimized and the second image to be optimized are fused.

[0012] Secondly, embodiments of this application also provide an image pose optimization device based on control point loop closure detection, the image pose optimization device comprising: The data loading module is used to acquire the initial pose of at least one image to be optimized and control point observation records; wherein, the control point observation records include at least control point labels and observation timestamps; The control point grouping module is used to combine the observation timestamps within the same control point label group based on the control point label to construct at least one candidate timestamp pair; The loop closure edge establishment module is used to, for each candidate timestamp pair, select the image set corresponding to each timestamp based on a preset time window, and generate a candidate image pair set by combining the determined image sets; for each candidate timestamp pair, filter out initial candidate image pairs with similarity matching; filter at least one initial candidate image pair according to preset filtering conditions to obtain at least one filtered candidate image pair; perform local feature matching and geometric filtering on each filtered candidate image pair to determine at least one target candidate image pair; and construct control point loop closure edges based on the similarity transformation results of each target candidate image pair. The image pose optimization module is used to add the control point loop edges, pre-constructed odometer edges, ordinary visual loop edges, and manual loop edges to the image to be optimized, and optimize the initial pose of the image to be optimized to obtain the optimized image pose.

[0013] Thirdly, embodiments of this application also provide an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the image pose optimization method based on control point loop closure detection as described in any of the first aspects.

[0014] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the image pose optimization method based on control point loop closure detection as described in any of the first aspects.

[0015] The image pose optimization method, apparatus, and device based on control point loop closure detection provided in this application acquire the initial pose and control point observation records of at least one image to be optimized; combine observation timestamps within the same control point label group based on control point labels to construct at least one candidate timestamp pair; filter candidate image sets based on at least one candidate timestamp pair, and perform image pair filtering and matching processing on the candidate image sets in each candidate timestamp pair to construct control point loop closure edges; add control point loop closure edges, pre-constructed odometry edges, ordinary visual loop closure edges, and manual loop closure edges to the image to be optimized, and optimize the initial pose of the image to be optimized to obtain the optimized image pose. In this way, by generating candidate timestamp pairs through control point labels and observation timestamps, the loop closure search is narrowed from the global image library to a local time window near the control points, reducing the computational load of retrieval and matching. Simultaneously, by using objectively existing control points as a benchmark during the construction of control point loop closure edges, the problem of loop closure errors caused by similar appearances and appearance changes is reduced. By increasing the constraints of control point loop closure edges, the efficiency and accuracy of image loop closure detection pose optimization are improved.

[0016] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating an image pose optimization method based on control point loop closure detection provided in this application embodiment; Figure 2 This is one of the structural schematic diagrams of an image pose optimization device based on control point loop closure detection provided in an embodiment of this application; Figure 3 This is a second schematic diagram of an image pose optimization method device based on control point loop closure detection provided in an embodiment of this application. Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0020] First, the applicable scenarios for this application will be introduced. This application can be applied to the field of computer vision image processing technology.

[0021] In visual 3D reconstruction or visual SLAM systems, the acquisition device continuously acquires images along the scene and estimates the camera pose based on the feature matching relationships between the images. Since odometry-based adjacent frame constraints accumulate errors as the acquisition distance increases, the system typically needs to use loop closure detection to identify repeated observations of the same physical area at different times or in different acquisition rounds, thereby establishing cross-temporal constraints and eliminating drift through pose graph optimization.

[0022] Existing loop closure detection methods mainly rely on global visual retrieval or co-occurrence relationship filtering. For example, the system retrieves historical images with similar appearances to the current image from an image database, and then establishes loop edges through local feature matching and geometric verification. While these methods are effective in general scenarios, they still suffer from the following problems in large-scale engineering data acquisition tasks: 1. The candidate search range is too large. Pure visual retrieval usually requires searching for candidate images in the entire image database. As the data scale increases, the computational cost of retrieval and subsequent matching increases significantly.

[0023] 2. Similar appearance leads to false loop closures. Repetitive structures such as urban roads, corridors, facades, and road signs can easily generate candidate images that are visually similar but physically different. Introducing false loop closure constraints may cause pose graph distortion or even optimization divergence.

[0024] 3. Appearance variations lead to missed loops. When the appearance of the same location varies significantly under different lighting, weather, viewing angles, or data collection rounds, relying solely on visual similarity may not be sufficient to recall valid loops, resulting in accumulated drift that cannot be contained in a timely manner.

[0025] 4. Engineering control point information is not fully utilized. In surveying, scanning, digital twin, and other data acquisition tasks, there are often control points, marker points, docking points, or other reproducibly identifiable physical anchor points with unique numbers or tags on site. Existing loopback processes typically only use these as post-processing alignment or manual inspection information, failing to fully utilize their tags and observation time to generate highly reliable loopback candidates.

[0026] 5. Loop closure quality filtering is not robust enough. Conventional methods often use fixed thresholds for hard filtering, which can easily lead to the accidental deletion of valid loops or the retention of incorrect loops under different data scales, loop density, and scene complexity, affecting the efficiency and accuracy of image loop closure detection pose optimization.

[0027] Based on this, this application provides an image pose optimization method based on control point loop closure detection to improve the efficiency and accuracy of image loop closure detection pose optimization.

[0028] Please see Figure 1 , Figure 1 This is a flowchart illustrating an image pose optimization method based on control point loop closure detection, provided as an embodiment of this application. Figure 1 As shown in the figure, the image pose optimization method based on control point loop closure detection provided in this application includes: S101. Obtain the initial pose and control point observation record of at least one image to be optimized; wherein the control point observation record includes at least control point labels and observation timestamps.

[0029] S102. Based on the control point label, combine the observation timestamps within the same control point label group to construct at least one candidate timestamp pair.

[0030] S103. For each candidate timestamp pair, taking the timestamps contained in the candidate timestamp pair as the center, select the image set corresponding to each timestamp based on a preset time window, and generate a candidate image pair set from the determined image sets; for each candidate timestamp pair, filter out initial candidate image pairs with similarity matching; filter at least one initial candidate image pair according to preset filtering conditions to obtain at least one filtered candidate image pair; perform local feature matching and geometric filtering on each filtered candidate image pair to determine at least one target candidate image pair; and construct control point lap edges according to the similarity transformation results of each target candidate image pair.

[0031] S104. Add the control point loop edge, the pre-constructed odometer edge, the ordinary visual loop edge, and the manual loop edge to the image to be optimized, and optimize the initial pose of the image to be optimized to obtain the optimized image pose.

[0032] The image pose optimization method based on control point loop closure detection provided in this application generates candidate timestamp pairs by using control point labels and observation timestamps. This shrinks the loop closure search from the global image library to a local time window near the control points, reducing the computational load of retrieval and matching. At the same time, by using objectively existing control points as a benchmark in the process of constructing control point loop closure edges, the problem of loop closure errors caused by similar appearances and appearance changes is reduced. By increasing the constraints on control point loop closure edges, the efficiency and accuracy of image loop closure detection pose optimization are improved.

[0033] The exemplary steps of the embodiments of this application are described below: S101. Obtain the initial pose and control point observation record of at least one image to be optimized; wherein the control point observation record includes at least control point labels and observation timestamps.

[0034] Here, control points can be manually set markers, automatically identified markers, or fixed anchor points with unique numbers in the scene. Specifically, they can be control points, markers, docking points, or other reproducibly identifiable physical anchor points with unique numbers or tags that exist on-site during surveying, scanning, digital twin, and other data collection tasks.

[0035] In one possible implementation, the registered image to be optimized can be acquired first, and then the initial pose, camera parameters, image database, feature matching data, optional Rig configuration, and control point observation records of the image to be optimized can be acquired.

[0036] Here, the control point observation record may include control point labels and observation timestamps, or it may include the spatial location of the control point, the acquisition round, the associated device number or the associated image number. Then, control point loop edges can be constructed based on the control point labels and observation timestamps, and the control point loop edges can be added to the image to be optimized for pose optimization.

[0037] In one possible implementation, control point loop edges can be generated using only control point labels and observation timestamps; the spatial location of the control points can be used as an additional filtering condition or quality assessment indicator, thereby improving the accuracy of control point loop edge filtering.

[0038] S102. Based on the control point label, combine the observation timestamps within the same control point label group to construct at least one candidate timestamp pair.

[0039] In one possible implementation, all control point observation records can be read first, and then grouped according to the control point labels.

[0040] Specifically, each group contains a set of observation timestamps under the same control point label. If any control point label appears only once and cannot be paired, then that control point label cannot constitute a cross-time control point loop candidate and no further processing is required.

[0041] In this way, the full image-level loop closure search problem is transformed into a local candidate search problem under control point label constraints, thereby reducing the number of erroneous candidate loops and improving the accuracy of loop edge generation.

[0042] In one possible implementation, observation timestamps within the same label group can be combined in pairs to generate at least one candidate timestamp pair.

[0043] For example, if the same control point label is observed at times t1, t2, and t3, then three candidate timestamp pairs (t1, t2), (t1, t3), and (t2, t3) are generated.

[0044] Here, each candidate timestamp pair represents repeated observations of the same control point or the same physical location at different times, in different rounds, or in different acquisition segments.

[0045] Furthermore, after identifying at least one candidate timestamp pair, multiple candidate image sets can be filtered based on the candidate timestamps, and then image filtering and matching processing can be performed to construct control point loop edges.

[0046] S103. For each candidate timestamp pair, taking the timestamps contained in the candidate timestamp pair as the center, select the image set corresponding to each timestamp based on a preset time window, and generate a candidate image pair set from the determined image sets; for each candidate timestamp pair, filter out initial candidate image pairs with similarity matching; filter at least one initial candidate image pair according to preset filtering conditions to obtain at least one filtered candidate image pair; perform local feature matching and geometric filtering on each filtered candidate image pair to determine at least one target candidate image pair; and construct control point lap edges according to the similarity transformation results of each target candidate image pair.

[0047] In one possible implementation, for each candidate timestamp pair, an image set can be selected within a preset time window based on the timestamps contained in the candidate timestamp pair.

[0048] Specifically, the step "For each candidate timestamp pair, taking the timestamps contained in the candidate timestamp pair as the center, selecting the image set corresponding to each timestamp based on a preset time window, and generating a candidate image pair set from the determined image sets" includes: a1: For each candidate timestamp pair, take the observation timestamp of the first time in the candidate timestamp pair as the center, select the images within the preset time window as the first image set; take the observation timestamp of the second time as the center, select the images within the preset time window as the second image set.

[0049] a2: Based on the first image set and the second image set, determine the set of candidate image pairs corresponding to the candidate timestamp pair.

[0050] In one possible implementation, for each candidate timestamp pair, the image set corresponding to each timestamp can be selected based on the timestamp contained in the candidate timestamp pair and a preset time window, and the determined image sets can be used to generate a candidate image pair set.

[0051] For example, for the candidate timestamp pair (t1, t2) in the above example, a first image set l1 containing the range of t1±Δt and a second image set l2 containing the range of t2±Δt can be obtained. The first image set l1 and the second image set l2 constitute the candidate image pair set. Other candidate timestamp pairs generate corresponding candidate image pair sets in the same way.

[0052] Δt is a preset time window used to compensate for the deviation between the control point identification time, the image acquisition time, and the device movement, thereby improving the accuracy of subsequent control point loopback generation.

[0053] Furthermore, after obtaining the candidate image pair set, visual retrieval verification, preliminary screening, local feature matching and geometric filtering, and Sim3 estimation can be performed on the candidate image pair set to generate control point loop edges.

[0054] Specifically, the steps "for each candidate timestamp pair, select initial candidate image pairs with similarity matching; filter at least one initial candidate image pair according to preset filtering conditions to obtain at least one filtered candidate image pair; perform local feature matching and geometric filtering on each filtered candidate image pair to determine at least one target candidate image pair; and construct control point lap edges based on the similarity transformation results of each target candidate image pair" include: b1: For each candidate timestamp pair, perform similarity matching between the candidate image in the first image set corresponding to the candidate timestamp pair and the candidate image in the second image set to determine at least one initial candidate image pair.

[0055] b2: Filter at least one initial candidate image pair according to preset filtering conditions to determine at least one filtered candidate image pair.

[0056] b3: Perform local feature matching and geometric filtering for each candidate image pair to determine at least one target candidate image pair.

[0057] b4: For each of the target candidate image pairs, perform a similarity transformation based on the two candidate images in the target candidate image pair to obtain the similarity transformation result corresponding to the target candidate image pair.

[0058] b5: Construct control point loop edges based on the similarity transformation results of each target candidate image pair.

[0059] In one possible implementation, for each candidate timestamp pair, image retrieval verification is first performed on the image sets located within two time windows. This can be done by using candidate images from the first image set as query images and candidate images from the second image set as candidate database images for similarity matching, or by using candidate images from the second image set as query images and candidate images from the first image set as candidate database images for similarity matching, thereby filtering out initial candidate image pairs with similarity matching.

[0060] Here, an image retrieval model can be used for image matching to output initial candidate image pairs with similarity matching.

[0061] For example, the image retrieval model can be a depth image retrieval model; in specific engineering implementations, MegaLoc or other models with global image description capabilities can be used. Homogeneous or heterogeneous retrieval modes can be selected depending on whether the camera models are consistent.

[0062] In this way, the prior control point labels will not directly become hard loop constraints, but will first be transformed into high-recall candidates and then verified by visual overlap relationships, which helps to improve the accuracy of control edge loop generation.

[0063] Furthermore, after obtaining at least one initial candidate image pair, the at least one initial candidate image pair can be filtered according to preset filtering conditions to obtain at least one filtered candidate image pair.

[0064] In one possible implementation, the preset screening conditions may include at least one of the following: both candidate images are located in the current 3D reconstruction model, the two candidate images are not the same image, the currently processed candidate image is not occupied by other loop edges, the time interval between the two candidate images is greater than the minimum loop interval, the initial spatial distance between the two candidate images is less than the maximum loop distance, and the number of existing loop edges in each candidate image is less than a preset connection number threshold.

[0065] In this way, by filtering at least one initial candidate image pair according to preset screening conditions, it is possible to avoid adjacent frames being mistakenly used as loop edges, and also to avoid a single candidate image being connected with too many loop edges, which would lead to excessive local constraints, thereby improving the accuracy of subsequent image pose optimization.

[0066] Furthermore, after determining at least one candidate image pair, local feature matching can be performed on each candidate image pair to determine the candidate image matching result, and then geometric filtering can be performed on the candidate image matching result to determine at least one target candidate image pair.

[0067] Specifically, the step "performing local feature matching and geometric filtering for each candidate image pair to determine at least one target candidate image pair" includes: c1: For each candidate image pair, perform local feature matching on the candidate image pair to obtain the candidate image matching result; based on the candidate image matching result, filter according to the preset geometric filtering conditions to determine at least one target candidate image pair.

[0068] In one possible implementation, the candidate image matching result can be a two-view geometry result. The two-view geometry result is filtered according to a preset geometric filtering condition to filter out candidate image pairs that do not meet the requirements and determine at least one target candidate image pair.

[0069] The geometric filtering conditions include at least one of the following: The following conditions must be met: the number of inliers in the candidate image is not less than the minimum inlier count threshold; the ratio of the number of inliers in the candidate image to the original number of matches is not less than the minimum inlier ratio threshold; the triangulation angle in the candidate image is not less than the minimum triangulation angle threshold; the configuration of the candidate image is not invalid; and the number of loop connections in the candidate image is less than the preset connection count threshold.

[0070] In this way, after local feature matching and geometric filtering, the control point prior, visual similarity and geometric consistency are combined to avoid establishing unverified hard constraints based solely on control point labels, thereby improving the accuracy of constructing control point loop edges.

[0071] Furthermore, for at least one determined target candidate image pair, corresponding two-dimensional points can be extracted from each target candidate image in the target candidate image pair, and further associated with the three-dimensional points in their respective reconstructions. A robust estimation algorithm is then used to estimate the similarity transformation between the two target candidate images, yielding similarity transformation results such as rotation, translation, and scaling.

[0072] Here, the Sim3 transformation can be estimated using LORANSAC and a similarity transform estimator, and then re-estimated using all interior points after obtaining the set of interior points.

[0073] Furthermore, control point loop edges can be constructed based on the similarity transformation results for the target candidate image pairs.

[0074] Specifically, the step "constructing control point loop edges based on the similarity transformation results of each target candidate image pair" includes: d1: For target candidate image pairs whose similarity transformation results are successful, construct complete relative pose constraint edges including rotation, translation and scaling.

[0075] d2: For target candidate image pairs whose similarity transformation results are unsuccessful, construct unit translation direction constraint edges when the estimated translation direction is valid.

[0076] d3: Based on the complete relative pose constraint edge and / or unit translation direction constraint edge, construct the control point loop edge.

[0077] In one possible implementation, if the similarity transformation is successful, the rotation and translation of the pose can be constrained simultaneously, and scale-related information can be provided to construct a complete relative pose constraint edge. If the similarity transformation is unsuccessful, a unit translation direction constraint edge can be constructed if the estimated translation direction is valid. Furthermore, the unit translation direction constraint edge and the complete relative pose constraint edge can be added to the image to be optimized as independent control point loop closure edges, or as a combined constraint of the same control point loop closure image pair, for subsequent pose optimization.

[0078] Furthermore, after determining the control point loop edges, the control point loop edges, pre-constructed odometer edges, ordinary visual loop edges, and manual loop edges can be combined with the image to be optimized to perform pose optimization, thereby obtaining the optimized image pose.

[0079] S104. Add the control point loop edge, the pre-constructed odometer edge, the ordinary visual loop edge, and the manual loop edge to the image to be optimized, and optimize the initial pose of the image to be optimized to obtain the optimized image pose.

[0080] Here, odometry edges are edges that connect adjacent images within a time window or sequence window, sorted by image timestamps according to camera or acquisition sequence, providing local continuity constraints; ordinary visual loop closure edges are loop closure edges added to the image to be optimized after reading the geometric relationship between two views from the image database and performing Sim3 estimation on the image pairs that meet the geometric constraints; manual loop closure edges are loop closure edges added to the image to be optimized after the system retrieves and verifies the image pairs within the corresponding time window, provided by the operator or external system, by the operator or external system.

[0081] In one possible implementation, ordinary visual loop closures, control point loop closures, and manual loop closures can share the same set of geometric filtering, Sim3 estimation, constraint edge construction, and switching variables for processing.

[0082] In the embodiments of this application, in order to enhance the robustness to erroneous loop closures during the pose optimization of the image to be optimized through different edges, the present invention can introduce switchable constraints to the loop closure image pairs, calculate the loop closure residuals by switching variables, dynamically adjust the odometer edge weights and loop closure edge weights, and perform pose optimization on the image to be optimized in two stages, thereby improving the accuracy of image pose optimization.

[0083] Specifically, the step "optimizing the initial pose of the image to be optimized to obtain the optimized image pose" includes: e1: Determine the number of odometer edges and loopback edges in the image to be optimized, and adjust the weights of odometer edges and loopback edges according to the ratio of the number of odometer edges to the number of loopback edges.

[0084] e2: Set a switch variable for the control point loop closure edge, and use the initial pose and the switch variable as optimization variables. Adjust the loop closure residual based on the switch variable, and perform the first stage optimization on the image to be optimized according to the adjusted odometry edge weight, the loop closure edge weight, the control point loop closure edge, the odometry edge, the ordinary visual loop closure edge, and the manual loop closure edge; wherein, the switch variable represents the confidence level of the loop closure edge.

[0085] e3: Determine the credibility of each loop edge based on the switch variable, and filter out at least one loop edge whose credibility is lower than the preset credibility threshold.

[0086] e4: After restoring the image pose of the image to be optimized to the initial pose before the first stage of optimization, the second stage of optimization is performed on the image to be optimized based on the filtered loop edges and odometer edges according to the adjusted odometer edge weights and loop edge weights, to obtain the optimized image pose.

[0087] In this embodiment, loop closures include ordinary visual loop closures, control point loop closures, and manual loop closures. In large-scale image sequences, the number of odometry edges is usually much greater than the number of loop closures. If fixed weights are used directly, loop closures may not be able to effectively eliminate global drift; if the weights of loop closures are blindly increased, the impact of erroneous loops may be amplified. Therefore, the odometry edge weights and loop closure weights can be dynamically adjusted by the ratio between the number of odometry edges and the number of loop closures.

[0088] Here, the number of effective loops can be calculated by introducing a loop equivalence coefficient k. Specifically, the number of effective loops can be calculated using the following formula: N_eff = k * N_loop; Where N_eff is the number of valid loops; k is the loop equivalence coefficient; and N_loop is the number of complete loop edges.

[0089] Furthermore, after determining the number of valid loopbacks, the odometer edge scaling factor and the loopback edge scaling factor can be calculated. The adjusted odometer edge weight is obtained by multiplying the preset odometer edge weight by the odometer edge scaling factor; the adjusted loopback edge weight is obtained by multiplying the preset loopback edge weight by the loopback edge scaling factor.

[0090] Specifically, the odometer scaling factor can be calculated using the following formula: alpha_odom = clamp(sqrt(N_eff / N_odom), alpha_min, 1); Where alpha_odom is the odometer edge scaling factor; clamp() restricts the value to a specified range; N_eff is the number of valid loops; N_odom is the number of odometer edges; alpha_min is the preset lower limit threshold of the odometer edge scaling factor; and sqrt() is the square root function.

[0091] Specifically, the loop scaling factor can be calculated using the following formula: alpha_loop = clamp(sqrt(N_odom / N_eff), 1, alpha_max); Where alpha_loop is the loop scaling factor; clamp() restricts the value to a specified range; N_eff is the number of valid loops; N_odom is the number of odometry edges; alpha_max is the preset upper limit threshold for the loop scaling factor; and sqrt() is the square root function.

[0092] Furthermore, the adjusted odometer edge weight is obtained by multiplying the preset odometer edge weight by the odometer edge scaling factor; the adjusted loopback edge weight is obtained by multiplying the preset loopback edge weight by the loopback edge scaling factor. Specifically, when the loopback edges are relatively sparse, the loopback edge weight is enhanced; when the odometer edges are relatively too dense, the odometer edge weight is appropriately reduced.

[0093] In one possible implementation, to enhance robustness to erroneous loopbacks, switchable constraints can be introduced into the loopback image pairs. For one or more residual terms corresponding to the same loopback image pair, a shared switch variable is set, and the loopback residuals are adjusted based on the switch variable.

[0094] The switch variable can range from 0 to 1. The switch variable represents the confidence level of the loop edge. The closer the switch variable is to 1, the higher the confidence level of the corresponding loop edge. The closer the switch variable is to 0, the lower the confidence level of the corresponding loop edge, and the more the loop edge should be suppressed.

[0095] Specifically, the loopback residuals can be adjusted using the following formula: r_switchable = s * r_base; Where r_switchable is the loopback residual after introducing the switch variable; s is the switch variable; and r_base is the loopback residual before introducing the switch variable.

[0096] In one possible implementation, to prevent all switching variables from unconstrainedly approaching 0, a switching prior cost can be introduced, which can be calculated using the following formula: r_prior =sqrt(w_prior)*(1-s); Where r_prior is the switch prior cost; w_prior is the switch prior weight; s is the switch variable; and sqrt() is the square root function.

[0097] Here, the switch prior weights keep the loop closure constraints in effect by default, and the optimizer will only reduce the corresponding switch variables when they obviously conflict with other constraints; the switch prior weights can be adjusted synchronously with the square of the loop closure edge scaling factor to maintain the balance between the loop closure residuals and the switch priors.

[0098] In another possible implementation, the loop closure residuals can also be adjusted using equivalent weighting functions, robust kernel functions, or monotonic confidence functions.

[0099] In one possible implementation, the first stage of optimization can be performed on the image to be optimized by using switch variables. Specifically, the initial pose and switch variables are used as optimization variables. The loop closure residuals are adjusted based on the switch variables. The first stage of optimization is performed on the image to be optimized according to the adjusted odometry edge weights and loop closure edge weights, as well as the control point loop closure edges, odometry edges, ordinary visual loop closure edges, and manual loop closure edges.

[0100] Furthermore, after completing the first stage of optimization, loop closure reliability can be determined based on the switch variables. Specifically, the switch variables corresponding to each loop closure image are counted. If the switch variable is less than the preset switch variable threshold, the reliability of the loop closure edge is determined to be lower than the preset reliability threshold, and the loop closure image pair is determined to be unreliable and removed from the final edge set.

[0101] Furthermore, after restoring the image pose of the image to be optimized to the initial pose before the first stage of optimization, the second stage of optimization is performed on the image to be optimized based on the filtered loop closure edges and odometry edges according to the adjusted odometry edge weights and loop closure edge weights. No switching variables are introduced, and the optimized image pose is obtained.

[0102] In this way, through two-stage optimization—first evaluating the reliability of loop closures and then performing clean optimization—it can balance the ability to filter out erroneous loop closures with the stability of the final optimization result.

[0103] In one possible implementation, the control point loop closure optimization method can be applied to two separate images.

[0104] Specifically, the image pose optimization method further includes: f1: Obtain the observation records of the first control point in the first image to be optimized and the observation records of the second control point in the second image to be optimized.

[0105] f2: Group the observation timestamps of the first image to be optimized and the second image to be optimized according to the control point labels of the first control point observation record and the second control point observation record.

[0106] f3: When the same target label exists in the first image to be optimized and the second image to be optimized, select the set of images within the preset time window of the observation time corresponding to the target label in the two first images to be optimized and the second image to be optimized, respectively.

[0107] f4: Perform cross-image retrieval and feature matching on the first image to be optimized and the image set selected from the second image to be optimized, and establish cross-image loop edges.

[0108] f5: Based on the cross-image loop edge, fuse the first image to be optimized and the second image to be optimized.

[0109] In this embodiment, control point observation records in the first and second images to be optimized are read respectively, and a mapping from two labels to a set of timestamps is established according to the control point labels. When the two images to be optimized have the same label, an image set near the time of the control point is selected in the two images to be optimized, and cross-map candidate image pairs are generated through image retrieval and feature matching. An image set within a preset time window of the observation time corresponding to the target label is selected, and then image retrieval and feature matching are performed on the selected image set to establish a cross-image loop edge. The first and second images to be optimized are fused according to the cross-image loop edge, or the first and second images to be optimized are aligned.

[0110] Here, an image set near the time of the control point is selected, and cross-map candidate image pairs are generated through image retrieval and feature matching. An image set within a preset time window corresponding to the observation time of the target label is selected. The method for constructing cross-image loop edges is the same as the method for constructing control point loop edges mentioned above, and will not be repeated here.

[0111] In one possible implementation, the method of constructing cross-image loop edges to participate in image optimization can also be used for pose alignment between multiple acquisition tasks, multi-map fusion, or block reconstruction results. The specific processing method is the same as that for aligning and fusing two images to be optimized, and will not be described in detail here.

[0112] The image pose optimization process based on control point loop closure detection in this application embodiment will be illustrated below with specific examples: Example 1: Multi-round data acquisition of urban street scenes: The data acquisition vehicle, equipped with a multi-camera system, conducts multiple rounds of data acquisition along urban roads. During the acquisition process, multiple control points with unique labels exist in the scene. The system records the label and timestamp each time a control point is observed.

[0113] Furthermore, after loading the reconstruction model and control point observation records, the system groups the control point observations according to labels. Taking label CP-001 as an example, this control point was observed at times t1, t2, and t3, and the system generates three candidate timestamp pairs: (t1, t2), (t1, t3), and (t2, t3). For each timestamp pair, the system selects image sets near both timestamps, uses an image retrieval model to obtain candidate image pairs, and then performs local feature matching and geometric filtering.

[0114] Furthermore, for the filtered image pairs, the system extracts 3D point pairs from interior point matching and estimates the Sim3 transform using LORANSAC. If the translation direction is valid, a unit translation direction constraint is added; if the Sim3 estimation is successful, a complete relative pose constraint is added. Subsequently, the system performs a two-stage optimization process of adaptive weight adjustment and switchable constraints to obtain a globally consistent image pose.

[0115] Example 2: Control point loop between two independent maps: The first and second maps were generated by different acquisition tasks. Control point observation information is recorded in both maps. The system groups the control points in both maps by label. When the label CP-010 is found in both maps, the system selects the image set near the observation time of that label in each map.

[0116] Furthermore, the system uses images from the first map as query images and images from the second map as candidate database images, obtaining cross-map image pairs through image retrieval and local feature matching. After geometric verification and Sim3 estimation, the system can establish cross-map loop relationships for subsequent map fusion or coordinate system integration.

[0117] Example 3: Control point records with only labels and timestamps: In some low-cost acquisition tasks, control point records only contain labels and observation timestamps, and do not include the 3D coordinates of the control points. The system can still generate candidate loop closures by grouping by label and pairing by timestamp, and filter out erroneous candidates through image retrieval, feature matching, and geometric verification to complete image pose optimization.

[0118] The image pose optimization method based on control point loop closure detection provided in this application involves acquiring the initial pose and control point observation records of at least one image to be optimized; combining observation timestamps within the same control point label group based on control point labels to construct at least one candidate timestamp pair; filtering candidate image sets based on at least one candidate timestamp pair, and performing image pair filtering and matching processing on the candidate image sets in each candidate timestamp pair to construct control point loop closure edges; adding control point loop closure edges, pre-constructed odometry edges, ordinary visual loop closure edges, and manually constructed loop closure edges to the image to be optimized, and optimizing the initial pose of the image to be optimized to obtain the optimized image pose. In this way, by generating candidate timestamp pairs using control point labels and observation timestamps, the loop closure search is narrowed from the global image library to a local time window near the control points, reducing the computational load of retrieval and matching. Simultaneously, by using objectively existing control points as a benchmark during the construction of control point loop closure edges, the problem of loop closure errors caused by similar appearances and appearance changes is reduced. By increasing the constraints of control point loop closure edges, the efficiency and accuracy of image loop closure detection pose optimization are improved.

[0119] Based on the same inventive concept, this application also provides an image pose optimization device based on control point loop closure detection, which corresponds to the image pose optimization method based on control point loop closure detection. Since the principle of the device in this application is similar to the image pose optimization method based on control point loop closure detection described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0120] Please see Figure 2 , Figure 3 , Figure 2 This is one of the structural schematic diagrams of an image pose optimization device based on control point loop closure detection provided in an embodiment of this application. Figure 3 This is a second schematic diagram of an image pose optimization method device based on control point loop closure detection provided in an embodiment of this application. Figure 2 As shown, the image pose optimization device 200 includes: The data loading module 210 is used to acquire the initial pose and control point observation records of at least one image to be optimized; wherein, the control point observation records include at least control point labels and observation timestamps; The control point grouping module 220 is used to combine the observation timestamps within the same control point label group based on the control point label to construct at least one candidate timestamp pair; The loop closure edge establishment module 230 is used to, for each candidate timestamp pair, select the image set corresponding to each timestamp based on a preset time window, and generate a candidate image pair set by combining the determined image sets; for each candidate timestamp pair, filter out initial candidate image pairs with similarity matching; filter at least one initial candidate image pair according to preset filtering conditions to obtain at least one filtered candidate image pair; perform local feature matching and geometric filtering on each filtered candidate image pair to determine at least one target candidate image pair; and construct control point loop closure edges according to the similarity transformation results of each target candidate image pair. The image pose optimization module 240 is used to add the control point loop edge, the pre-constructed odometer edge, the ordinary visual loop edge, and the manual loop edge to the image to be optimized, and optimize the initial pose of the image to be optimized to obtain the optimized image pose.

[0121] In one possible implementation, when the loop closure edge establishment module 230 selects the image set corresponding to each time stamp based on a preset time window for each candidate time stamp pair, and generates a candidate image pair set from the determined image sets, the loop closure edge establishment module 230 is used to: For each candidate timestamp pair, images within a preset time window are selected as the first image set, centered on the observation timestamp of the first time in the candidate timestamp pair; images within a preset time window are selected as the second image set, centered on the observation timestamp of the second time. Based on the first image set and the second image set, determine the set of candidate image pairs corresponding to the candidate timestamp pair.

[0122] In one possible implementation, the loop closure edge establishment module 230 is used to: filter initial candidate image pairs with similarity matching for each candidate timestamp pair; filter at least one initial candidate image pair according to preset filtering conditions to obtain at least one filtered candidate image pair; perform local feature matching and geometric filtering for each filtered candidate image pair to determine at least one target candidate image pair; and construct control point loop closure edges based on the similarity transformation results of each target candidate image pair. The loop closure edge establishment module 230 is used to: For each candidate timestamp pair, the candidate images in the first image set corresponding to the candidate timestamp pair are matched with the candidate images in the second image set to determine at least one initial candidate image pair; At least one initial candidate image pair is filtered according to preset filtering conditions to determine at least one filtered candidate image pair; For each candidate image pair, local feature matching and geometric filtering are performed to determine at least one target candidate image pair; For each of the target candidate image pairs, a similarity transformation is performed on the two candidate images in the target candidate image pair to obtain the similarity transformation result corresponding to the target candidate image pair; Based on the similarity transformation results of each target candidate image pair, control point loop edges are constructed.

[0123] In one possible implementation, when the loop closure edge establishment module 230 is used to perform local feature matching and geometric filtering for each candidate image pair to determine at least one target candidate image pair, the loop closure edge establishment module 230 is used to: For each candidate image pair, local feature matching is performed on the candidate image pair to obtain the candidate image matching result; based on the candidate image matching result, filtering is performed according to preset geometric filtering conditions to determine at least one target candidate image pair; The geometric filtering conditions include at least one of the following: The following conditions must be met: the number of inliers in the candidate image is not less than the minimum inlier count threshold; the ratio of the number of inliers in the candidate image to the original number of matches is not less than the minimum inlier ratio threshold; the triangulation angle in the candidate image is not less than the minimum triangulation angle threshold; the configuration of the candidate image is not invalid; and the number of loop connections in the candidate image is less than the preset connection count threshold.

[0124] In one possible implementation, when constructing control point loop closure edges based on the similarity transformation results of each target candidate image pair, the loop closure edge construction module 230 is used to: For target candidate image pairs whose similarity transformation results are successful, construct complete relative pose constraint edges that include rotation, translation, and scaling; For target candidate image pairs whose similarity transformation results are unsuccessful, a unit translation direction constraint edge is constructed when the estimated translation direction is valid; Based on the complete relative pose constraint edges and / or unit translation direction constraint edges, construct control point loop edges.

[0125] In one possible implementation, when the image pose optimization module 240 optimizes the initial pose of the image to be optimized to obtain an optimized image pose, the image pose optimization module 240 is used to: The number of odometer edges and loop closure edges in the image to be optimized are determined, and the weights of the odometer edges and loop closure edges are adjusted according to the ratio between the number of odometer edges and the number of loop closure edges. A switch variable is set for the control point loop closure edge, and the initial pose and the switch variable are used as optimization variables. The loop closure residual is adjusted based on the switch variable. The first stage of optimization is performed on the image to be optimized according to the adjusted odometry edge weight, the loop closure edge weight, the control point loop closure edge, the odometry edge, the ordinary visual loop closure edge, and the manual loop closure edge. The switch variable represents the confidence level of the loop closure edge. The credibility of each loop edge is determined based on the switch variable, and at least one loop edge with a credibility lower than a preset credibility threshold is filtered out. After restoring the image pose of the image to be optimized to the initial pose before the first stage of optimization, the second stage of optimization is performed on the image to be optimized based on the filtered loop closure edges and odometer edges according to the adjusted odometer edge weights and loop closure edge weights, to obtain the optimized image pose.

[0126] In one possible implementation, such as Figure 3 As shown, the image pose optimization device 200 further includes a multi-image fusion module 250, which is used for: Obtain the observation records of the first control point in the first image to be optimized and the observation records of the second control point in the second image to be optimized; The observation timestamps of the first image to be optimized and the second image to be optimized are grouped according to the control point labels of the first control point observation record and the second control point observation record. When the same target label exists in the first image to be optimized and the second image to be optimized, select the set of images within a preset time window corresponding to the observation time of the target label in the two first images to be optimized and the second image to be optimized respectively; Cross-image retrieval and feature matching are performed on the first image to be optimized and the selected image set of the second image to be optimized to establish cross-image loop edges; Based on the cross-image loop edges, the first image to be optimized and the second image to be optimized are fused.

[0127] The image pose optimization device based on control point loop closure detection provided in this application acquires the initial pose and control point observation records of at least one image to be optimized; combines observation timestamps within the same control point label group based on control point labels to construct at least one candidate timestamp pair; filters a candidate image set based on at least one candidate timestamp pair, and performs image pair filtering and matching processing on the candidate image set in each candidate timestamp pair to construct control point loop closure edges; adds the control point loop closure edges, pre-constructed odometry edges, ordinary visual loop closure edges, and manual loop closure edges to the image to be optimized, and optimizes the initial pose of the image to be optimized to obtain the optimized image pose. In this way, by generating candidate timestamp pairs through control point labels and observation timestamps, the loop closure search is narrowed from the global image library to a local time window near the control points, reducing the computational load of retrieval and matching. Simultaneously, by using objectively existing control points as a benchmark during the construction of control point loop closure edges, the problem of loop closure errors caused by similar appearances and appearance changes is reduced. By increasing the constraints of control point loop closure edges, the efficiency and accuracy of image loop closure detection pose optimization are improved.

[0128] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 400 includes a processor 410, a memory 420, and a bus 430.

[0129] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, they can perform the operations described above. Figure 1The steps of the image pose optimization method based on control point loop closure detection in the method embodiment shown are specifically implemented in the method embodiment and will not be repeated here.

[0130] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figure 1 The steps of the image pose optimization method based on control point loop closure detection in the method embodiment shown are specifically implemented in the method embodiment and will not be repeated here.

[0131] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0132] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0133] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0134] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0135] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0136] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image pose optimization method based on control point loop closure detection, characterized in that, The image pose optimization method includes: Acquire the initial pose and control point observation records of at least one image to be optimized; wherein, the control point observation records include at least control point labels and observation timestamps; Based on the control point labels, the observation timestamps within the same control point label group are combined to construct at least one candidate timestamp pair; For each candidate timestamp pair, taking the timestamps contained in the candidate timestamp pair as the center, select the image set corresponding to each timestamp based on a preset time window, and generate a candidate image pair set from the determined image sets; for each candidate timestamp pair, filter out initial candidate image pairs with similarity matching; filter at least one initial candidate image pair according to preset filtering conditions to obtain at least one filtered candidate image pair; perform local feature matching and geometric filtering on each filtered candidate image pair to determine at least one target candidate image pair; and construct control point loop edges based on the similarity transformation results of each target candidate image pair. The control point loop edges, pre-constructed odometer edges, ordinary visual loop edges, and manual loop edges are added to the image to be optimized, and the initial pose of the image to be optimized is optimized to obtain the optimized image pose.

2. The image pose optimization method according to claim 1, characterized in that, For each candidate timestamp pair, taking the timestamps contained in the candidate timestamp pair as the center, and selecting the image set corresponding to each timestamp based on a preset time window, and generating a candidate image pair set from the determined image sets, including: For each candidate timestamp pair, images within a preset time window are selected as the first image set, centered on the observation timestamp of the first time in the candidate timestamp pair; images within a preset time window are selected as the second image set, centered on the observation timestamp of the second time. Based on the first image set and the second image set, determine the set of candidate image pairs corresponding to the candidate timestamp pair.

3. The image pose optimization method according to claim 2, characterized in that, For each candidate timestamp pair, initial candidate image pairs with similarity matching are selected; at least one initial candidate image pair is selected according to preset selection conditions to obtain at least one selected candidate image pair. For each candidate image pair, local feature matching and geometric filtering are performed to determine at least one target candidate image pair; Based on the similarity transformation results of each target candidate image pair, control point lap edges are constructed, including: For each candidate timestamp pair, the candidate images in the first image set corresponding to the candidate timestamp pair are matched with the candidate images in the second image set to determine at least one initial candidate image pair; At least one initial candidate image pair is filtered according to preset filtering conditions to determine at least one filtered candidate image pair; For each candidate image pair, local feature matching and geometric filtering are performed to determine at least one target candidate image pair; For each of the target candidate image pairs, a similarity transformation is performed on the two candidate images in the target candidate image pair to obtain the similarity transformation result corresponding to the target candidate image pair; Based on the similarity transformation results of each target candidate image pair, control point loop edges are constructed.

4. The image pose optimization method according to claim 3, characterized in that, The step of performing local feature matching and geometric filtering for each candidate image pair to determine at least one target candidate image pair includes: For each candidate image pair, local feature matching is performed on the candidate image pair to obtain the candidate image matching result; based on the candidate image matching result, filtering is performed according to preset geometric filtering conditions to determine at least one target candidate image pair; The geometric filtering conditions include at least one of the following: The following conditions must be met: the number of inliers in the candidate image is not less than the minimum inlier count threshold; the ratio of the number of inliers in the candidate image to the original number of matches is not less than the minimum inlier ratio threshold; the triangulation angle in the candidate image is not less than the minimum triangulation angle threshold; the configuration of the candidate image is not invalid; and the number of loop connections in the candidate image is less than the preset connection count threshold.

5. The image pose optimization method according to claim 3, characterized in that, The construction of control point lap edges based on the similarity transformation results of each target candidate image pair includes: For target candidate image pairs whose similarity transformation results are successful, construct complete relative pose constraint edges that include rotation, translation, and scaling; For target candidate image pairs whose similarity transformation results are unsuccessful, a unit translation direction constraint edge is constructed when the estimated translation direction is valid; Based on the complete relative pose constraint edges and / or unit translation direction constraint edges, construct control point loop edges.

6. The image pose optimization method according to claim 1, characterized in that, The optimization of the initial pose of the image to be optimized to obtain the optimized image pose includes: The number of odometer edges and loop closure edges in the image to be optimized are determined, and the weights of the odometer edges and loop closure edges are adjusted according to the ratio between the number of odometer edges and the number of loop closure edges. A switch variable is set for the control point loop closure edge, and the initial pose and the switch variable are used as optimization variables. The loop closure residual is adjusted based on the switch variable. The first stage of optimization is performed on the image to be optimized according to the adjusted odometry edge weight, the loop closure edge weight, the control point loop closure edge, the odometry edge, the ordinary visual loop closure edge, and the manual loop closure edge. The switch variable represents the confidence level of the loop closure edge. The credibility of each loop edge is determined based on the switch variable, and at least one loop edge with a credibility lower than a preset credibility threshold is filtered out. After restoring the image pose of the image to be optimized to the initial pose before the first stage of optimization, the second stage of optimization is performed on the image to be optimized based on the filtered loop closure edges and odometer edges according to the adjusted odometer edge weights and loop closure edge weights, to obtain the optimized image pose.

7. The image pose optimization method according to claim 1, characterized in that, The image pose optimization method further includes: Obtain the observation records of the first control point in the first image to be optimized and the observation records of the second control point in the second image to be optimized; The observation timestamps of the first image to be optimized and the second image to be optimized are grouped according to the control point labels of the first control point observation record and the second control point observation record. When the same target label exists in the first image to be optimized and the second image to be optimized, select the set of images within a preset time window corresponding to the observation time of the target label in the two first images to be optimized and the second image to be optimized respectively; Cross-image retrieval and feature matching are performed on the first image to be optimized and the selected image set of the second image to be optimized to establish cross-image loop edges; Based on the cross-image loop edges, the first image to be optimized and the second image to be optimized are fused.

8. An image pose optimization device based on control point loop closure detection, characterized in that, The image pose optimization device includes: The data loading module is used to acquire the initial pose of at least one image to be optimized and control point observation records; wherein, the control point observation records include at least control point labels and observation timestamps; The control point grouping module is used to combine the observation timestamps within the same control point label group based on the control point label to construct at least one candidate timestamp pair; The loop closure edge establishment module is used to, for each candidate timestamp pair, select the image set corresponding to each timestamp based on a preset time window, and generate a candidate image pair set by combining the determined image sets; for each candidate timestamp pair, filter out initial candidate image pairs with similarity matching; filter at least one initial candidate image pair according to preset filtering conditions to obtain at least one filtered candidate image pair; perform local feature matching and geometric filtering on each filtered candidate image pair to determine at least one target candidate image pair; and construct control point loop closure edges based on the similarity transformation results of each target candidate image pair. The image pose optimization module is used to add the control point loop edges, pre-constructed odometer edges, ordinary visual loop edges, and manual loop edges to the image to be optimized, and optimize the initial pose of the image to be optimized to obtain the optimized image pose.

9. An electronic device, characterized in that, include: The device includes a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the image pose optimization method based on control point loop closure detection as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the image pose optimization method based on control point loop closure detection as described in any one of claims 1 to 7.

Images