Relocation method, storage medium, electronic device, and program product

Abstract

The application provides a repositioning method, a storage medium, an electronic device and a program product, and relates to the technical field of positioning. The repositioning method comprises the following steps: if the pose of a movable device is lost, the first pose of each particle corresponding to the movable device and the pose corresponding to each key frame are determined; the second pose of each particle under a plurality of movement distances of the movable device is calculated based on the first pose of each particle; each second pose of each particle is taken as a query point, and a target pose most adjacent to the second pose is searched from the pose corresponding to each key frame; a target particle is determined from the plurality of particles based on the target pose of each particle; and the repositioning result of the movable device is determined based on the target particle. The application avoids the influence of noise under a single movement distance, guarantees the optimality and stability of the target particle, and can more accurately estimate the pose of the movable device.

Description

Technical Field

[0001] This application relates to the field of positioning technology, specifically to a relocation method, storage medium, electronic device, and program product. Background Technology

[0002] Mobile devices or vehicles may lose their pose during operation, requiring relocalization. However, existing relocalization methods mainly rely on feature matching based on single-frame information or geometric optimization strategies, failing to fully utilize multi-frame information. They are also susceptible to complex conditions such as dynamic environments, lighting changes, and scene occlusion, resulting in low robustness of feature matching and difficulty in ensuring stable relocalization results. Summary of the Invention

[0003] In view of this, embodiments of this application provide a relocation method, a storage medium, an electronic device, and a program product.

[0004] In a first aspect, one embodiment of this application provides a relocation method, comprising: if the pose of a mobile device is lost, determining the first pose of each of a plurality of particles corresponding to the mobile device and the pose of each of a plurality of keyframes; calculating the second pose of each of the plurality of particles at a plurality of moving distances of the mobile device based on the first pose of each of the plurality of particles; using each second pose of each particle as a query point, searching for the target pose closest to the second pose in the poses of each of the plurality of keyframes; determining the target particle from the plurality of particles based on the target pose of each of the plurality of particles; and determining the relocation result of the mobile device based on the target particle.

[0005] In conjunction with the first aspect, in some implementations of the first aspect, determining the target particle from multiple particles based on their respective target poses includes: determining the target frame acquired by the mobile device in each second pose; performing feature matching between the target frame of each particle and the key frame corresponding to the target pose to obtain the particle matching result; and determining the target particle from multiple particles based on their respective matching results.

[0006] In conjunction with the first aspect, in some implementations of the first aspect, the particle matching result is represented by a weight value. Based on the matching results of multiple particles, the target particle is determined from multiple particles, including: based on the weight values ​​of multiple particles, determining at least one candidate particle whose weight value satisfies the target condition; when the sum of multiple movement distances is greater than the target distance, if there is a unique candidate particle whose weight value is increased, and the weight value of the candidate particle is greater than the first target weight value, then the candidate particle is determined as the target particle, wherein the increase in weight value indicates a successful match.

[0007] In conjunction with the first aspect, in some implementations of the first aspect, the target conditions include: the weight value is greater than the second target weight value, the second target weight value is greater than the third target weight value and less than the first target weight value; and / or, the weight value is the largest.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes at least one of the following: if the matching result indicates a successful match, then the second pose of the successfully matched particle is updated to the target pose; if the matching result indicates a failed match, then the number of times the second pose of the unsuccessfully matched particle has not been updated is recorded, and the particle is removed if the number of times it has not been updated reaches the target number; if the matching result indicates that multiple particles have failed to match, then at least one candidate keyframe is determined based on the global descriptor of the current frame and the global descriptors corresponding to the multiple keyframes, and a new particle is created based on the relative pose between the at least one candidate keyframe and the current frame.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, the particle matching result is represented based on weight values. The method further includes: if the distance between any particle and the second pose of other particles is less than the target distance, and the weight value of the particle is less than the weight values ​​of other particles, then the particle is removed.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, before determining the first pose of each of the multiple particles corresponding to the mobile device and the pose of each of the multiple keyframes, the method further includes: if the number of particles corresponding to the mobile device is less than the target number, then based on the global descriptor of the current frame and the global descriptor of each of the multiple keyframes, at least one candidate keyframe is determined; and based on the relative pose between the at least one candidate keyframe and the current frame, a new particle is created.

[0011] Secondly, one embodiment of this application provides a relocation device, comprising: a first determining module, configured to determine the first pose of each of a plurality of particles corresponding to the mobile device and the pose of each of a plurality of keyframes if the pose of the mobile device is lost; a calculation module, configured to calculate the second pose of each of the plurality of particles at multiple moving distances of the mobile device based on the first pose of each of the plurality of particles; a search module, configured to search for the target pose closest to the second pose in the poses corresponding to each of the plurality of keyframes, using each second pose of each particle as a query point; a second determining module, configured to determine the target particle from the plurality of particles based on the target pose of each of the plurality of particles; and a third determining module, configured to determine the relocation result of the mobile device based on the target particle.

[0012] Thirdly, one embodiment of this application provides a computer-readable storage medium storing a computer program for performing the relocation method described in the first aspect.

[0013] Fourthly, one embodiment of this application provides an electronic device, the electronic device comprising: a processor; a memory for storing processor-executable instructions; the processor being configured to perform the relocation method described in the first aspect.

[0014] Fifthly, one embodiment of this application provides a computer program product including instructions that, when executed on an electronic device, cause the electronic device to implement the relocation method described in the first aspect.

[0015] In this application, when the pose of the mobile device is lost, multiple particles are generated for the mobile device and assigned an initial pose (first pose), while using the pose information of keyframes as a reference. Next, the new pose of the particles (second pose) is predicted based on the movement distance of the mobile device, and this second pose is used as a query point to search for the nearest target pose in the poses of the keyframes. By evaluating the target pose of the particles, the optimal target particle is determined, and finally, the relocalization result of the device is determined based on the target pose of this particle. This scheme fully considers the continuity and dynamic changes of the mobile device's motion. Through comprehensive consideration of multiple movement distances, it avoids the noise influence of a single movement distance, ensuring the optimality and stability of the target particle, and enabling more accurate estimation of the mobile device's pose. Furthermore, by matching across multiple movement distances, it can better adapt to scene changes and dynamic environmental changes, thereby improving the adaptability of the method. Attached Figure Description

[0016] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0017] Figure 1 The diagram shown is a flowchart of a relocation method provided in an embodiment of this application.

[0018] Figure 2 The diagram shown is a flowchart illustrating the process of determining a target particle according to an embodiment of this application.

[0019] Figure 3 The diagram shown is a structural schematic of a repositioning device provided in an embodiment of this application.

[0020] Figure 4The diagram shown is a structural schematic of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] In applications such as mobile devices, autonomous driving, and augmented reality, accurate relocalization capabilities are crucial. Current Simultaneous Localization and Mapping (SLAM) methods largely rely on feature matching or geometric optimization strategies based on single-frame information. For example, vision-based loop closure detection methods often use local features to match the current frame with historical frames to achieve relocalization. However, because these methods depend on single-frame information, their feature matching robustness is poor when facing complex conditions such as dynamic environments, lighting changes, and scene occlusion. This can easily lead to matching failures or mismatches, resulting in localization errors or even system failure.

[0023] In recent years, deep learning and multi-frame fusion techniques have brought new directions to SLAM relocalization, but the applied relocalization methods still have many shortcomings:

[0024] First, relying on single-frame information for matching makes it susceptible to noise, dynamic occlusion, and changes in lighting, resulting in a high probability of matching failure or mismatch.

[0025] Second, the temporal consistency of multi-frame information is not fully utilized, making the matching results susceptible to noise interference from single-frame information, and making it difficult to ensure stable relocation results.

[0026] Third, in large-scale environments or similar scenarios, the matching method for single-frame information is prone to incorrect matching, which leads to incorrect pose estimation and affects the overall accuracy of relocalization.

[0027] Fourth, when faced with long-term scene changes (such as seasonal and lighting changes) or dynamic environmental changes (such as pedestrian and vehicle occlusion), existing relocation methods have poor matching stability and weakened robustness.

[0028] In summary, existing relocation methods have many shortcomings in terms of accuracy, real-time performance, and robustness. Based on this, this application is proposed. The relocation method in this application utilizes the second pose of particles at multiple movement distances to improve the stability and accuracy of matching, thereby enhancing the overall performance of relocation.

[0029] Figure 1The diagram shown is a flowchart illustrating a relocation method provided in an embodiment of this application. Figure 1 As shown, the method includes the following steps.

[0030] Step S110: If the pose of the mobile device is lost, determine the first pose of each of the multiple particles corresponding to the mobile device, and the pose of each of the multiple keyframes.

[0031] Keyframes are representative frames selected during SLAM mapping. These frames have sufficient feature information, and there are sufficient pose or angle differences between keyframes.

[0032] The pose corresponding to a keyframe represents its position and orientation in the global coordinate system of the map, which contains the area where the mobile device lost its pose. Specifically, during map construction, keyframes are selected from the video stream or image sequence according to certain rules (such as frame interval, inter-frame angle changes, etc.). For each keyframe, its pose, i.e., its position and orientation in the map, is calculated and recorded. In one implementation, for convenient querying, a KDTree can be constructed. This is a data structure that enables fast retrieval of keyframe poses and is used to store the poses of keyframes.

[0033] In situations involving signal interference, sensor malfunction, or dynamic environmental changes, mobile devices may experience pose loss, meaning they cannot determine their precise location within the environment. In such cases, multiple particles are generated, with each particle's first pose representing a pose estimate of the mobile device. Optionally, the first pose of a particle is obtained by retrieving keyframes that match the pose loss scenario and calculating them using perspective-n-point or iterative nearest-point methods. Then, the poses of the keyframes saved during mapping are retrieved.

[0034] For example, consider an unmanned forklift in a warehouse environment. Suppose the forklift loses its pose due to signal interference while moving between shelves. Multiple particles will be generated to represent the possible positions of the forklift. If a map has been previously built for this area, the poses of keyframes (such as feature points on shelves and aisles) will be further extracted.

[0035] Step S120: Based on the first pose of each of the multiple particles, calculate the second pose of each of the multiple particles at multiple moving distances of the mobile device.

[0036] Optionally, based on the relative pose between the previous and current moving distances and the particle's pose at the previous moving distance, a second pose of the particle at the current moving distance is calculated. The particle's pose at the previous moving distance is derived from its first pose; the relative pose of the mobile device between two adjacent moving distances can be calculated based on sensor data from the mobile device (such as wheel speedometer).

[0037] For example, a mobile device has five movement distances, denoted as movement distance 1, movement distance 2, movement distance 3, movement distance 4, and movement distance 5, respectively. The first pose of the particle is denoted as movement distance 0. Then, the second pose of the particle at movement distance 1 is equal to the first pose of the particle multiplied by the relative pose between movement distance 0 and movement distance 1; the second pose of the particle at movement distance 2 is equal to the second pose of the particle at movement distance 1 multiplied by the relative pose between movement distance 1 and movement distance 2; the second pose of the particle at movement distance 3 is equal to the second pose of the particle at movement distance 2 multiplied by the relative pose between movement distance 2 and movement distance 3, and so on, until the second pose of the particle at movement distance 5 is obtained.

[0038] Step S130: Using each second pose of each particle as a query point, search for the target pose that is closest to the second pose in the poses corresponding to each of the multiple keyframes.

[0039] The role of the query point is to find the nearest target pose within the keyframe poses. For example, suppose the particle's second pose is (x = 5, y = 3, θ = 45°), this pose is the query point. Then, the keyframe poses are searched for the nearest keyframe pose to this query point. For example, the pose of keyframe C is (x = 5.1, y = 3.0, θ = 45°), which has the smallest distance to the query point and is therefore selected as the target pose. This target pose will be used for subsequent pose verification and updates to determine the particle's optimal pose.

[0040] Step S140: Based on the target poses of the multiple particles, determine the target particle from the multiple particles.

[0041] Optionally, for each movement distance, the distance (e.g., Euclidean distance) between the particle's target pose and the particle's second pose is calculated. If this distance is less than a preset distance threshold, the particle is considered to have successfully matched the target pose at that movement distance. The particle with the most successful matches is then selected as the target particle.

[0042] For example, suppose there are particles A, B, and C. During a certain relocalization process, particle A's distance from the target pose to the second pose is less than the distance threshold in all five movement distances, resulting in 5 successful matches; particle B's distance is less than the distance threshold in three movement distances, resulting in 3 successful matches; and particle C's distance is less than the distance threshold in one movement distance, resulting in 1 successful match. Therefore, particle A has the most successful matches and is selected as the target particle.

[0043] Step S150: Based on the target particle, determine the relocation result of the mobile device.

[0044] Optionally, after determining the target particle, the second pose of the target particle is updated using the target pose found at the current moving distance, and the updated pose is determined as the relocalization result of the mobile device.

[0045] For example, target particle A is selected from multiple particles. The target pose of target particle A is (x = 5.0, y = 3.0, θ = 45°). The target pose of target particle A (x = 5.0, y = 3.0, θ = 45°) is...

[0046] The second pose of particle A is updated to (x = 5.0, y = 3.0, θ = 45°). Finally, the updated second pose (x = 5.0, y = 3.0, θ = 45°) is used as the relocalization result for the mobile device.

[0047] In this embodiment, when the pose of the mobile device is lost, multiple particles are generated for the mobile device and assigned an initial pose (first pose), while the pose information of the keyframe is used as a reference. Next, the new pose of the particles (second pose) is predicted based on the movement distance of the mobile device, and this second pose is used as a query point to search for the nearest target pose in the poses of the keyframe. By evaluating the target pose of the particles, the optimal target particle is determined, and finally, the relocalization result of the device is determined based on the target pose of this particle. This scheme fully considers the continuity and dynamic changes of the mobile device's motion. Through comprehensive consideration of multiple movement distances, it avoids the noise influence of a single movement distance, ensuring the optimality and stability of the target particle, and enabling more accurate estimation of the mobile device's pose. Furthermore, by matching across multiple movement distances, it can better adapt to scene changes and dynamic environmental changes, thereby improving the adaptability of the method.

[0048] Figure 2 The diagram shown is a flowchart illustrating the process of determining a target particle according to an embodiment of this application. Figure 1 Based on the illustrated embodiment, the following extensions are made: Figure 2 The illustrated embodiment will be described in detail below. Figure 2 The illustrated embodiments and Figure 1The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0049] like Figure 2 As shown, in this embodiment, the target particle is determined from the multiple particles based on their respective target poses, including the following steps.

[0050] Step S210: Determine the target frame acquired by the mobile device in each second pose.

[0051] The target frame acquired by the mobile device in the second pose represents the environmental data collected by the mobile device's sensors (such as cameras, lidar, etc.) when the mobile device is in the second pose of a certain particle.

[0052] For example, suppose a mobile device is navigating in an indoor environment, and particle A's second pose is (x = 5.1, y = 3.0, θ = 45°). When the mobile device is in this pose, its camera captures an image, which is the target frame in that pose.

[0053] Step S220: Perform feature matching between the target frame and the key frame corresponding to the target pose of each particle to obtain the particle matching result.

[0054] Optionally, if the target frame and keyframe are point cloud data, the nearest point in the keyframe is found for each point in the target frame, and then the rigid transformation (translation and rotation) that minimizes the distance between the two point sets is calculated. This process is iterated until a convergence condition (such as an error threshold or a maximum number of iterations) is met, thus obtaining the pose transformation relationship (i.e., error) between the target frame and the keyframe. If the final error is within the threshold range, the particle matching result is considered successful; otherwise, it is considered a failure.

[0055] Optionally, if the target frame is image data and the keyframe is point cloud data, the camera pose of the target frame is calculated using 2D feature points in the target frame and the corresponding 3D feature points in the keyframe. By solving a nonlinear optimization problem, the camera pose parameters (rotation matrix and translation vector) that minimize the relationship error between the 2D and 3D feature points are obtained. Similarly, if the calculated error is within a threshold range, the particle matching result is considered successful; otherwise, it is considered a failure.

[0056] These two algorithms can effectively evaluate the matching degree between the target frame and the key frame of a particle, and obtain the matching result for each particle.

[0057] Step S230: Based on the matching results of multiple particles, determine the target particle from the multiple particles.

[0058] Specifically, based on the matching results of multiple particles, the optimal target particle is determined from the multiple particles. Optionally, the screening method in step S130 can be used to determine the target particle from the multiple particles.

[0059] In this embodiment, the target frame for each particle in its second pose is first determined, providing a data foundation for feature matching. Next, feature matching is performed between the target frame and its corresponding keyframe to quantify the similarity between the particle and the keyframe. Finally, the optimal target particle is determined from multiple particles based on the matching results. This process fully considers multi-particle information, reduces the impact of single-particle errors, and makes the relocalization results more accurate. Furthermore, the feature matching algorithm is used to quickly and accurately evaluate the particle matching degree, improving relocalization efficiency.

[0060] Figure 2 This embodiment provides a method for determining a target particle from multiple particles. It is understood that particle matching results are a crucial basis for selecting target particles, with different matching results indicating variations in the fit and relevance of particles under specific matching rules. To more accurately determine the target particle, this embodiment analyzes these matching results in depth and provides an intuitive and effective method for selecting target particles using these results. Specifically, the particle matching results are represented by weight values. Further, based on the respective weight values ​​of multiple particles, at least one candidate particle whose weight value satisfies the target condition is determined. If the sum of multiple movement distances is greater than the target distance, and there exists a unique candidate particle whose weight value is increased, and the candidate particle's weight value is greater than the first target weight value, then the candidate particle is determined as the target particle. The increase in weight value indicates a successful match.

[0061] The weight value represents the probability that a particle is the optimal particle. The weight value increases upon a successful match, reflecting the degree of matching between the particle and the current environment. The target condition is the criterion used to determine candidate particles. Particles that meet this condition can be considered candidate particles for further selection of target particles. Optionally, the target condition includes: a weight value greater than a second target weight value; a second target weight value greater than a third target weight value and less than a first target weight value; and / or, a maximum weight value.

[0062] On the one hand, using a weight value greater than the second target weight value as one of the conditions is to initially filter out particles with a high probability of matching. The second target weight value, falling between the third and first target weight values, indicates that the particle's matching degree is at a slightly above-average level. This helps to quickly identify particles with relatively good matching effects among many particles, narrowing the selection range. On the other hand, using the maximum weight value as the target condition is to directly determine the optimal particle, ensuring that the selected target particle has the best matching result among all current particles. Both of these screening mechanisms can effectively reduce the probability of false matches while maintaining accuracy, improving the overall performance and stability of relocation.

[0063] The condition that the sum of multiple movement distances exceeds the target distance is designed to ensure that the particle undergoes a sufficient number of matching verifications during the relocation process. Each movement distance corresponds to one matching attempt; only when the particle's cumulative movement distance reaches the target distance can it be guaranteed that it has undergone enough matching processes. This makes the particle's matching results more reliable.

[0064] When the sum of multiple movement distances is greater than the target distance, if there is a unique candidate particle whose weight value is increased and its weight value is greater than the first target weight value, then the particle can be considered to have shown high stability and reliability in multiple matching processes, and thus it can be identified as the target particle, thereby realizing the relocation of the mobile device.

[0065] For example, suppose a mobile device is relocalizing in a mapped environment, and five particles are currently involved in the relocalization process. In a certain iteration, the particle weights are [0.8, 0.6, 0.9, 0.7, 0.85], the target condition is a weight greater than 0.8, and the first target weight is set to 0.83. At this time, the candidate particles that meet the target condition are particle 1, particle 3, and particle 5. Assuming that the sum of the distances of the multiple movement distances is greater than the target distance, among these candidate particles, if only the weight of particle 3 is increased (indicating a successful match), and its weight of 0.9 is greater than the first target weight of 0.83, then particle 3 is determined as the target particle, and its pose is the current relocalization result of the mobile device.

[0066] In this embodiment, the matching status of each particle is quantified using weight values. Then, by determining at least one candidate particle that meets the target conditions based on the weight values ​​of multiple particles, particles with a high matching probability can be initially screened, narrowing the scope of further analysis. When the sum of multiple movement distances is greater than the target distance, it is further required that there is a unique candidate particle whose weight value is increased, and that the weight value of this particle is greater than the first target weight value. This setting ensures that only the particle that has undergone multiple matching verifications and has the best matching effect can be identified as the target particle, thereby improving the accuracy and reliability of relocation.

[0067] In some embodiments, if the matching result indicates a successful match, the second pose of the successfully matched particle is updated to the target pose.

[0068] This mechanism ensures that the particle's second pose can be corrected and updated in a timely manner upon successful matching, improving the accuracy of relocalization. Simultaneously, the updated particle pose can serve as the basis for subsequent iterations, further enhancing the stability of the matching. This dynamic update mechanism makes the method more adaptable to environmental changes, thereby improving the overall performance and reliability of the relocalization method.

[0069] In some embodiments, if the matching result indicates that the matching is unsuccessful, the number of times the second pose of the unmatched particle has not been updated is recorded, and the particle is removed if the number of times it has not been updated reaches the target number.

[0070] Understandably, when a particle's matching result indicates an unsuccessful match, it means that the particle's current pose is inconsistent with the actual environmental features. To ensure the accuracy and efficiency of relocalization, the number of times the particle has not been updated is recorded. Each time a match fails, the number of times it has not been updated is incremented. When the number of times it has not been updated reaches a preset target number, it is considered that the particle's pose has been inaccurate for a long time and may have a negative impact on subsequent relocalization processes, so it is removed.

[0071] In this scheme, removing unreliable particles avoids the cumulative effect of mismatches, helps to eliminate unreliable particles in a timely manner, prevents them from wasting computational resources, and ensures that computational resources can be concentrated on particles that are more likely to succeed. Furthermore, this scheme can dynamically adjust the number of particles, ensuring that only the most likely particles are processed in each iteration, thereby accelerating convergence to the correct pose and improving the relocalization success rate.

[0072] In some embodiments, if the distance between any particle and the second pose of other particles is less than the target distance, and the weight value of any particle is less than the weight values ​​of other particles, then the particle is removed.

[0073] For example, three particles A, B, and C have second poses p respectively.A p B p C The weight values ​​are W A =0.6, W B =0.8, W C =0.7. The preset target distance is d. After calculation, it was found that p A With p B The distance is less than d, and W A Less than W B Therefore, particle A will be removed because it is too close to particle B and has a lower weight.

[0074] This scheme aims to optimize the particle set, remove redundant or unlikely particles, and make the remaining particles more unique and accurate, thereby improving relocation efficiency and accuracy.

[0075] In some embodiments, if the matching result indicates that multiple particles fail to match, then based on the global descriptor of the current frame and the global descriptors corresponding to each of the multiple key frames, at least one candidate key frame is determined, and a new particle is created based on the relative pose between the at least one candidate key frame and the current frame.

[0076] Alternatively, before determining the first pose of each of the multiple particles corresponding to the mobile device and the pose of each of the multiple keyframes, if the number of particles corresponding to the mobile device is less than the target number, then at least one candidate keyframe is determined based on the global descriptor of the current frame and the global descriptor of each of the multiple keyframes; and new particles are created based on the relative pose between the at least one candidate keyframe and the current frame.

[0077] The current frame refers to the latest data frame collected when multiple particles fail to match and a new particle is planned to be constructed; or, it is the latest data frame collected when the number of particles is less than the target number.

[0078] Each keyframe corresponds to a global descriptor, which represents the feature vector extracted from the keyframe and is used to describe the global features of the keyframe. Optionally, during graph construction, the global descriptor of each keyframe is stored in a KDTree, a data structure that supports efficient nearest neighbor search, which can quickly find the most similar keyframe based on the global descriptor of the current frame.

[0079] For example, the similarity between the global descriptor of the current frame and the global descriptor of the keyframes is calculated, and then the keyframe with the highest similarity is selected as a candidate keyframe. Then, when both the current frame and the candidate keyframes are point cloud data, the nearest point correspondence between the current frame and the candidate keyframes is found iteratively, and the distance between these corresponding points is minimized, thereby calculating the relative pose of the current frame and subsequent keyframes. If the current frame is image data and the candidate keyframes are point cloud data, the relative pose of the current frame and the candidate keyframes is calculated using the known positions of 3D points in the point cloud data in the global coordinate system and their 2D projections in the image coordinate system.

[0080] Furthermore, this relative pose is applied to the absolute pose of the candidate keyframes to obtain the pose of the new particle. The pose of the new particle is the possible pose hypothesis for the current frame. In addition, other attributes of the new particle, such as initial weights and the number of times it has not been updated, are initialized. In this way, the new particle constructed based on the relative pose is added to the particle set, providing more possibilities for subsequent relocalization calculations.

[0081] Furthermore, the number of new particles can be determined based on the target number of particles required for relocalization. For example, if the target number is 10, then 10 new particles need to be created if all particles fail to match. For instance, if there are 8 particles before determining the first pose of each particle and the pose of each keyframe, then 2 particles need to be reconstructed to reach the target number of 10.

[0082] This embodiment aims to ensure a sufficient and diverse number of particles during the relocation process. When all particle matching fails, a candidate keyframe most similar to the current frame is found through global descriptor matching, and new particles are created based on this candidate keyframe. Similarly, if the initial number of particles is insufficient for the target number, particles are supplemented in the same way. This mechanism ensures particle diversity and coverage, increasing the success rate of relocation.

[0083] The above text combined Figure 1 and Figure 2 The relocation method embodiments of this application are described in detail below, in conjunction with... Figure 3 This application provides a detailed description of embodiments of the repositioning device. It should be understood that the descriptions of the repositioning method embodiments correspond to the descriptions of the repositioning device embodiments; therefore, any parts not described in detail can be found in the preceding method embodiments.

[0084] Figure 3 The diagram shown is a structural schematic of a repositioning device provided in an embodiment of this application. Figure 3 As shown, the repositioning device 30 provided in this embodiment includes:

[0085] The first determining module 310 is used to determine the first pose of each of the multiple particles corresponding to the mobile device and the pose of each of the multiple keyframes if the pose of the mobile device is lost.

[0086] The calculation module 320 is used to calculate the second pose of each of the multiple particles at multiple moving distances of the mobile device, based on the first pose of each of the multiple particles.

[0087] Search module 330 is used to search for the target pose that is closest to the second pose in the poses corresponding to each of the multiple keyframes, with each second pose of each particle as the query point.

[0088] The second determining module 340 is used to determine the target particle from multiple particles based on the target poses of each particle.

[0089] The third determining module 350 is used to determine the repositioning result of the mobile device based on the target particle.

[0090] In one embodiment of this application, the second determining module 340 is further configured to: determine the target frame acquired by the mobile device in each second pose; perform feature matching between the target frame of each particle and the key frame corresponding to the target pose to obtain the matching result of the particle; and determine the target particle from the multiple particles based on the matching results of the multiple particles.

[0091] In one embodiment of this application, the particle matching result is represented based on weight values. The second determining module 340 is further configured to determine at least one candidate particle whose weight value satisfies the target condition based on the weight values ​​of the multiple particles. If the sum of the distances of the multiple moving distances is greater than the target distance, and there is a unique candidate particle whose weight value is increased, and the weight value of the candidate particle is greater than the first target weight value, then the candidate particle is determined as the target particle, wherein the increase in weight value indicates a successful match.

[0092] In one embodiment of this application, the second determining module 340 is further configured to include the following target conditions: the weight value is greater than the second target weight value, the second target weight value is greater than the third target weight value and less than the first target weight value; and / or, the weight value is the maximum.

[0093] In one embodiment of this application, the second determining module 340 is further configured to: if the matching result indicates a successful match, update the second pose of the successfully matched particle to the target pose; if the matching result indicates a failed match, record the number of times the second pose of the unsuccessfully matched particle has not been updated, and remove the particle if the number of times it has not been updated reaches the target number; if the matching result indicates that multiple particles have failed to match, determine at least one candidate keyframe based on the global descriptor of the current frame and the global descriptors corresponding to each of the multiple keyframes, and create a new particle based on the relative pose between the at least one candidate keyframe and the current frame.

[0094] In one embodiment of this application, the second determining module 340 is further configured to remove a particle if the distance between the second pose of any particle and other particles is less than the target distance and the weight value of the particle is less than the weight value of other particles.

[0095] In one embodiment of this application, the first determining module 310 is further configured to: if the number of particles corresponding to the mobile device is less than the target number, determine at least one candidate key frame based on the global descriptor of the current frame and the global descriptors corresponding to each of the multiple key frames; and create new particles based on the relative pose between the at least one candidate key frame and the current frame.

[0096] Below, for reference Figure 4 This describes an electronic device according to embodiments of the present application. Figure 4 The diagram shown is a structural schematic of an electronic device provided in an exemplary embodiment of this application.

[0097] like Figure 4 As shown, the electronic device 40 includes one or more processors 401 and memory 402.

[0098] The processor 401 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 40 to perform desired functions.

[0099] The memory 402 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 401 may execute the program instructions to implement the relocation methods of the various embodiments of this application described above and / or other desired functions. The computer-readable storage medium may also store various contents such as the first pose of multiple particles, the pose of keyframes, the second pose of multiple particles, etc.

[0100] In one example, the electronic device 40 may also include an input device 403 and an output device 404, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0101] The input device 403 may include, for example, a keyboard, a mouse, etc.

[0102] The output device 404 can output various information to the outside, including the first pose of multiple particles, the pose of keyframes, the second pose of multiple particles, etc. The output device 404 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0103] Of course, for the sake of simplicity, Figure 4 Only some of the components of the electronic device 40 relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device 40 may include any other suitable components depending on the specific application.

[0104] In addition to the methods and apparatus described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the relocation methods according to various embodiments of this application described above.

[0105] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0106] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the relocation methods according to various embodiments of this application described above.

[0107] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0108] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0109] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0110] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0111] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0112] The above description has been given for illustrative and descriptive purposes and is not intended to limit the embodiments of this application to the forms disclosed herein. Although several exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A relocation method, characterized in that, include: If the pose of the mobile device is lost, then the first pose of each of the multiple particles corresponding to the mobile device and the pose of each of the multiple keyframes are determined. Based on the first pose of each of the plurality of particles, calculate the second pose of each of the plurality of particles at multiple moving distances of the mobile device; Using each of the second poses of each particle as a query point, search for the target pose that is closest to the second pose in the poses corresponding to each of the plurality of keyframes; Based on the target poses of the plurality of particles, the target particle is determined from the plurality of particles; Based on the target particle, the relocation result of the mobile device is determined.

2. The relocation method according to claim 1, characterized in that, The step of determining the target particle from the plurality of particles based on their respective target poses includes: The target frame acquired by the mobile device in each second pose is determined; The target frame of each particle and the key frame corresponding to the target pose are matched by feature matching to obtain the matching result of the particle. Based on the matching results of the plurality of particles, the target particle is determined from the plurality of particles.

3. The relocation method according to claim 2, characterized in that, The matching result of the particles is represented based on weight values, and the step of determining the target particle from the plurality of particles based on the matching results of each of the plurality of particles includes: Based on the weight values ​​of the plurality of particles, at least one candidate particle whose weight value satisfies the target condition is determined. If the sum of the multiple movement distances is greater than the target distance, and there exists a unique candidate particle whose weight value is increased, and the weight value of the candidate particle is greater than the first target weight value, then the candidate particle is determined as the target particle, wherein the increase in weight value indicates a successful match.

4. The relocation method according to claim 3, characterized in that, The target conditions include: The weight value is greater than the second target weight value, the second target weight value is greater than the third target weight value, and less than the first target weight value; and / or, the weight value is the largest.

5. The relocation method according to claim 2, characterized in that, It also includes at least one of the following: If the matching result indicates a successful match, then the second pose of the successfully matched particle is updated to the target pose; If the matching result indicates that the matching is unsuccessful, the number of times the second pose of the unmatched particle has not been updated is recorded, and if the number of times the second pose has not been updated reaches the target number, the particle is removed. If the matching result indicates that all of the multiple particles fail to match, then based on the global descriptor of the current frame and the global descriptors corresponding to each of the multiple keyframes, at least one candidate keyframe is determined, and a new particle is created based on the relative pose between the at least one candidate keyframe and the current frame.

6. The relocation method according to claim 2, characterized in that, The matching result of the particles is represented based on weight values, and the method further includes: If the distance between any particle and the second pose of other particles is less than the target distance, and the weight value of the particle is less than the weight values ​​of other particles, then the particle is removed.

7. The relocation method according to claim 1, characterized in that, Before determining the first pose of each of the multiple particles corresponding to the mobile device, and the pose of each of the multiple keyframes, the method further includes: If the number of particles corresponding to the mobile device is less than the target number, then at least one candidate keyframe is determined based on the global descriptor of the current frame and the global descriptors corresponding to each of the multiple keyframes. Create new particles based on the relative pose between the at least one candidate keyframe and the current frame.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for executing the relocation method according to any one of claims 1 to 7.

9. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the relocation method according to any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes instructions that, when executed on an electronic device, cause the electronic device to perform the relocation method according to any one of claims 1 to 7.

Images