Positioning method, storage medium, electronic device, and program product

Abstract

The application provides a positioning method, a storage medium, an electronic device and a program product, and relates to the technical field of positioning. The positioning method comprises the following steps: determining a plurality of reference frames in an environment where a mobile device is located, and a pose corresponding to each reference frame; acquiring an image frame collected by the mobile device at a current time; determining a plurality of target frames matched with features of the image frame from the plurality of reference frames; determining a spatial coordinate of a key point based on a pixel coordinate of the key point in each target frame and the pose corresponding to each target frame; and determining a pose of the mobile device at the current time based on the spatial coordinate of the key point and the pixel coordinate of the key point in the image frame. The application calculates the spatial coordinate of the key point by using the known poses of the plurality of target frames and the observation information thereof, instead of single-frame prediction, thereby fundamentally avoiding systematic deviation and improving the accuracy and reliability of positioning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of positioning, in particular to a positioning method, a storage medium, an electronic device and a program product. BACKGROUND

[0002] Traditional positioning methods usually index the current frame features with a bag-of-words map to match the map so as to find the most co-visible old frames, and finally obtain the pose. Such a method is susceptible to repeated textures and scale errors.

[0003] Alternatively, the most similar old frame is retrieved by using a global feature vector of deep learning, and the pose is solved by using the depth information saved by the old frame. However, the accuracy is still affected by the front-end depth deviation, and it is difficult to balance robustness and high accuracy. SUMMARY

[0004] In view of this, the embodiments of the present application provide a positioning method, a storage medium, an electronic device and a program product.

[0005] In a first aspect, an embodiment of the present application provides a positioning method, comprising: determining a plurality of reference frames in an environment where a mobile device is located, and a pose corresponding to each reference frame; acquiring an image frame collected by the mobile device at a current time; determining a plurality of target frames matched with features of the image frame from the plurality of reference frames; determining a spatial coordinate of a key point based on a pixel coordinate of the key point in each target frame and the pose corresponding to each target frame; and determining a pose of the mobile device at the current time based on the spatial coordinate of the key point and a pixel coordinate of the key point in the image frame.

[0006] In combination with the first aspect, in some implementations of the first aspect, determining a plurality of target frames matched with features of the image frame from the plurality of reference frames comprises: determining a plurality of reference frames matched with global features of the image frame from the plurality of reference frames; and determining a plurality of target frames matched with local features of the image frame from the plurality of reference frames matched with the global features of the image frame.

[0007] In combination with the first aspect, in some implementations of the first aspect, determining a plurality of reference frames matched with global features of the image frame from the plurality of reference frames comprises: determining a global descriptor corresponding to each of the plurality of reference frames and a global descriptor of the image frame; calculating a similarity between the global descriptor of the image frame and the global descriptor of each reference frame; and determining a reference frame with a similarity higher than a preset threshold as a reference frame matched with the global features of the image frame; and / or determining a plurality of target frames matched with local features of the image frame from the plurality of reference frames matched with the global features of the image frame comprises: matching the image frame and the plurality of reference frames matched with the global features of the image frame based on a feature descriptor of the local features of the image frame to obtain the plurality of target frames.

[0008] With reference to the first aspect, in some implementations of the first aspect, the local features of the image frame include key points in the image frame.

[0009] With reference to the first aspect, in some implementations of the first aspect, the key points in the image frame refer to feature points that are successfully matched in at least two reference frames in the matching process of the image frame and the plurality of reference frames.

[0010] With reference to the first aspect, in some implementations of the first aspect, determining the spatial coordinates of the key points based on the pixel coordinates of the key points in each target frame and the pose corresponding to each target frame comprises: determining a projection center of each target frame based on the pose corresponding to each target frame; determining a ray from the projection center of the target frame to the pixel coordinates of the key points in the target frame and a converging position of the plurality of rays, with the projection center of each target frame as a starting point; and determining the spatial coordinates of the key points based on the converging position of the plurality of rays.

[0011] With reference to the first aspect, in some implementations of the first aspect, determining the pose of the mobile device at the current time based on the spatial coordinates of the key points and the pixel coordinates of the key points in the image frame comprises: constructing an optimization function based on the spatial coordinates of the key points and the pixel coordinates of the key points in the image frame; solving the optimization function to obtain an optimal pose, the optimal pose minimizing the error between the spatial coordinates of the key points and the pixel coordinates of the key points in the image frame; and determining the optimal pose as the pose of the mobile device at the current time.

[0012] With reference to the first aspect, in some implementations of the first aspect, the current time includes a time when the mobile device is started; and / or, the current time includes a time when the mobile device loses the pose.

[0013] The second aspect provides a positioning device, comprising: a first determining module configured to determine a plurality of reference frames in an environment where a mobile device is located and a pose corresponding to each reference frame; an obtaining module configured to obtain an image frame collected by the mobile device at a current time; a second determining module configured to determine a plurality of target frames matched with features of the image frame from the plurality of reference frames; a third determining module configured to determine spatial coordinates of key points in the image frame based on pixel coordinates of the key points in each target frame and the pose corresponding to each target frame; and a fourth determining module configured to determine a pose of the mobile device at the current time based on the spatial coordinates of the key points and the pixel coordinates of the key points in the image frame.

[0014] In a third aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program for executing the positioning method in the first aspect.

[0015] In a fourth aspect, an embodiment of the present application provides an electronic device, which comprises: a processor; a memory for storing processor-executable instructions; and the processor is configured to execute the positioning method in the first aspect.

[0016] In a fifth aspect, an embodiment of the present application provides a computer program product, which comprises instructions for causing an electronic device to implement the positioning method in the first aspect when the instructions are executed on the electronic device.

[0017] The scheme in the present application discards the rough matching strategy relying on visual word statistical similarity, and screens a plurality of target frames from the reference frame by directly performing feature matching, thereby effectively reducing the matching ambiguity caused by repeated structures. More importantly, the present application does not use the depth prediction value with inherent generalization error, but calculates the spatial coordinates of the key points by using the known poses of the plurality of target frames and the observation information thereof. This process makes the spatial coordinates of the key points directly derived from high-precision geometric constraints, rather than single-frame prediction, thereby fundamentally avoiding systematic bias. Finally, by using these high-confidence spatial coordinates and their projections in the image frame at the current time, a more accurate pose of the self-moving device is calculated, thereby overcoming the inherent defects of traditional methods in robustness and accuracy, and realizing more reliable and accurate positioning performance. BRIEF DESCRIPTION OF DRAWINGS

[0018] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description of embodiments of the present application taken in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of embodiments of the present application and are incorporated in and constitute a part of this specification, illustrate embodiments of the present application and serve to explain the principles of the present application, but are not intended to limit the present application. In the drawings, like reference numerals refer to like elements or steps throughout.

[0019] Figure 1 Fig. 1 shows a flowchart of a positioning method according to an embodiment of the present application.

[0020] Figure 2 Fig. 2 shows a structural diagram of a positioning device according to an embodiment of the present application.

[0021] Figure 3 Fig. 3 shows a structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the scope of protection of the present application.

[0023] In the current positioning technology, there are two major schemes for how to quickly and accurately recover the initial pose of an image acquisition device (such as a camera) in a constructed map. Although the two schemes differ slightly in implementation details, their core ideas both follow the two-stage framework of finding correspondence first and then solving pose.

[0024] Specifically, the first type of method is to fix the scene in the form of a bag-of-words model in the pre-mapping stage. For example, first, local feature extraction is performed on the reference frame, and then the two-dimensional coordinates of the feature points and their corresponding three-dimensional coordinates are stored in the map database, and a bag-of-words tree structure is used for indexing to speed up subsequent retrieval. When the mobile device loses positioning due to occlusion, severe motion, sensor failure, etc. during operation and needs to re-estimate its pose, a repositioning thread is immediately started. Specifically, the same feature extraction process is repeated for the newly acquired current frame image to obtain a set of two-dimensional feature points. Then, with the help of the bag-of-words model, the candidate reference frames with the closest visual word distribution to the current frame are retrieved from the map within milliseconds. For each candidate frame, further low-level feature matching is performed to retain high-quality homonymous points. Once a sufficient number of two-dimensional and three-dimensional correspondences are obtained, the extrinsic matrix of the current frame can be solved iteratively, and finally the initial pose of the camera in the map coordinate system is obtained.

[0025] The advantage of this link is that the process is transparent and the computational complexity is controllable, and it can achieve fast repositioning in a medium-sized scene with rich texture and no severe changes in lighting. However, its defects are also significant. On the one hand, the bag-of-words model only makes a statistical-level judgment on whether the appearance is similar, and cannot guarantee geometric consistency. When there are a large number of repeated textures in the scene, perceptual ambiguity is likely to occur, leading to distorted candidate reference frame ordering. On the other hand, the entire link has a high sensitivity to the accuracy of the three-dimensional coordinates generated during the front-end mapping. If there is scale drift, depth estimation skew, etc. in the initial mapping stage, even if the two-dimensional and three-dimensional matching is perfect, it will also bring a non-negligible systematic error. In addition, when the camera experiences pure rotation, low parallax, or image blur, the number of effective features that can be extracted decreases sharply, and the entire repositioning process will fail directly due to the lack of corresponding points.

[0026] The second type of method is to solve the problem of similar appearance but different geometry by means of the representation ability of deep learning. The basic idea is: instead of relying on manually designed local features and bag-of-words statistics, a convolutional neural network is used to learn the global features of an image end-to-end on a large amount of scene data. In actual positioning, a forward inference is first performed on the current frame to obtain a compact global vector; then, in the map database, a nearest neighbor search algorithm such as brute force traversal is used to quickly lock the frame or several frames of old images with the closest Euclidean distance to the current vector. These old images also store the two-dimensional feature points and their corresponding depth values output by the front end at the time of mapping, so the known depth information of the old images can be used to complete the pose solution. However, its accuracy is still limited by the reliability of the front-end depth information. For example, if there is a depth estimation bias in the original mapping of the old image, even if there are more matching point pairs, the solved external parameters will also continue this error.

[0027] In view of this, the present application is proposed. Specifically, Figure 1 As shown in FIG. 1, a flowchart of a positioning method provided by an embodiment of the present application is shown. Exemplarily, as shown in FIG. 2, the method comprises the following steps. Figure 1 As shown in FIG. 1, a flowchart of a positioning method provided by an embodiment of the present application is shown. Exemplarily, as shown in FIG. 2, the method comprises the following steps.

[0028] Step S110, determining a plurality of reference frames in an environment in which a mobile device is located, and a pose corresponding to each reference frame.

[0029] The reference frame represents a representative frame selected during mapping. These frames have sufficient feature information, and the reference frames have sufficient pose or angle difference between them.

[0030] The pose corresponding to the reference frame represents the position and direction of the reference frame in the global coordinate system corresponding to the map containing the area in which the mobile device is located.

[0031] Specifically, during the mapping process, reference frames are selected from the video stream or image sequence according to certain rules (such as frame spacing, frame angle change, etc.). For each reference frame, its pose, i.e., the position and direction of the reference frame in the map, is calculated and recorded. In one implementation, to facilitate query, a KDTree can be constructed, which is a data structure that can quickly retrieve the pose of the reference frame, used to store the pose of the reference frame.

[0032] Step S120, acquiring an image frame collected by the mobile device at a current time.

[0033] It should be noted that the current time refers to the moment when the mobile device is running and needs to calculate its own position, which is different from the historical time when the reference frame is used during mapping.

[0034] Exemplarily, the self-moving device captures an image at the current time through a camera mounted thereon, and takes the image as input data for subsequent positioning calculation. It can be understood that the image contains rich visual information, especially feature points in the environment, such as corners of a table, door frames, edges of furniture, textures, and the like. These feature points are the basis for image matching and calculation in subsequent steps.

[0035] In some embodiments, the current time includes a time when the self-moving device is started.

[0036] Specifically, the scenario corresponds to the cold start or initial positioning requirement of the self-moving device. For example, when a floor cleaning robot is woken up from a charging pile and starts to perform a task, it has no idea of its own position in a pre-constructed map. At this time, it captures the first environmental image (i.e., the current image frame) after starting through the camera mounted thereon, and matches the image with all reference frames in the map database. If it can successfully identify that the features in the image (such as specific furniture arrangement, door frame structure) are highly consistent with the reference frame of the living room area, it can calculate the accurate pose of itself relative to the global coordinate system from this, so as to know the starting point of the task and plan the path from this position. This capability enables the self-moving device to achieve truly full automation without manual specification of the starting position by the user.

[0037] In other embodiments, the current time includes a time when the self-moving device loses the pose.

[0038] This scenario reflects the fault recovery or repositioning capability of the self-moving device during operation. For example, the floor cleaning robot described above can be accidentally picked up and moved to another room (such as from the living room to the dining room) by the user during operation, which external interference causes a serious deviation between the pose calculated based on the internal odometer and the true sensor observation value, and the self-moving device determines that it is in a pose loss state and triggers the repositioning process. It will collect a new image frame as soon as it stabilizes, and also perform a matching process with the reference frame. By identifying the features in the current dining room environment (such as a dining table and dining chairs), and matching them with the corresponding reference frame in the map, the accurate position in the new environment is recalculated.

[0039] This repositioning capability is an important indicator for measuring the robustness of the positioning system of the self-moving device, which enables the self-moving device to cope with unexpected interference and correct itself from errors, ensuring the continuity of the task and greatly reducing the dependence on external manual intervention.

[0040] Step S130 determines a plurality of target frames matched with the features of the image frame from a plurality of reference frames.

[0041] Optionally, the current image frame and all stored reference frames are processed, and feature points (i.e., significant corner points or edge points) in the image frame are identified using a specific feature detector, and feature vectors for each feature point are calculated.

[0042] Then, the feature matching stage begins. The purpose of this stage is to identify which feature points in the current image frame correspond to the same physical point in the environment among the feature points in each reference frame. Specifically, this is achieved by calculating the difference (distance) between the feature vectors of the feature points in the current image frame and the feature vectors of the feature points in the reference frames. Optionally, to ensure the reliability of the matching, strategies such as ratio testing can be used to initially filter out ambiguous or incorrect matching pairs, thus obtaining a preliminary set of matching point pairs and their number for each pair of image frames and reference frames.

[0043] Finally, target frames are selected based on the matching results. Understandably, given the large number of reference frames, performing fine-grained matching directly with all reference frames would be computationally intensive. Therefore, in practical applications, a fast retrieval structure such as KD-Tree can be used to quickly narrow down the candidate range from a large number of reference frames, identifying some of the most likely candidate frames. Subsequently, considering the number and quality of matches—for example, whether the distribution of matched feature points in the current image frame is uniform, and the co-viewing relationship between reference frames—several optimal frames are determined from the candidate frames as target frames.

[0044] Step S140: Determine the spatial coordinates of the key points in each target frame based on the pixel coordinates of the key points in the image frame and the pose corresponding to each target frame.

[0045] Keypoints in an image frame refer to pixels extracted from the image frame that can significantly characterize local features of the environment. These keypoints are usually regions in the image with obvious gradient changes, such as corners, edge intersections, or spots with unique textures. Algorithmically, they are feature points that can be stably detected and easily re-identified and matched in subsequent frames.

[0046] Understandably, a correct 3D point, when reprojected back across all the target frames that observed it, should have a projection that is as close as possible to the actual matching pixel coordinates within those target frames. Specifically, for a keypoint in an image frame, let its desired spatial coordinates be... It is known that it is observed in n target frames, and the pose of each target frame i is determined by the rotation matrix. Translation vector Description, its corresponding pixel coordinates are The camera intrinsic parameter matrix is ​​K.

[0047] For each target frame, the reprojection of spatial point P onto its image plane can be calculated:

[0048]

[0049] Furthermore, an optimization problem can be constructed regarding the coordinates P of a spatial point, with the objective function being to minimize the sum of reprojection errors across all target frames:

[0050]

[0051] That is, to find an optimal three-dimensional point. This method minimizes the sum of squared Euclidean distances between the reprojected position and the actual observed pixel positions in all target frames. For example, the Gauss-Newton method or the Levenberg-Marquardt algorithm can be used to solve this problem. The advantage of this method is that it considers the constraints of all available views through iterative optimization, which can eliminate errors caused by observation noise to some extent.

[0052] Step S150: Determine the pose of the self-moving device at the current moment based on the spatial coordinates of the key points and the pixel coordinates of the key points in the image frame.

[0053] In one implementation, a deep neural network can be trained using a large amount of data. This network can learn the pose of the mobile device at the current moment from the spatial coordinates of keypoints and the pixel coordinates of the keypoints in the image frame.

[0054] In the application phase, no complex geometric calculations are required. Simply input the spatial coordinates of the keypoints, along with their pixel coordinates within the image frame, into the pre-trained model to directly output the pose of the mobile device at the current moment. The advantage of this method lies in its high computational speed.

[0055] The scheme in this application abandons the coarse matching strategy that relies on visual word statistical similarity. Instead, it directly performs feature matching to filter multiple target frames from the reference frames, effectively reducing matching ambiguity caused by repetitive structures. More importantly, this application does not use depth predictions, which have inherent generalization errors. Instead, it uses the known poses and observation information of multiple target frames to calculate the spatial coordinates of key points. This process ensures that the spatial coordinates of key points are directly derived from high-precision geometric constraints, rather than single-frame predictions, fundamentally avoiding systematic biases. Finally, using these high-confidence spatial coordinates and their projections into the current image frame, a more accurate pose of the self-moving device is calculated, overcoming the inherent deficiencies of traditional methods in robustness and accuracy, and achieving more reliable and accurate localization performance.

[0056] To improve the accuracy and computational efficiency of frame matching, this application introduces a hierarchical feature matching scheme. Specifically, it determines multiple target frames that match the features of an image frame from multiple reference frames, including: determining multiple reference frames that match the global features of an image frame from multiple reference frames; and determining multiple target frames that match the local features of an image frame from multiple reference frames that match the global features of an image frame.

[0057] Global features refer to abstract feature vectors that characterize the macroscopic content of the entire image. For example, they can be extracted and generated by deep learning networks. They are characterized by low dimensionality and insensitivity to local image deformation, making them suitable for quickly calculating the similarity between images. Local features, on the other hand, refer to key points and their high-dimensional features extracted from the image. They finely describe the texture and structural information of local regions of the image and are used for accurate point-to-point correspondence matching.

[0058] Optionally, the global features of the current image frame are calculated and compared with the global features of all reference frames (e.g., cosine distance is calculated). By setting a threshold, the candidate range is narrowed down from a massive number of reference frames, resulting in a significantly reduced subset of reference frames that are macroscopically similar to the current image frame. The purpose of this stage is to eliminate obviously irrelevant frames, greatly reducing the burden of subsequent calculations.

[0059] Then, fine-grained matching based on local features is performed within this subset of reference frames. For example, for the current image frame and each reference frame in the subset, local features are extracted and feature matching is performed (e.g., brute-force matching combined with ratio testing to eliminate false matches). The number of successfully matched feature points between each pair of frames is counted, and then, based on a preset threshold or by selecting several frames with the highest number of matches, multiple target frames are finally determined. This stage ensures that there are a sufficient number and accurate point-to-point correspondences between the reference frames used for pose calculation and the current frame, guaranteeing the reliability of subsequent geometric calculations.

[0060] In this embodiment, global features are first used for rapid preliminary screening, efficiently eliminating a large number of irrelevant reference frames, narrowing the candidate range, and greatly reducing the computational load of subsequent processing, thus providing efficiency assurance for real-time localization. After obtaining multiple reference frames that match the global features of the image frames, fine-grained matching and verification based on local features are then performed. This stage provides the foundation for establishing accurate point-to-point correspondences, not only significantly improving the matching success rate but also effectively reducing the false matching rate, ensuring high accuracy and high consistency of the final output correspondences.

[0061] To significantly improve the retrieval efficiency and reduce the computational burden while ensuring the matching accuracy, the embodiment of the present application provides a specific global feature fast screening scheme. Specifically, the global features of the image frame are matched from a plurality of reference frames, comprising: determining the global descriptors corresponding to each of the plurality of reference frames and the global descriptor of the image frame; calculating the similarity of the global descriptor of the image frame and the global descriptor of each reference frame; determining the reference frame with a similarity higher than a preset threshold as the reference frame matched with the global feature of the image frame.

[0062] The global descriptor is a low-dimensional numerical vector that can compactly represent the macroscopic visual content of the entire image. For example, the descriptor can be extracted by a deep learning model (such as a convolutional neural network), which has the characteristics of strong representation ability and low computational complexity.

[0063] Optionally, in the mapping stage, the global descriptor corresponding to each reference frame in the map is calculated in advance and stored in a specific data structure for fast access. When positioning is needed, the global descriptor of the current image frame is extracted in real time.

[0064] The similarity is used to quantify the degree of association between two global descriptors, which is usually achieved by calculating the distance or correlation between vectors, such as using cosine similarity, Euclidean distance or Hamming distance for measurement. Further, by using a vector similarity calculation algorithm, the global descriptor of the current image frame is compared with the global descriptors of all reference frames in the database one by one, and a series of similarity scores are obtained.

[0065] Finally, the similarity scores calculated above are compared with a preset threshold, and all reference frames with a similarity higher than the threshold are screened out to form a candidate set for subsequent local feature matching. The preset threshold is a pre-set threshold value used to determine whether the similarity is sufficient to support that two images are considered to be macroscopically matched. For example, the threshold value can be empirically or adaptively set according to the requirements of recall rate and accuracy rate in actual application scenarios.

[0066] In the embodiment, the low-dimensional global descriptor is used to abstractly represent the macroscopic content of the image, and the similarity is calculated to realize the fast traversal and screening of a large number of reference frames. This content-based fast retrieval mechanism controls the number of candidate frames that need to be processed in detail within a limited range, fundamentally reducing the overall computational burden and making real-time positioning possible for resource-constrained mobile platforms.

[0067] Meanwhile, by presetting a threshold, the scheme can ensure that the selected candidate reference frames have sufficient similarity with the current image frame in terms of overall scene structure, spatial layout and other high-level semantic levels, laying a foundation for subsequent local feature matching and avoiding the interference of obviously irrelevant frames, thereby improving the success rate and robustness of the entire positioning process.

[0068] In some embodiments, from the plurality of reference frames matched with the global features of the image frame, a plurality of target frames matched with the local features of the image frame are determined, including: based on the feature descriptors of the local features of the image frame, matching the image frame and the plurality of reference frames matched with the global features of the image frame to obtain the plurality of target frames.

[0069] The feature descriptor of the local feature refers to a high-dimensional numerical vector calculated by a specific neural network for each local feature in the image frame. The descriptor can deeply and robustly represent the visual appearance around the local feature.

[0070] Optionally, the feature descriptors of all local features in the image frame are first extracted. At the same time, the plurality of reference frames selected through the global feature matching are obtained, and the feature descriptors of the local features in these reference frames are pre-extracted. Then, a matching loop is started, for example, the image frame is paired with each selected reference frame, and a lightweight matcher (such as LightGlue or SuperGlue) is used for feature matching. The matcher finds the key point pairs belonging to the same local feature in the two frames by calculating the similarity between the feature descriptors of the two parties and combining attention mechanism and other algorithms, and outputs a stable matching pair list. Finally, for each successfully matched local feature in the image frame, the matching result is recorded as a set containing multiple pieces of information. The set indicates that the local feature of the image frame successfully establishes a corresponding relationship with which local feature in which selected reference frame. The reference frame in which the corresponding relationship is successfully established is determined as the target frame.

[0071] For example, assuming that a local feature A in the image frame at the current time successfully matches a local feature B in the selected reference frame 1 and a local feature C in the selected reference frame 2. Then, the matching result of the key point A will be recorded as: {local feature B, reference frame 1}, {local feature C, reference frame 2}. Finally, candidate frames such as reference frames 1 and 2, which have a large number of such matching relationships, are selected as target frames.

[0072] The feature descriptor of the local feature in this embodiment has stronger representation ability and distinguishability, and can realize higher-precision feature correspondence matching in a complex scene with high dynamic light, view angle change and repeated texture. In addition, this scheme forms an efficient cooperation with the aforementioned global feature screening stage. The global screening greatly reduces the number of reference frames to be matched, so that this stage can concentrate limited computing resources on high-quality frame pair matching, thereby optimizing the overall computing efficiency.

[0073] In some embodiments, the local feature of the image frame includes a key point in the image frame.

[0074] Specifically, the key point itself is located at a position with significant gradient change (such as a corner point or an edge) in the image frame, and the corresponding feature descriptor (such as ORB or SIFT) can mathematically represent the visual pattern around the point. This detailed description enables the establishment of a unique and stable pixel-level correspondence between the current image frame and the target frame among a plurality of reference frames matched with the global feature of the image frame. This accurate point-to-point matching is a prerequisite for calculating the spatial coordinates of the key point in the subsequent step S140.

[0075] Secondly, step S140 needs to use the pixel coordinates and the pose of the key point in the target frame to calculate the spatial coordinates. Step S150 needs to use the pixel coordinates and the spatial coordinates of the key point in the current image frame to solve the pose. Therefore, directly using the features of the key points for comparison in the matching stage (step S130) can obtain all the input data required by the above two key steps: the matched key point pair and its pixel positions in the two images. This design ensures the high consistency of the data flow and avoids the information loss or additional computational overhead caused by the conversion between different feature forms.

[0076] Finally, the features of the key points can be used for efficient calculation and support fast similarity comparison, making it possible to perform a large number of local feature matching in the reference frames screened by the global feature. At the same time, since it describes the local invariant features of the image, it has good robustness to changes in light, view angle and partial occlusion, thereby ensuring reliable matching in complex environments and ultimately ensuring the accuracy and stability of the entire positioning system.

[0077] In some embodiments, the key point in the image frame refers to a feature point that is successfully matched in at least two reference frames in the matching process of the image frame and the plurality of reference frames.

[0078] Specifically, the definition firstly ensures the feasibility of subsequently calculating the spatial coordinates of the key points by using a triangulation operation. It can be understood that the precondition for triangulation to solve the spatial coordinates is that a feature point must be captured by at least two observations of different perspectives. Therefore, the scheme ensures that each key point put into calculation meets this requirement by the above screening condition, thereby providing stable and effective input data for the triangulation process and avoiding invalid calculation. In addition, after the screening process, the number of key points participating in the triangulation calculation and pose solving is reduced, effectively reducing the computational complexity of the algorithm and reducing the occupation of processing resources.

[0079] To improve the solving accuracy and robustness of three-dimensional space point coordinates by using multi-perspective geometry, an embodiment of the present application provides a three-dimensional reconstruction scheme based on ray convergence. Specifically, based on the pixel coordinates of the key points in each target frame and the pose corresponding to each target frame, the spatial coordinates of the key points are determined, including: determining the projection center of each target frame based on the pose corresponding to each target frame; determining the rays from the projection center of the target frame to the pixel coordinates of the key points in the target frame, and determining the convergence position of the multiple rays, with the projection center of each target frame as the starting point; determining the spatial coordinates of the key points based on the convergence position of the multiple rays.

[0080] The pose corresponding to the target frame refers to the rotation and translation transformation relationship of the camera coordinate system of the reference frame relative to the environment global coordinate system, which defines the position and orientation of the frame camera in space. The projection center refers to the three-dimensional space coordinates of the optical center of the camera corresponding to the global coordinate system under the pose of the current target frame.

[0081] The ray in the embodiment is a virtual straight line with directionality in three-dimensional space. The starting point is the projection center of the target frame, and the direction is determined by the intrinsic model of the camera of the target frame and the pixel coordinates of the key points on the image plane. Specifically, the direction is obtained by projecting the pixel coordinates back to the normalized plane in the camera coordinate system, and then transforming to the global coordinate system according to the pose (rotation matrix) of the target frame.

[0082] Optionally, for a key point to be solved, the pixel coordinates of the key point in multiple target frames and the poses corresponding to each target frame are obtained. For each target frame, a ray is constructed from the projection center thereof to the direction of the key point observed by the target frame.

[0083] In some embodiments, due to the existence of camera observation noise, feature positioning error and pose estimation error, these rays cannot strictly intersect in reality. Therefore, an optimal spatial point coordinate can be determined by constructing an optimization problem for the convergence position of multiple rays. For example, a three-dimensional spatial point is calculated such that the sum of the Euclidean distances from each ray is minimized, and this point is determined as the spatial coordinate of the key point. Through this optimization in the least squares sense, various types of observation errors can be effectively reduced, thereby obtaining a high-precision and most consistent spatial point estimate.

[0084] In the present embodiment, by utilizing different perspective observations provided by multiple target frames and constructing their corresponding projection rays, and then solving the optimal convergence point of multiple rays, this process can effectively reduce the observation noise, feature positioning error and pose estimation deviation of a single perspective, thereby obtaining an optimal and more accurate spatial coordinate estimate. In addition, compared with methods relying on a single perspective or limited perspectives, the multi-ray intersection mechanism reduces the dependence on the observation quality of any particular perspective. Even if there are large errors or mis-matches in the observation of individual target frames, the optimization algorithm can suppress the influence of outliers through the constraints of a large number of accurate observations, stably output the correct spatial point position, and thus ensure the reliability of the entire positioning process.

[0085] To accurately and stably convert these three-dimensional spatial information into the real-time pose of the self-moving device, the embodiments of the present application adopt a pose solving scheme based on re-projection error optimization, the specific implementation of which is described as follows. Based on the spatial coordinates of the key points and the pixel coordinates of the key points in the image frame, the pose of the self-moving device at the current time is determined, including: based on the spatial coordinates of the key points and the pixel coordinates of the key points in the image frame, an optimization function is constructed; the optimization function is solved to obtain an optimal pose, the optimal pose makes the error between the pixel coordinates of the spatial coordinates of the key points projected into the image frame and the pixel coordinates of the key points in the image frame minimum; and the optimal pose is determined as the pose of the self-moving device at the current time.

[0086] The optimization function is a mathematical objective function aimed at quantifying the accuracy of pose estimation, and its core is the re-projection error, i.e., the sum of the squares of the differences between the theoretical pixel coordinates of a three-dimensional spatial point projected onto the current image plane according to a to-be-solved pose and its actually observed pixel coordinates.

[0087] Optionally, the pose of the camera to be solved is an optimization variable. For each pair of known two-dimensional and three-dimensional point correspondences, the re-projection error is calculated. The sum of the squares of the re-projection errors of all point correspondences is summed, i.e. the final optimization function is constructed. Subsequently, the optimization function is solved to obtain the optimal pose. Exemplarily, an iterative optimization algorithm is used for solving, such as the Gauss-Newton algorithm. The solving process starts from an initial pose guess, and through iteratively adjusting the pose parameters, the value of the optimization function is constantly reduced, and finally converges to a local optimal solution, i.e. the optimal pose. The optimal pose is the pose estimate that minimizes the sum of the re-projection errors of all spatial points in the least squares sense. Finally, the optimal pose obtained by solving, i.e. the rotation matrix and the translation vector, is determined as the pose of the self-moving device at the current time. The pose accurately describes the position and direction of the self-moving device relative to the global coordinate system.

[0088] In the embodiment, by minimizing the error of re-projecting the spatial coordinates to the image plane, the observation information of all available two-dimensional and three-dimensional point correspondences can be effectively fused and the inherent noise error can be reduced, thereby obtaining the optimal pose and significantly improving the absolute accuracy of the positioning result. In addition, the optimization algorithm can naturally weigh the contribution of each observation point to the overall error in the iterative solving process. For abnormal observation points with larger errors, the gradient direction generated thereby will be different from that of other correct observation points, so that the influence of the abnormal observation points on the final solution is effectively suppressed. This mechanism enables the positioning system to maintain stable output performance when facing a small amount of matching errors or spatial coordinate errors.

[0089] The positioning method embodiments of the present application are described in detail above Figure 1 , and the positioning device embodiments of the present application are described in detail below Figure 2 . It should be understood that the description of the positioning method embodiments corresponds to the description of the positioning device embodiments, and therefore, the parts not described in detail can be referred to the foregoing method embodiments.

[0090] Figure 2 Fig. 1 shows a structural schematic diagram of a positioning device provided by an embodiment of the present application. As shown in Fig. 1, the positioning device 20 provided by the embodiment of the present application comprises: Figure 2

[0091] A first determining module 210 is configured to determine a plurality of reference frames in an environment in which a self-moving device is located, and a pose corresponding to each reference frame;

[0092] An acquiring module 220 is configured to acquire an image frame collected by the self-moving device at a current time;

[0093] A second determining module 230 is configured to determine a plurality of target frames matched with features of the image frame from the plurality of reference frames;

[0094] ​The third determining module 240 is configured to determine the spatial coordinates of the key point based on the pixel coordinates of the key point in each target frame and the pose corresponding to each target frame.

[0095] The fourth determining module 250 is configured to determine the pose of the mobile device at the current time based on the spatial coordinates of the key point and the pixel coordinates of the key point in the image frame.

[0096] In an embodiment of the present application, the second determining module 230 is further configured to determine a plurality of reference frames matching the global feature of the image frame from the plurality of reference frames, and determine a plurality of target frames matching the local feature of the image frame from the plurality of reference frames matching the global feature of the image frame.

[0097] In an embodiment of the present application, the second determining module 230 is further configured to determine the global descriptor corresponding to each of the plurality of reference frames and the global descriptor of the image frame, calculate the similarity between the global descriptor of the image frame and the global descriptor of each reference frame, determine the reference frame with a similarity higher than a preset threshold as the reference frame matching the global feature of the image frame, and / or match the image frame and the plurality of reference frames matching the global feature of the image frame based on the feature descriptor of the local feature of the image frame to obtain the plurality of target frames.

[0098] In an embodiment of the present application, the local feature of the image frame includes the feature of the key point in the image frame.

[0099] In an embodiment of the present application, the key point in the image frame refers to a feature point successfully matched in at least two reference frames in the matching process of the image frame and the plurality of reference frames.

[0100] In an embodiment of the present application, the third determining module 240 is further configured to determine the projection center of each target frame based on the pose corresponding to each target frame, determine the rays from the projection center of the target frame to the pixel coordinates of the key point in the target frame and the converging position of the plurality of rays based on the projection center of each target frame as the starting point, and determine the spatial coordinates of the key point based on the converging position of the plurality of rays.

[0101] In an embodiment of the present application, the fourth determining module 250 is further configured to construct an optimization function based on the spatial coordinates of the key point and the pixel coordinates of the key point in the image frame, solve the optimization function to obtain an optimal pose, the optimal pose making the error between the pixel coordinates of the spatial coordinates of the key point projected to the image frame and the pixel coordinates of the key point in the image frame minimum, and determine the optimal pose as the pose of the mobile device at the current time.

[0102] In an embodiment of the present application, the current time includes the time when the mobile device is started, and / or the current time includes the time when the mobile device loses the pose.

[0103] Below, with reference to Figure 3 an electronic device according to an embodiment of the present application will be described. Figure 3 Fig. 1 shows a schematic configuration of an electronic device according to an example embodiment of the present application.

[0104] As Figure 3 shown, the electronic device 30 includes one or more processors 301 and a memory 302.

[0105] The processor 301 can be a central processing unit (CPU) or other form of processor that has data processing capabilities and / or instruction execution capabilities, and can control other components in the electronic device 30 to execute desired functions.

[0106] The memory 302 can include one or more computer program products that can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory, for example, can include random access memory (RAM), cache memory, and / or the like. The non-volatile memory, for example, can include read only memory (ROM), hard disk, flash memory, and / or the like. One or more computer program instructions can be stored on the computer-readable storage media, and the processor 301 can run the program instructions to implement the positioning method of various embodiments of the present application described above and / or other desired functions.

[0107] In one example, the electronic device 30 can further include an input device 303 and an output device 304, which are interconnected through a bus system and / or other form of connection mechanism (not shown).

[0108] The input device 303 can include, for example, a keyboard, a mouse, and / or the like.

[0109] The output device 304 can output various information to the outside, and can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and / or the like.

[0110] Of course, in order to simplify, Figure 3 only some of the components in the electronic device 30 related to the present application are shown, and components such as buses, input / output interfaces, and / or the like are omitted. In addition, the electronic device 30 can include any other appropriate components according to specific application cases.

[0111] In addition to the methods and devices described above, embodiments of the present application can also be a computer program product that includes computer program instructions that, when run by a processor, cause the processor to perform steps of the positioning methods according to various embodiments of the present application described above in the specification.

[0112] The computer program instructions can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, etc., and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server.

[0113] In addition, embodiments of the present application can also be a computer readable storage medium having stored thereon computer program instructions that, when run by a processor, cause the processor to perform steps of the positioning methods according to various embodiments of the present application described above in the specification.

[0114] The computer readable storage medium can be any combination of one or more computer readable media. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can include, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples (a non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0115] The above describes the basic principles of the present application in combination with specific embodiments, but it should be noted that the advantages, advantages, effects, etc. mentioned in the present application are only examples and are not limiting, and these advantages, advantages, effects, etc. cannot be considered as the must-have of each embodiment of the present application. In addition, the above specific details are only for the purpose of example and understanding, and are not limiting, and the above details do not limit the present application to the must-use of the above specific details.

[0116] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0117] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0118] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0119] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A positioning method, characterized by, The method comprises: determining a plurality of reference frames in an environment where the self-moving device is located, and a pose corresponding to each of the reference frames; obtaining an image frame collected by the self-moving device at a current time; determining a plurality of target frames from the plurality of reference frames, the target frames matching features of the image frame; determining spatial coordinates of a key point in the image frame based on pixel coordinates of the key point in each of the target frames and the pose corresponding to each of the target frames; determining a pose of the self-moving device at the current time based on the spatial coordinates of the key point and the pixel coordinates of the key point in the image frame.

2. The positioning method according to claim 1, characterized in that, The determining of the plurality of target frames from the plurality of reference frames, the target frames matching features of the image frame, comprises: determining a plurality of reference frames from the plurality of reference frames, the reference frames matching global features of the image frame; determining a plurality of target frames from the plurality of reference frames, the target frames matching local features of the image frame.

3. The positioning method according to claim 2, characterized in that, The determining of the plurality of reference frames from the plurality of reference frames, the reference frames matching global features of the image frame, comprises: determining global descriptors corresponding to each of the plurality of reference frames and a global descriptor of the image frame; calculating a similarity between the global descriptor of the image frame and the global descriptor of each of the reference frames; determining a reference frame with a similarity higher than a preset threshold as a reference frame matching global features of the image frame. And / or, the determining of the plurality of target frames from the plurality of reference frames, the target frames matching local features of the image frame, comprises: matching the image frame and the plurality of reference frames matching global features of the image frame based on feature descriptors of local features of the image frame to obtain the plurality of target frames.

4. The positioning method according to claim 2 or 3, characterized in that, The local features of the image frame comprise key points in the image frame.

5. The positioning method according to claim 4, characterized in that, The key points in the image frame refer to feature points that are successfully matched in at least two of the reference frames in a matching process of the image frame and the plurality of reference frames.

6. The positioning method according to any one of claims 1 to 3, characterized in that, The determining of the spatial coordinates of the key point based on the pixel coordinates of the key point in each of the target frames and the pose corresponding to each of the target frames, comprises: determining a projection center of each of the target frames based on the pose corresponding to each of the target frames; determining a ray from the projection center of the target frame to the pixel coordinates of the key point in the target frame and a convergence position of a plurality of rays, with the projection center of each of the target frames as a starting point; determining the spatial coordinates of the key point based on the convergence position of the plurality of rays.

7. The positioning method according to any one of claims 1 to 3, characterized in that, The determining of the pose of the self-moving device at the current time based on the spatial coordinates of the key point and the pixel coordinates of the key point in the image frame, comprises: constructing an optimization function based on the spatial coordinates of the key point and the pixel coordinates of the key point in the image frame; solving the optimization function to obtain an optimal pose, the optimal pose making an error between the spatial coordinates of the key point and the pixel coordinates of the key point in the image frame after projection of the spatial coordinates of the key point to the image frame to be minimum. The optimal pose is determined as a pose of the self-moving device at a current time.

8. The positioning method of any one of claims 1-3, characterized in that, the current time comprises a time when the self-moving device is started; and / or, the current time comprises a time when the self-moving device loses a pose.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and the computer program is used for executing the positioning method in any one of claims 1-8.

10. An electronic device, comprising: comprising: a processor; a memory for storing instructions executable by the processor; the processor is configured to execute the positioning method in any one of claims 1-8.

11. A computer program product, characterised in that, The computer program product comprises instructions for causing an electronic device to implement the positioning method in any one of claims 1-8 when the instructions are executed on the electronic device.

Images