Pose determination method, apparatus, device, storage medium, and program product

Abstract

This application provides a pose determination method, apparatus, device, storage medium, and program product. The method includes: extracting semantic and geometric features from multiple frames of images captured by a camera, encoding the semantic and geometric features into a unified data structure, performing temporal matching on the extracted features to generate a feature set containing temporal information, calculating the camera pose based on the feature set, and optimizing the pose based on joint semantic and geometric constraints to obtain a more accurate pose. This approach enables deep fusion of geometric and semantic features, effectively improving feature stability and pose accuracy in traffic scenarios where single features are sparse.

Description

Technical Field

[0001] This application relates to the field of visual positioning, and more particularly to a pose determination method, apparatus, device, storage medium, and program product. Background Technology

[0002] Visual SLAM (Simultaneous Localization and Mapping) technology aims to achieve self-localization and map building in unknown environments through sensors such as cameras. This technology is widely used in the fields of autonomous driving and intelligent mobility.

[0003] Currently, visual SLAM systems rely on stable visual feature points to calculate poses. However, in scenarios such as intersections and rural roads, there are problems of sparse and unstable features, which can easily cause pose estimation drift or loss, affecting the overall positioning accuracy and robustness of visual SLAM systems. Summary of the Invention

[0004] This application provides a pose determination method, apparatus, device, storage medium, and program product to improve pose accuracy in sparsely characterized scenarios such as intersections.

[0005] In a first aspect, embodiments of this application provide a pose determination method, including:

[0006] Semantic and geometric features are extracted from multiple frames of images captured by the camera, and the semantic and geometric features are encoded into a unified data structure.

[0007] Based on a unified data structure, features extracted from multiple frames of images are temporally matched to generate a feature set containing temporal information.

[0008] A unified set of constraints is constructed based on semantic and geometric features. The camera pose is then optimized according to the constraints and feature set to determine the optimized pose.

[0009] In some possible implementations, temporal matching is performed on features extracted from multiple frames of images to generate a feature set containing temporal information, including:

[0010] For each extracted semantic feature, perform cross-frame matching and generate temporal information of the successfully matched semantic features; and / or,

[0011] Batch cross-frame matching is performed on the extracted geometric features, and temporal information of the successfully matched geometric features is generated.

[0012] In the above approach, matching semantic features one by one across frames ensures their stability and accuracy, and avoids interference between different semantic features. For geometric features, which are numerous, batch matching can improve the efficiency of cross-frame tracking and matching, ensure the continuity of matching, and provide a stable data foundation for subsequent pose optimization.

[0013] In some possible implementations, batch cross-frame matching is performed on the extracted geometric features, including:

[0014] For two frames in a multi-frame image, the geometric features extracted from the earlier frame are grouped into a first set, and the geometric features extracted from the later frame are grouped into a second set.

[0015] For any geometric feature in the first set, the geometric feature and the geometric feature that is closest to it in the second set are identified as positive matching pairs;

[0016] For the geometric features in the second set, the geometric feature and the geometric feature that is closest to the geometric feature in the first set are identified as reverse matching pairs;

[0017] If the forward match and the backward match are the same, the match is successful.

[0018] In the above method, when matching geometric features, forward and reverse matching are performed according to the temporal sequence of image frames. Only the matching results of geometric features that are consistent in both directions are retained, and inconsistent matching results are eliminated. In this way, the mismatch problem that may be caused by the texture similarity of a geometric feature to multiple geometric features in other frames can be eliminated, thereby further improving the data quality of the features.

[0019] In some possible implementations, before optimizing the camera pose according to constraints and feature sets, the following steps are also included:

[0020] Based on the camera's shooting angle, any frame of image is divided into regions to obtain multiple image regions;

[0021] For each image region, the camera's extrinsic parameter changes are calculated based on the geometric features within that region, and geometric features that do not meet the preset conditions are eliminated based on the calculated extrinsic parameter changes.

[0022] In the above method, dividing the image region according to the camera shooting angle can make the distribution of feature points in each image region more consistent with the characteristics of the scene. For example, the feature points in the sky are sparse and have poor usability, while the feature points on the ground are dense and have strong usability. Thus, a suitable extrinsic parameter calculation method can be selected to retain geometric feature points.

[0023] In some possible implementations, the camera's extrinsic parameter variations are calculated based on the geometric features within the region, including:

[0024] If the number of geometric features in the image region is greater than or equal to a preset threshold, the essential matrix is ​​used to calculate the changes in the camera's extrinsic parameters based on the geometric features.

[0025] If the number of geometric features in the image region is less than a preset threshold, the homography matrix is ​​used to calculate the changes in the camera's extrinsic parameters based on the geometric features.

[0026] In some possible implementations, unified constraints are constructed based on semantic and geometric features, and the camera pose is optimized according to the constraints and feature set, including:

[0027] The real-time pose of the camera is calculated based on the initial pose of the camera and the geometric and semantic features in the feature set.

[0028] Calculate the three-dimensional coordinates of the geometric and semantic features in the feature set in the world coordinate system;

[0029] Based on the real-time pose of the camera, the three-dimensional coordinates are reprojected onto the image plane acquired by the camera to obtain the reprojection features;

[0030] The residual between the reprojected features and the original features in the image plane is calculated, and the real-time pose of the camera is optimized with the goal of minimizing the residual. The original features are semantic and geometric features extracted from multiple frames of images acquired by the camera.

[0031] In the above approach, an objective function that includes semantic and geometric constraints is constructed for a unified reprojection framework. Pose optimization is performed by minimizing the objective function, which enables the coordinated constraint of semantic and geometric information under the unified optimization framework. This can improve the robustness and accuracy of pose estimation in complex scenes with sparse features, such as intersections.

[0032] In some possible implementations, the linear semantic features are discretized into several sampling points when performing reprojection and calculating residuals.

[0033] In the above method, the continuous linear semantic features are discretized into sampling points, which makes it easier to calculate the reprojection residual based on the original image coordinates and the reprojected image coordinates of the sampling points, reducing the computational difficulty and improving the optimization efficiency.

[0034] Secondly, embodiments of this application provide a pose determination device, comprising:

[0035] The extraction module is used to extract semantic and geometric features from multiple frames of images captured by the camera, and encode the semantic and geometric features into a unified data structure.

[0036] The matching module is used to perform temporal matching of features extracted from multiple frames of images based on a unified data structure, and generate a feature set containing temporal information.

[0037] The optimization module is used to construct unified constraints based on semantic and geometric features, optimize the camera pose according to the constraints and feature set, and determine the optimized pose.

[0038] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0039] The memory stores instructions that the computer executes;

[0040] The processor executes computer execution instructions stored in memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0041] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0042] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0043] The pose determination method, apparatus, device, storage medium, and program product provided in this application extract semantic and geometric features from multiple frames of images captured by a camera, and encode these features into a unified data structure. This allows for temporal matching of the extracted features, generating a feature set containing temporal information. The camera pose is then calculated based on the feature set, and optimized using joint semantic and geometric constraints, resulting in a more accurate pose. This approach enables deep fusion of geometric and semantic features, effectively improving feature stability and pose accuracy in traffic scenarios where single features are sparse. Attached Figure Description

[0044] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0045] Figure 1 A flowchart illustrating a pose determination method provided as an example in this application;

[0046] Figure 2 This application provides an exemplary schematic diagram of adjacent frame image feature extraction.

[0047] Figure 3 A schematic diagram illustrating the calculation of three-dimensional coordinates is provided as an example in this application;

[0048] Figure 4 A schematic diagram illustrating a pose optimization process based on the fusion of SuperPoint and semantic features, provided as an example of this application;

[0049] Figure 5 A schematic diagram illustrating the process of SuperPoint participating in pose optimization, provided as an example of this application;

[0050] Figure 6 A schematic diagram of pose optimization provided as an example in this application;

[0051] Figure 7 This is a schematic diagram of the structure of a pose determination device provided as an example in this application;

[0052] Figure 8 This is a schematic diagram of the structure of an electronic device provided as an example in this application.

[0053] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0054] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0055] Visual SLAM (Simultaneous Localization and Mapping) technology has been widely used in intelligent mobility fields such as autonomous driving, mobile robots, and augmented reality. Currently, visual SLAM solutions mainly include SLAM systems based on geometric features and SLAM systems based on semantic information.

[0056] Geometric feature-based SLAM systems (such as ORB-SLAM and SuperPoint-SLAM) extract local feature points (e.g., ORB and SuperPoint) to perform inter-image feature matching and pose estimation, and then use keyframes to optimize and generate sparse maps. These methods perform well in scenes with rich textures and stable features, but in scenes with sparse features and many dynamic targets, such as intersections and squares, tracking is prone to loss, which can easily affect pose accuracy. Here, ORB and SuperPoint can refer to different geometric feature extraction algorithms and the feature points extracted by those algorithms.

[0057] Semantic information-based SLAM systems incorporate semantic elements such as object categories, ground, and road boundaries into the SLAM framework through methods like semantic segmentation and object detection, achieving semantic-level localization and mapping. While this approach can utilize semantic information, it often lacks a sufficient number of semantic elements in scenarios such as intersections, and its accuracy is not high enough.

[0058] In summary, in traffic areas with sparse features, such as intersections, turns, and merging points, both geometric and semantic SLAM systems often face difficulties in extracting and matching stable features due to issues such as feature sparsity, lighting variations, or motion blur. This can easily lead to pose estimation drift or loss, affecting positioning accuracy.

[0059] The inventors discovered that the key to the above problem is that single-dimensional features are easily lost in feature-sparse scenarios such as intersections, making it difficult to continuously track and match, and making it difficult to provide enough effective features when calculating pose.

[0060] Based on this, a technical concept is proposed to incorporate geometric and semantic features into a unified SLAM framework for unified management and joint optimization, achieving deep fusion of geometric and semantic information. Specifically, geometric and semantic features can be encoded into the same data structure to achieve unified representation and indexing of geometric feature points and semantic labels. Then, a unified optimization model is generated by constructing geometric and semantic constraints for pose optimization.

[0061] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0062] Figure 1 This application provides an exemplary flowchart of a pose determination method, as shown below. Figure 1 As shown, the method includes:

[0063] Step S101: Extract semantic and geometric features from the multi-frame images captured by the camera, and encode the semantic and geometric features into a unified data structure.

[0064] In this context, "camera" can refer to visual perception devices equipped on mobile units such as vehicles or intelligent robots, such as vehicle-mounted cameras, webcams, or other visual sensors.

[0065] In this embodiment, before calculating the pose based on the images captured by the camera, the SLAM system can first obtain the initial pose of the camera. This initial pose can be used to transform between the camera coordinate system and the coordinate system of the moving unit such as the vehicle. The initial pose can refer to the relative positional relationship between the camera and the moving unit, such as the camera's mounting position on the vehicle, shooting angle, and other parameters. These parameters can be read from the configuration file of the vehicle or robot.

[0066] For example, for images captured by a camera, semantic features can be extracted using semantic segmentation or object detection networks to obtain semantic elements such as lane lines, traffic signs, and poles, along with their corresponding category labels. Furthermore, geometric feature points can be extracted using the SuperPoint network to obtain the pixel coordinates, descriptors, and response values ​​of these geometric feature points. The pixel coordinates represent the location of the geometric feature point in the image, the descriptor represents the appearance information of the local feature point, and the response value represents the confidence level of the geometric feature point in the feature extraction algorithm.

[0067] After extracting semantic and geometric features, they can be stored and used using the same data structure. For example, semantic features may include fields such as semantic category, coordinates, and feature identifier, while geometric features include a descriptor field but not a semantic category field. Based on the data structure of semantic features, fields unique to geometric features, such as descriptors, can be added, and the extraction method of geometric features (such as ORB, SuperPoint, etc.) can be encoded as semantic categories. Thus, this data structure can be used to represent geometric features.

[0068] Step S102: Based on a unified data structure, perform temporal matching on the features extracted from multiple frames of images to generate a feature set containing temporal information.

[0069] For example, the extracted features can first be divided into semantic features and geometric features based on the semantic category field in the unified data structure. For semantic features, cross-frame association and trajectory tracking are performed based on their feature identifiers, pixel coordinates and other information to obtain the temporal matching results of semantic features. For geometric features, cross-frame association and trajectory tracking are performed based on their feature identifiers and descriptors to obtain the temporal matching results of geometric features. Finally, the matching results of the two types of features are integrated to generate a temporal feature set containing cross-frame correspondences, providing stable observation data for subsequent pose calculation.

[0070] Step S103: Construct unified constraints based on semantic and geometric features, optimize the camera pose according to the constraints and feature set, and determine the optimized pose.

[0071] In this embodiment, the real-time pose of the camera can be initially calculated based on the initial pose of the camera and the geometric and semantic features in the feature set. Then, a unified composite constraint condition can be constructed based on the semantic and geometric features. The real-time pose can be corrected by the constraint condition so that the real-time pose meets the constraint condition and a more accurate pose is obtained.

[0072] For example, the structure and coordinates of a feature in three-dimensional space can be calculated based on semantic and geometric features, and then the features in three-dimensional space can be reprojected back onto the image plane, with the reprojection error calculated as a constraint.

[0073] For example, the topological consistency of semantic features in three-dimensional space can be used as a constraint to ensure that the same semantic feature (such as traffic poles or signs) has a stable geometric relationship in three-dimensional space in different frames.

[0074] For example, the relative motion speed and direction of the camera can also be inferred by analyzing the temporal position and shape changes of semantic features in multiple frames of images (such as the rate of change of the scale of semantic features in different frames of images and the consistency of motion direction), thereby establishing a constraint between two-dimensional image observation and camera motion parameters.

[0075] In the above embodiments, by extracting semantic and geometric features from multiple frames of images captured by the camera and encoding these features into a unified data structure, temporal matching of the extracted features can be performed to generate a feature set containing temporal information. Then, the camera pose is calculated based on the feature set, and optimized based on joint semantic and geometric constraints, resulting in a more accurate pose. This approach enables deep fusion of geometric and semantic features, effectively improving feature stability and pose accuracy in traffic scenarios where single features are sparse.

[0076] In one embodiment, temporal matching is performed on features extracted from multiple frames of images to generate a feature set containing temporal information, including:

[0077] For the extracted semantic features, cross-frame matching can be performed on each semantic feature, and the successfully matched features can generate temporal information according to the image frame number where the feature is located.

[0078] For example, semantic features in images such as lane lines, traffic signs, and poles can be used to create a separate temporal trajectory for each feature, managing matching and temporal information feature by feature. For instance, the coordinates of a triangular traffic sign in the first frame are {P1(X11,Y11), Q1(X12,Y12), R1(X13,Y13)}, its coordinates in the second frame are {P2(X21,Y21), Q2(X22,Y22), R2(X23,Y23)}, and its coordinates in the i-th frame are {Pi(Xi1,Yi1),Qi(Xi2,Yi2), Ri(Xi3,Yi3)}. Based on the time interval between adjacent frames and the number of image frames, the pixel coordinates of the semantic features can be recorded temporally, and these coordinates can be temporally transformed to the camera coordinate system or other three-dimensional coordinate systems to obtain the temporal trajectory of the semantic feature.

[0079] If an image contains multiple semantic features of the same category, such as two traffic signs, their temporal trajectories can be tracked separately after being distinguished by feature identifiers. This feature-by-feature tracking method can ensure the stability and accuracy of semantic features and avoid interference between different semantic features.

[0080] Based on the extracted geometric features, batch cross-frame matching can be performed on the geometric features in the same image. The successfully matched features can generate temporal information according to the image frame number where the feature is located.

[0081] For example, in the batch cross-frame matching process, the SLAM system, based on the descriptor similarity principle, finds the closest visual counterpart for each geometric feature point in subsequent frames, thereby achieving cross-frame association. Specifically, taking a continuous image sequence as an example, the system first extracts M geometric feature points and their descriptors from the first frame image, generating a matrix A (dimension M×d, where d is the dimension of the descriptor) with descriptors as elements; then, it extracts N feature points and their descriptors from the second frame image, generating a matrix B (dimension N×d). By calculating the similarity matrix between matrices A and B (e.g., using Hamming distance or cosine similarity), the similarity measure between all feature point pairs can be obtained at once. Subsequently, for each feature point in the first frame, the corresponding point with the highest similarity in the second frame is selected to form a preliminary matching pair, with the number of preliminary matching pairs between the first and second frames not exceeding M. After matching is completed, the system uses the matching result of the current frame as a benchmark to continue batch matrix similarity calculation and matching with the third frame, and proceeds sequentially in subsequent frames. The process is transmitted frame by frame in chronological order, which allows the positions of the same feature point in multiple frames to be linked together, forming a continuous trajectory spanning multiple frames.

[0082] By replacing point-by-point matching with matrix-based batch similarity calculation, the efficiency of cross-frame tracking and matching of geometric features can be improved, the continuity of matching can be guaranteed, and a stable data foundation can be provided for subsequent pose optimization.

[0083] In some possible implementations, when forming initial matching pairs, it can be determined whether the similarity between the two geometric features in the matching pair is greater than a preset threshold. If it is greater than the threshold, the matching pair is retained; otherwise, it is discarded. This method can retain more reliable matching results, eliminate false matches, and improve the data quality of the feature set.

[0084] In some possible implementations, batch cross-frame matching of extracted geometric features may also include:

[0085] S201, for two frames in a multi-frame image, the geometric features extracted from the earlier frame are combined into a first set, and the geometric features extracted from the later frame are combined into a second set.

[0086] S202, for any geometric feature in the first set, determine the geometric feature and the geometric feature that is closest to the geometric feature in the second set as a positive matching pair.

[0087] S203, for the geometric features in the second set, determine the geometric features and the geometric features that are closest to the geometric features in the first set as reverse matching pairs.

[0088] S204. If the forward matching pair matches the backward matching pair, then the match is successful.

[0089] For example, the frame with the earlier time sequence is called the first frame, and the frame with the later time sequence is called the second frame. In this case, the descriptors of each geometric feature in the first set can form matrix A, and the descriptors of each geometric feature in the second set can form matrix B. By calculating the descriptor with the maximum similarity between matrix A and matrix B, a positive matching pair can be obtained, and by calculating the descriptor with the maximum similarity between matrix B and matrix A, a negative matching pair can be obtained. For example, element a1 in matrix A represents the descriptor of the geometric feature point with feature label 101 in the first frame image, element a2 in matrix A represents the descriptor of the geometric feature point with feature label 102 in the first frame image, and element b1 in matrix B represents the descriptor of the geometric feature point with feature label 201 in the second frame image; assuming that the element with the highest similarity to a1 in matrix B is b1, then the 101 feature point in the first frame image and the 201 feature point in the second frame image form a positive matching pair; if the element with the highest similarity to b1 in matrix A... If the element in matrix A with the highest similarity to b1 is a1, then the 101 feature points in the first frame and the 201 feature points in the second frame form a reverse matching pair. In this case, both the forward and reverse matching pairs are composed of 101 and 201 feature points, which is consistent with the forward and reverse matching pairs, and the matching is successful. If the element in matrix A with the highest similarity to b1 is a2, then the 102 feature points in the first frame and the 201 feature points in the second frame form a reverse matching pair. In this case, the forward and reverse matching pairs are inconsistent, and the matching is unsuccessful.

[0090] The above-mentioned forward and reverse matching based on the temporal sequence of image frames retains the consistent geometric feature matching results of bidirectional matching and removes inconsistent matching results. In this way, mismatched features can be further removed, thereby improving the data quality of the feature set.

[0091] In one embodiment, before optimizing the camera pose according to constraints and a feature set, the method further includes:

[0092] Based on the camera's shooting angle, any frame of image is divided into regions to obtain multiple image regions. For each image region, the changes in the camera's extrinsic parameters are calculated based on the geometric features within that region, and geometric features that do not meet the preset conditions are eliminated based on the calculated changes in extrinsic parameters.

[0093] In this embodiment, the image can be divided into regions based on the camera's shooting angle to adapt to the characteristics of different regions for differentiated processing. Specifically, the image can be divided into multiple regions such as upper, middle, and lower along the vertical direction. When the camera's shooting angle is nearly parallel to the ground plane, the upper region often corresponds to the sky or distant scenery, with sparse and unstable features; the lower region usually includes the ground, roads, and nearby objects, with denser and more reliable features. Simultaneously, the range of the vertical region division can be adjusted by incorporating the height of ground reference objects (such as houses and plants). Furthermore, the image can also be divided into left, middle, and right regions in the horizontal direction: the middle region is less affected by camera lens distortion, and its feature coordinates are more accurate; the left and right edge regions have greater distortion, and their feature observation errors are usually higher. After division, the camera extrinsic parameter changes are calculated for each region using its internal geometric features. Then, based on the calculation confidence or error distribution of each region, abnormal features with high deviation are removed, thereby improving data quality in the feature processing stage and providing more reliable geometric constraints for subsequent pose optimization.

[0094] In some possible implementations, the camera's extrinsic parameter variations are calculated based on the geometric features within the region, including:

[0095] If the number of geometric features in the image region is greater than or equal to a preset threshold, the intrinsic matrix is ​​used to calculate the camera's extrinsic parameter changes based on the geometric features; if the number of geometric features in the image region is less than the preset threshold, the homography matrix is ​​used to calculate the camera's extrinsic parameter changes based on the geometric features.

[0096] After dividing the image into regions, the number of matching feature points in each region can be counted. If the number is sufficient, the essential matrix and the RANSAC (Random Sample Consensus) algorithm can be used to estimate the relative pose and filter interior points. If the number is insufficient to calculate the essential matrix, the homography matrix and the RANSAC algorithm can be used to filter interior points and remove outliers with abnormal matches. Finally, the interior points selected from each region are accumulated to form a reliable feature set.

[0097] In one embodiment, unified constraints are constructed based on semantic and geometric features, and the camera pose is optimized according to the constraints and feature set, including:

[0098] S301 calculates the real-time pose of the camera based on the initial pose of the camera and the geometric and semantic features in the feature set.

[0099] The core of this step lies in using the temporal trajectory of geometric and semantic features to deduce the changes in the camera's extrinsic parameters, including temporal changes such as camera translation and rotation.

[0100] S302 calculates the three-dimensional coordinates of the geometric and semantic features in the feature set in the world coordinate system.

[0101] In this step, based on the geometric feature point and the real-time pose of the camera, the three-dimensional coordinates of the feature point in the physical world can be calculated using the ray intersection method.

[0102] Figure 2 This is an exemplary schematic diagram of adjacent frame image feature extraction provided in this application. Figure 2 As shown, taking the first and second frames as adjacent frames as an example, the black dashed box represents image plane 1 when the camera captures the first frame image, and the blue dashed box represents image plane 2 when the camera captures the second frame image. Frame 1 and Frame 2 represent the camera's processing viewpoints for the first and second frames, respectively, and C0 and C1 represent the optical centers of the camera when capturing the first and second frames, respectively. It can be seen that the change in the camera's optical center reflects the translational transformation of the camera's pose from the first frame to the second frame, and the coordinates of feature points in the image also change. Based on the camera pose transformation and feature point coordinate transformation, the three-dimensional target points P0 and P1 can be calculated.

[0103] Figure 3 This is a schematic diagram illustrating the calculation of three-dimensional coordinates, provided as an example of this application. Figure 3 As shown in the schematic diagram of feature extraction based on adjacent frame images, spatial triangulation can be performed based on the geometric feature trajectory obtained by camera pose change and feature matching, and the 3D target point P_world in the real world coordinate system can be calculated by ray intersection.

[0104] For example, the matching two-dimensional points of a certain feature trajectory in two frames of images are: The camera poses corresponding to the two frames are respectively and Then the three-dimensional coordinates of this feature point in the physical world It can be calculated using the ray intersection method, and the calculation formula is as follows (1):

[0105] (1)

[0106] In the above formula, Here, K represents the homogeneous pixel coordinates, and K can represent the camera intrinsic parameter matrix. This represents an algorithm for finding the intersection point of two rays with the minimum distance.

[0107] For planar semantic features such as lane lines and ground markings, the inverse pinhole imaging transformation model can be used to solve for the three-dimensional coordinates of these semantic features. The calculation formula is as follows (2):

[0108] (2)

[0109] In the above formula, h is the height of the reference plane.

[0110] For non-planar semantic features, the three-dimensional coordinates can be calculated according to formula (1), which can be referenced from the calculation method of geometric features.

[0111] S303, based on the real-time pose of the camera, reprojects the three-dimensional coordinates onto the image plane acquired by the camera to obtain reprojection features.

[0112] S304 calculates the residual between the reprojected features and the original features in the image plane, and optimizes the real-time pose of the camera with the goal of minimizing the residual.

[0113] The original features are semantic and geometric features extracted from multiple frames of images captured by the camera.

[0114] In the embodiments of this application, a unified graph optimization model can be constructed to incorporate semantic features and geometric features into the same reprojection error framework.

[0115] For geometric constraints, let the camera pose be... 3D coordinates The camera projection model is Then the reprojection residual of the geometric features It can be defined as:

[0116] (3)

[0117] In the above formula, Let be the observed pixel coordinates of the i-th geometric feature in the image.

[0118] For reprojection modeling of semantic constraints, semantic constraints can also be expressed in the form of reprojection residuals. Different types of semantic features (such as lane lines, traffic signs, and poles) can adopt corresponding projection models according to their spatial characteristics.

[0119] For point-like semantic features (such as the base of a rod-shaped object or a corner point of a marker), their three-dimensional coordinates are: Reprojection residual It can be represented as:

[0120] (4)

[0121] For linear semantic features (such as lane lines), these semantic features can be discretized into several sampling points, and the three-dimensional coordinates of the k-th sampling point are: Reprojection residual It can be represented as:

[0122] (5)

[0123] In the above formula, This represents the pixel coordinates of the j-th point semantic feature in the image plane acquired by the camera. This represents the pixel coordinates of the k-th sampling point in the camera-captured image plane after the linear semantic features have been discretized.

[0124] By constructing semantic reprojection residuals such as point-like and line-like structures, we can comprehensively reflect semantic characteristics such as the spatial continuity of lane lines, the scale consistency of traffic signs, and the vertical constraints of pole-like structures.

[0125] After obtaining the reprojection residuals of the aforementioned geometric and semantic features, these residuals can be used as constraints to construct an objective function for a unified reprojection framework that includes both semantic and geometric constraints. By minimizing this objective function, the camera pose can be determined. With feature three-dimensional coordinates By performing joint optimization, semantic and geometric information are coordinated under a unified optimization framework, thereby improving the robustness and accuracy of pose estimation in complex scenes with sparse features, such as intersections.

[0126] In some possible implementations, after joint optimization based on semantic and geometric constraints, the camera's pose sequence based on consecutive image frames can also be optimized. Second-order smoothing is performed, which can include translation smoothing and rotation smoothing.

[0127] For example, translation smoothing can be represented as:

[0128] (6)

[0129] Rotational smoothing can be expressed as:

[0130] (7)

[0131] In the above formula, and These represent the pose parameters after translational smoothing and rotational smoothing, respectively. Smoothing can improve the continuity and stability of the pose trajectory.

[0132] The pose determination method of this application can be applied to scenarios involving visual positioning and environmental understanding, such as autonomous driving, intelligent transportation, robot navigation, and high-precision urban map construction. It can be applied to pure vision solutions as well as to fusion positioning frameworks with multi-sensor systems such as LiDAR and Inertial Measurement Units. The application of this method is illustrated below with a specific example.

[0133] Figure 4This is an exemplary schematic diagram of a pose optimization process based on the fusion of SuperPoint and semantic features, provided for the purposes of this application.

[0134] like Figure 4 As shown, this scheme uses SuperPoint as the geometric feature. This feature has no semantic category attribute, while semantic features can include lane lines, poles, signs, etc. After the process starts, continuous image frames captured by the camera can be input. The AI ​​(artificial intelligence) model extracts SuperPoint and semantic features from the images and stores them in a unified feature pool. Semantic features can be stored according to the identified semantic category, and SuperPoint features can be uniformly encoded and stored as the same semantic category. When performing SLAM localization, the density of semantic features in the image can be used to determine whether the road scene is a feature-sparse scene such as an intersection. If it is not an intersection but a regular road scene with rich semantic features, only semantic features can be used for BA (Bundle Adjustment) to calculate and optimize the pose accuracy. This method is suitable for regular road poses. If it is an intersection scene, semantic features can be used as the main feature, supplemented by SuperPoint features for joint optimization to determine the pose in the intersection scene.

[0135] Figure 5 This is a schematic diagram illustrating a SuperPoint-based pose optimization process, provided as an example in this application. Figure 5 As shown, in road scenes with sparse semantic features, SuperPoint features can be extracted from the image, including the positions (pixel coordinates) and descriptors of two-dimensional feature points in the image. Then, cross-frame matching is performed based on the descriptors, and false matches can be eliminated using RANSAC. Based on the feature trajectory generated by the matching, the SuperPoint points in three-dimensional space can be calculated by spatial triangulating the SuperPoint feature trajectory. Then, the three-dimensional coordinates of the semantic features are projected onto the image plane of the camera according to a unified reprojection framework, with the goal of minimizing the reprojection residual to solve and optimize the pose.

[0136] Figure 6 This is an example of a pose optimization reprojection diagram provided in this application. Figure 6 It contains several dots (such as red and blue hollow dots and purple solid dots), which can represent geometric features at different stages. In the embodiments of this application, determining the camera pose can include three stages. The first stage is to extract geometric features (such as SuperPoint points) and semantic features from the images acquired by the camera. The second stage is to calculate the three-dimensional spatial coordinates of the features based on the features of multiple frames of images and reproject them back to the two-dimensional image plane. The third stage is to optimize based on the reprojection results and the dual constraints of geometric and semantic features until the reprojection results meet the constraints.

[0137] For geometric features, in Figure 6 In the diagram, the red hollow dots represent SuperPoint points directly extracted from the camera images in the first stage; the blue hollow dots represent the reprojection of the 3D coordinates of the SuperPoint points extracted in the first stage onto the image plane before optimization in the second stage; and the purple solid dots represent the reprojection of the SuperPoint points onto the image plane after optimization in the third stage. Figure 6 As shown, before optimization, the initial extracted geometric features and the reprojected geometric features had low overlap (the red hollow dots and blue hollow dots were mostly separated), while after optimization, the initial extracted geometric features and the reprojected geometric features had high overlap (the red hollow dots were basically covered by the reprojected purple solid dots). It can be seen that the optimized pose can better reflect the distribution of geometric features under the camera's view.

[0138] For semantic features, Figure 6 The image contains semantic features categorized into poles, lane lines, and signs. The yellow outline represents the semantic features of poles extracted in the first stage. The blue and green lines running vertically through the image represent the reprojection results of the pole features in the second and third stages, respectively. It can be seen that for the same pole feature, the optimized reprojection result represented by the green line is closer to the yellow outline than the blue line. This indicates that the reprojection result determined based on the optimized pose better reflects the position of the original semantic features in the image from the first stage. Figure 6 The pink solid dots represent the corner points of the semantic features of sign categories and lane line categories extracted in the first stage. The lines with the pink solid dots as endpoints represent the lane line features extracted in the first stage. The blue and green lines around these two types of features represent the reprojection results of these two types of semantic features in the second and third stages, respectively. It can be seen that the reprojection result optimized in the third stage is closer to the position of the original semantic features in the image in the first stage than that in the second stage.

[0139] Based on the above Figure 6 By comparing the reprojection of geometric and semantic features, the pose optimized in the third stage can obtain a reprojection result that is closer to the original spatial distribution of features. This shows that the pose optimization method of this application can improve the accuracy of the pose.

[0140] Figure 7 This is a schematic diagram of the structure of a pose determination device provided in this application, as shown below. Figure 7 As shown, the pose determination device 700 provided in this embodiment includes:

[0141] The extraction module 701 is used to extract semantic and geometric features from multiple frames of images captured by the camera, and encode the semantic and geometric features into a unified data structure.

[0142] The matching module 702 is used to perform temporal matching on features extracted from multiple frames of images based on a unified data structure, and generate a feature set containing temporal information.

[0143] The optimization module 703 is used to construct unified constraints based on semantic and geometric features, optimize the camera pose according to the constraints and feature set, and determine the optimized pose.

[0144] In some possible implementations, the matching module 702 can also be used to: perform cross-frame matching for each extracted semantic feature and generate temporal information of the successfully matched semantic features; and / or, perform batch cross-frame matching for the extracted geometric features and generate temporal information of the successfully matched geometric features.

[0145] In some possible implementations, the matching module 702 can also be used to: for two frames in a multi-frame image, form a first set of geometric features extracted from the earlier frame and form a second set of geometric features extracted from the later frame; for any geometric feature in the first set, determine the geometric feature and its closest geometric feature in the second set as a forward matching pair; for any geometric feature in the second set, determine the geometric feature and its closest geometric feature in the first set as a reverse matching pair; if the forward matching pair and the reverse matching pair are consistent, the matching is successful.

[0146] In some possible implementations, the matching module 702 can also be used to: divide any frame of image into regions according to the shooting angle of the camera to obtain multiple image regions; for each image region, calculate the changes in the camera's extrinsic parameters based on the geometric features within that region, and remove geometric features that do not meet the preset conditions based on the calculated changes in the extrinsic parameters.

[0147] In some possible implementations, the matching module 702 can also be used to: if the number of geometric features in the image region is greater than or equal to a preset threshold, then use the essential matrix to calculate the camera's extrinsic parameter changes based on the geometric features; if the number of geometric features in the image region is less than a preset threshold, then use the homography matrix to calculate the camera's extrinsic parameter changes based on the geometric features.

[0148] In some possible implementations, the optimization module 703 can also be used to: calculate the real-time pose of the camera based on the initial pose of the camera and the geometric and semantic features in the feature set; calculate the three-dimensional coordinates of the geometric and semantic features in the feature set in the world coordinate system; reproject the three-dimensional coordinates onto the image plane acquired by the camera based on the real-time pose of the camera to obtain the reprojected features; calculate the residual between the reprojected features and the original features in the image plane, and optimize the real-time pose of the camera with the goal of minimizing the residual, wherein the original features are the semantic and geometric features extracted from the multi-frame images acquired by the camera.

[0149] In some possible implementations, the optimization module 703 can also be used to discretize the linear semantic features into several sampling points when performing reprojection and calculating residuals.

[0150] The pose determination device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0151] Figure 8 This is a schematic diagram of the structure of an electronic device provided in this application. Figure 8 As shown, the electronic device 80 provided in this embodiment includes at least one processor 801 and a memory 802. Optionally, the device 80 further includes a communication component 803. The processor 801, memory 802, and communication component 803 are connected via a bus 804.

[0152] In a specific implementation, at least one processor 801 executes computer execution instructions stored in memory 802, causing at least one processor 801 to perform the above-described method.

[0153] The specific implementation process of processor 801 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0154] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0155] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0156] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0157] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0158] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0159] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0160] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0161] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0162] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0163] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0164] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0165] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0166] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A pose determination method, characterized in that, include: Semantic and geometric features are extracted from multiple frames of images captured by the camera, and the semantic and geometric features are encoded into a unified data structure. Based on the unified data structure, the features extracted from multiple frames of images are temporally matched to generate a feature set containing temporal information. Based on the semantic features and the geometric features, a unified set of constraints is constructed, and the pose of the camera is optimized according to the constraints and the feature set to determine the optimized pose.

2. The method according to claim 1, characterized in that, The step of performing temporal matching on features extracted from multiple frames of images to generate a feature set containing temporal information includes: For each extracted semantic feature, perform cross-frame matching and generate temporal information of the successfully matched semantic features; and / or, Batch cross-frame matching is performed on the extracted geometric features, and temporal information of the successfully matched geometric features is generated.

3. The method according to claim 2, characterized in that, The batch cross-frame matching of the extracted geometric features includes: For two frames in a multi-frame image, the geometric features extracted from the earlier frame are grouped into a first set, and the geometric features extracted from the later frame are grouped into a second set. For any geometric feature in the first set, the geometric feature and the geometric feature that is closest to the geometric feature in the second set are identified as positive matching pairs; For the geometric features in the second set, the geometric feature and the geometric feature that is closest to the geometric feature in the first set are identified as reverse matching pairs; If the forward matching pair matches the reverse matching pair, then the match is successful.

4. The method according to any one of claims 1 to 3, characterized in that, Before optimizing the camera pose according to the constraints and the feature set, the method further includes: Based on the shooting angle of the camera, any frame of image is divided into regions to obtain multiple image regions; For each image region, the extrinsic parameter changes of the camera are calculated based on the geometric features within that region, and geometric features that do not meet the preset conditions are eliminated based on the calculated extrinsic parameter changes.

5. The method according to claim 4, characterized in that, The calculation of the camera's extrinsic parameter changes based on the geometric features within the region includes: If the number of geometric features in the image region is greater than or equal to a preset threshold, then the essential matrix is ​​used to calculate the extrinsic parameter changes of the camera based on the geometric features; If the number of geometric features in the image region is less than a preset threshold, then the homography matrix is ​​used to calculate the extrinsic parameter changes of the camera based on the geometric features.

6. The method according to any one of claims 1 to 3, characterized in that, The step of constructing unified constraints based on the semantic and geometric features, and optimizing the camera pose according to the constraints and the feature set, includes: The real-time pose of the camera is calculated based on the initial pose of the camera and the geometric and semantic features in the feature set. Calculate the three-dimensional coordinates of the geometric and semantic features in the feature set in the world coordinate system; Based on the real-time pose of the camera, the three-dimensional coordinates are reprojected onto the image plane acquired by the camera to obtain the reprojection features; The residual between the reprojection feature and the original feature in the image plane is calculated, and the real-time pose of the camera is optimized with the goal of minimizing the residual. The original feature is the semantic and geometric features extracted from multiple frames of images acquired by the camera.

7. The method according to claim 6, characterized in that, When performing reprojection and calculating residuals, the linear semantic features are discretized into several sampling points.

8. A pose determination device, characterized in that, include: An extraction module is used to extract semantic and geometric features from multi-frame images captured by a camera, and encode the semantic and geometric features into a unified data structure. The matching module is used to perform temporal matching on features extracted from multiple frames of images based on the unified data structure, and generate a feature set containing temporal information. The optimization module is used to construct unified constraints based on the semantic features and the geometric features, optimize the pose of the camera according to the constraints and the feature set, and determine the optimized pose.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium / computer program product, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7; and / or, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images