Robot 3D point cloud positioning method and system based on multi-view image matching

Abstract

The invention provides a robot 3D point cloud positioning method and system based on multi-view image matching, and relates to the technical field of robot positioning and navigation. The method comprises the following steps: generating a multi-view map image and constructing a database; multi-view images of the robot are collected in real time; performing feature extraction on the acquired multi-view image to obtain a global descriptor vector; performing feature matching on the multi-view image subjected to feature extraction processing and a multi-view map image in a database based on the global descriptor vector; based on a feature matching result, performing pose estimation on a query camera carried by the robot; and based on the pose estimation result, performing optimization processing on the relative pose of the robot in the map. According to the method, 3D point cloud positioning which does not depend on the initial pose, adapts to the complex environment and is efficient and accurate is achieved, and the positioning flexibility and practicability of the robot in the complex environment are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot positioning and navigation, and particularly relates to a robot 3D point cloud positioning method and system based on multi-view image matching. BACKGROUND

[0002] With the rapid development of robot technology, the autonomous navigation and positioning capability of robots in various complex environments has become a key indicator of their intelligent level. Among the many positioning technologies, the global positioning method based on known 3D point cloud maps has attracted widespread attention and research in the robot field due to its ability to provide high precision and rich three-dimensional spatial information. 3D point cloud maps are constructed by collecting three-dimensional data of the environment using sensors such as laser radars and depth cameras, and can accurately describe the shape, position and spatial relationship of objects in the environment, providing detailed navigation reference for robots.

[0003] During the execution of tasks by robots, global positioning aims to determine the precise position and pose (pose) of the robot in the known 3D point cloud map, which is the basis for subsequent path planning, obstacle avoidance and task execution. Accurate pose information can ensure that the robot accurately reaches the target position according to the predetermined route, avoiding problems such as collision and getting lost due to positioning errors, which is crucial for improving the work efficiency and reliability of robots.

[0004] However, the current robot global positioning in the known 3D point cloud map faces the following technical bottlenecks: No initial value, difficult to position and time-consuming: traditional iterative closest point (ICP), normal distribution transformation (NDT) and other point cloud registration algorithms rely heavily on accurate initial pose estimation. In the absence of prior information (i.e. "cold start"), global search is usually required, which is computationally intensive and time-consuming, making it difficult to meet real-time requirements.

[0005] Environment structure degradation leads to matching failure: in long corridors, open squares, warehouses and other environments with sparse features or many repetitive structures, 3D point clouds lack unique geometric features, causing point cloud registration-based algorithms to fail to converge or produce incorrect pose estimates.

[0006] Sensor configuration limits versatility: Many low-cost robot platforms are equipped with only monocular cameras, RGB-D cameras or 2D laser radars, which cannot directly obtain 3D point cloud data for matching, limiting their positioning capability in 3D maps.

[0007] The prior art such as ScanContext, PointNetVLAD, etc. performs place recognition through a global descriptor, which provides a train of thought for solving the initial value-free positioning, but still has limitations: the former is sensitive to rotation and is not robust to point cloud density changes; the latter, as a deep learning model, needs a large amount of labeled data for training, and the model is large, which makes it difficult to deploy on embedded devices. SUMMARY

[0008] In view of the technical problems that the robot in the prior art faces difficulties in initial value-free positioning when performing global positioning in a known 3D point cloud map, the positioning is time-consuming, the matching fails due to environmental structure degradation, and the sensor configuration is limited, the application provides a robot positioning method which is highly universal, robust and independent of the initial pose. By simulating the multi-view observation behavior of humans, the 3D point cloud map and the real-time perception data of the robot are uniformly converted into image representations under multi-view, and a rotation-robust image feature matching technology is used to achieve fast positioning.

[0009] To achieve the above object, the technical scheme adopted by the application is: A robot 3D point cloud positioning method based on multi-view image matching, the method comprising: generating multi-view map images and constructing a database; real-time acquisition of robot multi-view images; feature extraction is performed on the acquired robot multi-view images to obtain a global descriptor vector; based on the global descriptor vector, feature matching is performed on the multi-view images processed by feature extraction and the multi-view map images in the database; based on the feature matching result, pose estimation is performed on the query camera carried by the robot; based on the pose estimation result, the relative pose of the robot in the map is optimized.

[0010] On the other hand, the application also provides a robot 3D point cloud positioning system based on multi-view image matching, which comprises a memory for storing computer program instructions and a processor for executing program instructions, wherein when the computer program instructions are executed by the processor, the system triggers the execution of the above-mentioned robot 3D point cloud positioning method based on multi-view image matching.

[0011] Compared with the prior art, the application has the following advantages: 1. Completely get rid of the dependence on the initial pose Traditional point cloud registration algorithms often require accurate initial value estimation when used for robot localization. In "cold start" scenarios, the lack of reliable initial pose information may prevent the algorithm from working properly. The present invention bypasses the dependence on initial pose by achieving coarse localization through image retrieval. The system converts 3D point cloud maps and real-time robot data into multi-view image representations and uses rotationally robust image feature matching techniques for localization. Without prior knowledge of the initial position and attitude, the system can quickly determine the approximate position and lay the foundation for precise localization. This feature makes it suitable for scenarios that require fast start-up localization, improving the flexibility and practicality of robot localization.

[0012] 2. Strong environmental robustness The environment in which robots operate is complex and diverse, and single-view image features may not be sufficient to prevent localization failure. The present invention solves this problem by integrating multi-view information. When a robot collects multi-view data, if the features of a certain view (e.g., a straight view of a corridor) are missing, other views (e.g., a top view or a side view) can provide effective matching information. The system analyzes and matches multi-view images comprehensively, and uses the complementary nature of the views to improve the reliability and success rate of localization.

[0013] 3. High accuracy Single-view matching can be prone to errors in complex or feature-similar scenarios. The present invention uses multi-view matching and consistency checking to significantly reduce the rate of false matches. During feature matching, the system calculates the global descriptor similarity and uses methods such as approximate nearest neighbor indexing for fast matching, followed by multiple reordering and verification to select true matching pairs. In the pose estimation stage, the relative rotation is estimated based on rotationally invariant features, the translation vector is calculated after filtering the global rotation, and finally the multi-view pose results are robustly fused. This multi-level, multi-view mechanism makes localization more accurate and reliable, meeting high-precision requirements.

[0014] 4. More extensive compatibility with sensors and platforms The present invention relies on image processing and has low dependence on sensors and platforms. Robots equipped with cameras and capable of motion can use it to complete initial localization and relocalization. Traditional localization methods often require robots to be equipped with expensive 3D laser radars and other high-precision sensors, which increases the cost and limits their application on low-cost robot platforms. The present invention uses image information collected by cameras for localization, eliminating the need for complex sensors and significantly reducing the cost of the localization system. Moreover, the image processing algorithm is highly versatile and can be adapted to different types of platforms such as wheeled robots, legged robots, and drones. As long as a camera can be mounted to obtain multi-view images, accurate localization can be achieved. This extensive compatibility makes the present invention widely applicable and can promote the popularization and development of robot technology in more fields.

[0015] 5. High computational efficiency The robot positioning needs to have real-time performance, the image matching algorithm is highly optimized, the rotation invariant network adopts lightweight design, a large amount of image data can be quickly processed, and the calculation time and resource consumption are reduced. In the feature extraction stage, the pre-training network ensures the quality and quickly calculates the rotation invariant descriptor; in the matching stage, the approximate nearest neighbor search improves the speed. The optimized algorithm can run in real time (>10Hz) on an embedded platform, meets the requirements, and is suitable for high real-time scenarios such as automatic driving and logistics distribution.

[0016] 6. Low training data requirement The traditional positioning method requires expensive 6-DOF accurate pose labels for network training, and the cost is high and difficult to obtain. The network training of the present application only needs weak supervision, and it can be determined whether the image pair is from a nearby place, without the need for accurate pose information. This reduces the training data requirement and cost, and data can be obtained by simple labeling, which is easy to collect samples, speed up training and reduce application threshold. Moreover, weak supervision enables the model to have generalization ability, adapt to different environments, and improve practicality and scalability.

[0017] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a flow chart of a robot 3D point cloud positioning method based on multi-view image matching according to the present application; Figure 2 is a schematic diagram of a rotation invariant module design according to the present application. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application, so that the purpose, characteristics and advantages of the present application can be better understood. It should be understood that the embodiments shown in the drawings are not a limitation on the scope of the present application, but are only to illustrate the essential spirit of the technical solutions of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0020] Unless the context requires otherwise, throughout the specification and claims, the words "comprise" and variations such as "comprising" and "comprises" are to be construed in an open, inclusive sense, that is, as "including, but not limited to".

[0021] Reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0022] As used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the content clearly dictates otherwise. It should be noted that the term "or" is generally employed in its sense including "and / or" unless the content clearly dictates otherwise.

[0023] In the following description, for purposes of clarity, directional terms are used to describe the orientation of the present application. However, it will be understood that the directional terms are used for convenience only and do not necessarily require a particular orientation of the present application.

[0024] The implementation details of the embodiments of the present application will be specifically described below with reference to the accompanying drawings. The following implementation details are provided for the convenience of understanding only and are not essential for implementing the present application.

[0025] The present application provides a robot 3D point cloud positioning method and system based on multi-view image matching, wherein, as shown in the figure, the specific steps of the unit method are as follows: Figure 1 S1, generate multi-view map images and construct a database. S1, generate multi-view map images and construct a database.

[0026] First, set the global point cloud map M = {p i |p i R 3 as the reference environment representation. In order to realize accurate positioning under multi-view, define V fixed observation angles, typically set V = 6, covering front, back, left, right, up and down six directions, to fully capture environmental information. For each preset view v, define a virtual camera coordinate system, which is connected to the coordinate system of the robot itself through the transformation matrix , at the same time, the transformation relationship from the current pose of the robot to the map is used to associate the virtual camera coordinate system with the global point cloud map M in space, to ensure the accuracy of view conversion.

[0027] Next, we proceed to the specific image generation process. For each key location k in the map, starting from each viewpoint v, the following operations are performed to construct a multi-view image database: First, a local point cloud patch Pk is cropped according to the viewpoint direction. Taking the front view as an example, a cube region with sides of 40 meters on all sides is selected as the cropping range to retain sufficient environmental details. Subsequently, this 3D local point cloud patch Pk is projected onto a 2D plane to generate the corresponding image representation. In this process, the physical size of the projection grid is set to 0.2 meters. Therefore, a 40-meter-side 3D local point cloud patch will be converted into an image with a resolution of 800×800 pixels, with each pixel corresponding to a tiny area in the environment.

[0028] Furthermore, to effectively represent the density information of the point cloud in the image, a rule for calculating pixel values ​​is defined: For each cell (i.e., each pixel in the image), the number of point clouds contained within it is counted. If the number of points in a cell exceeds 2250, the pixel value is set to 225 (i.e., near the maximum value of the grayscale image, to represent a high-density region); if the number of points is less than 2250, the result of dividing the number of points by 10 and rounding down is used as the pixel value. In this way, the point cloud density is linearly mapped within the grayscale range of 0~255 to generate a grayscale image with environmental features. .

[0029] Ultimately, an image of 800×800 pixels with a grayscale range of 0~255 was obtained. This image not only captures the geometry of the environment but also implicitly encodes point cloud density information through grayscale values. (This image...) Together with the transformation matrix used for viewpoint conversion And the robot's pose information on the map. These images are stored together in database D, forming a multi-view map image database that can be queried and matched later.

[0030] Each image in the database Both represent the environmental view of key location k from a specific viewpoint v on the map, used for comparison with images acquired by the robot in real time; while the transformation matrix With pose information This serves as the basis for coordinate system transformation, used to calculate the robot's spatial position and attitude on the map, providing a solid data foundation for subsequent localization and navigation tasks.

[0031] S2. Real-time acquisition of multi-view images of the robot.

[0032] When a robot is in an unknown pose on a map, a series of operations are required to achieve real-time acquisition, feature extraction, and accurate matching of multi-view images in order to determine the robot's precise position and orientation on the map. The specific process of real-time acquisition of multi-view images of the robot is as follows: When the robot is in an unknown pose, it mimics human surround-viewing behavior by rotating its motors to acquire images from various perspectives. This surround-viewing behavior aims to comprehensively capture information about the robot's surroundings, similar to how humans turn their heads to observe their surroundings in unfamiliar environments to gain a more complete understanding of the scene. Images from each perspective... Each image contains environmental features in a specific direction, which are crucial for subsequent matching with pre-stored images in the map. The data will be used as input data and compared with images of various views of the map pre-stored in database D to determine the robot's position and orientation.

[0033] S3. Extract features from the acquired multi-view images of the robot to obtain global descriptor vectors.

[0034] Using a pre-trained feature extraction network N Each acquired image is processed. The network contains two key modules: the rotationally equivalent module (REM) and the rotationally invariant module (RIM).

[0035] The schematic diagram of the rotational equivariant module is as follows: Figure 2 As shown, the rotation equivariance module pre-processes the image by rotating it at different angles. The purpose of this is to ensure that the rotationally equivariant features obtained after rotation correspond to the rotation angle of the image. For example, when the image is rotated by a certain angle, the features extracted by this module will change accordingly, and this change has a specific correspondence with the rotation angle. This characteristic allows for more accurate capture of the image's rotation information when processing images at different rotation angles, providing a foundation for rotation calculations in pose estimation.

[0036] Rotation-Invariant Module: Rotation-invariant descriptors are further obtained based on rotation-equivariant features. The core idea is that they are unaffected by image rotation transformations. Taking vectors as an example, the magnitude of a vector does not change with rotation, and rotation-invariant descriptors borrow this property. In specific implementation, rotated views are generated for the input image at a preset set of angles, and then features from these rotated views are extracted using a convolutional backbone network with shared weights. Next, based on rotation-equivariant features, an inverse rotation alignment operation is used to adjust the features extracted at different rotation angles to the same reference direction. Finally, aggregation is performed along the angle dimension to obtain rotation-invariant descriptors. Spatial information is then compressed through global pooling, and after adjusting the feature dimensions through a mapping layer, the output vector is L2 normalized to obtain a normalized global descriptor vector. This global descriptor vector accurately represents the features of the image and is unaffected by image rotation, providing a stable and reliable feature representation for subsequent image matching.

[0037] S4. Based on global descriptor vectors, perform feature matching on multi-view images.

[0038] During feature matching, images acquired from various viewpoints are first processed by a rotation-invariant module to obtain rotation-invariant descriptors. Then, through angle aggregation, global pooling, and a mapping layer, followed by L2 normalization, a numerical vectorized representation—the global descriptor vector (i.e., the unitized global representation of the rotation-invariant descriptor)—is formed. The global descriptor vector of the query image is matched with the pre-calculated and stored global descriptor vectors of the map images from each viewpoint in S1. During the matching process, the similarity between two global descriptor vectors is measured by calculating the similarity (e.g., inner product / cosine similarity) or distance (e.g., Euclidean distance). Under unitized conditions, higher similarity or smaller distance indicates a better match. To achieve fast and accurate matching, the process consists of three levels: Measurement—directly comparing global descriptor vectors to obtain similarity; Retrieval / Acceleration—performing nearest neighbor retrieval of global descriptor vectors based on vector indexes to obtain Top-K candidates; Re-ranking and Consistency Verification—re-ranking candidates and combining geometric consistency verification with local features to finally determine the matching result. The three levels of the fast matching process are as follows: 1) Metrics: Similarity or distance is measured between one or more global descriptor vectors and the set of global descriptor vectors in the library. Commonly used metrics include cosine similarity, Euclidean distance, etc., and learned metrics can also be used. These metrics can yield candidate matches sorted by score, providing a foundation for subsequent retrieval and filtering.

[0039] 2) Retrieval / Acceleration: To avoid performing full pairwise comparisons of all global descriptor vectors in the library and improve matching efficiency, an approximate nearest neighbor retrieval method is adopted. Specifically, there are two optional methods: Graph indexing based on hierarchical small-world graphs: Global descriptor vectors in the graph library are constructed into a hierarchical small-world graph structure. Candidate descriptors similar to the query descriptor are quickly located through the graph's edge relationships. This indexing structure leverages the characteristics of small-world networks, enabling the finding of similar nodes within a shorter path, thereby improving retrieval speed.

[0040] Vector indexing based on inverted lists and (optional) product quantization: First, an inverted index is created on the descriptors in the library, grouping descriptors with similar features into the same inverted list. Simultaneously, product quantization techniques can be used to compress and quantize descriptors, reducing storage space and computational load. During querying, candidate descriptors are quickly located using the inverted list, and then precise distance recalculation is performed on the candidate set to obtain the final Top-... k There are several candidate matches. The index structure parameters here (such as the number of edges, search width, number of inverted clusters, number of probe clusters, and number of quantized codewords) can be adjusted according to the actual application scenario and performance requirements, and optimized without deviating from the spirit of the invention.

[0041] 3) Reordering and Consistency Verification: Perform one or more reordering and consistency verification operations on candidate matches to improve matching reliability. Specific methods include ratio testing, nearest neighbor filtering, multi-perspective consistency voting, and relevance verification or geometric consistency verification on angle alignment features. Ratio testing compares the score ratio of the best and second-best matches among candidate matches to filter out reliable matches with significant score differences; nearest neighbor filtering requires that the query descriptor and candidate descriptors are each other's nearest neighbors to eliminate some false matches; multi-perspective consistency voting uses matching results from multiple perspectives for comprehensive judgment, and only when multiple perspectives agree that a pair of matches is reliable is it adopted as the final matching result; relevance verification or geometric consistency verification on angle alignment features further checks the geometric and feature-related rationality of the matching results to ensure matching accuracy. Among these, Top- k This means "ranking the results by similarity (or distance)". k "Top-1" means that if the nearest neighbor is selected, then the candidate is selected. k ( k ≥2) means take the nearest previous one. k A neighbor.

[0042] S5. Based on the feature matching results, perform pose estimation on the query camera mounted on the robot.

[0043] After obtaining candidate matches between the query viewpoint and the map viewpoint, the relative rotation of each candidate pair is first estimated based on rotational isovariance features. Consistency screening is then performed using either Weighted Random Sample Consistency (RANSAC) or circular statistical methods to obtain the global rotation. The Weighted RANSAC method eliminates outlier matching point pairs through random sampling and consistency checks, thereby improving the accuracy of rotation estimation. The circular statistical method uses statistical analysis of the geometric relationships of the matching point pairs after rotation to screen for rotation angles that meet consistency requirements. Given the rotation, two methods are used to calculate the translation: Translation direction determination based on 2D–2D geometric constraints using local features: When depth information is unavailable, the translation direction between two images is calculated using local feature points in the image and 2D–2D geometric constraints, such as epipolar geometry. This method can only determine the direction of translation, not the scale of the translation.

[0044] When a map view is associated with depth / 3D points, a 2D–3D Perspective-n-Point (PnP) algorithm is used to obtain a scaled translation vector. If the map view is associated with depth information or 3D points, the correspondence between feature points in the 2D image and 3D points can be used to solve for the camera's position and pose using the PnP algorithm, thus obtaining a scaled translation vector. Here, PNP (Perspective-n-Point) represents perspective-n-point, used when several 3D points are known. Given 2D pixel correspondences, determine the camera position and orientation.

[0045] Finally, robust fusion / joint optimization is performed on the pose results of multiple viewpoint pairs to obtain the final pose of the query camera. The position is calculated using a weighted average method, and the pose is calculated using a quaternion-weighted average method. The weighted average method calculates position and pose by weighting them according to the matching reliability and contribution of different viewpoint pairs, thereby improving the accuracy of the final pose estimation.

[0046] S6. Based on the pose estimation results, optimize the robot's relative pose in the map.

[0047] In reality, multi-view images and multi-view map images will yield multiple similar rotations and displacements. An optimization equation can be established, or a simple averaging method can be used, to obtain an optimized relative pose. Position was calculated using a weighted average method, and attitude was calculated using a quaternion-weighted average method. This was then processed through a database. D The transformation matrix of the map image recorded in the database and the robot's pose to the map are compared with the optimized relative pose. Multiplication (i.e., coordinate system transformation) is used to calculate a correct and accurate robot pose on the map. The optimization equation can be optimized using robust least squares, and its functional form is as follows: in, R For the final rotation, t For the final translation; The sample weight can be determined by factors such as similarity, visibility, and reprojection error. The larger the weight, the greater the contribution of the sample to the final pose estimation. is a robust kernel function, its function is to resist outliers. When there are some matching point pairs with large errors, the robust kernel function can reduce the impact of these points on the optimization results; ||·|| represents the norm; The parameters balancing the dimensions of rotation and translation are used to adjust their relative importance in the optimization process. By solving this optimization equation, a more accurate and stable robot pose in the map can be obtained.

[0048] The present invention also provides a robot 3D point cloud localization system based on multi-view image matching. The system includes a memory for storing computer program instructions and a processor for executing the program instructions. When the computer program instructions are executed by the processor, the system is triggered to execute the aforementioned robot 3D point cloud localization method based on multi-view image matching.

[0049] This invention proposes a robot 3D point cloud localization method and system based on multi-view image matching. By generating multi-view map images to build a database, the system acquires multi-view images of the robot in real time and extracts its rotation-variable and invariant features. It uses fast matching and consistency verification to determine the image correspondence, and then estimates and optimizes the robot pose. This achieves efficient and accurate 3D point cloud localization that does not depend on the initial pose, adapts to complex environments, and has strong environmental robustness, high accuracy, wider compatibility with sensors and platforms, high computational efficiency, and low training data requirements, significantly improving the robot's localization flexibility and practicality in complex environments.

[0050] Although the present invention has been described in detail with reference to the accompanying drawings and preferred embodiments, the invention is not limited thereto. Various equivalent modifications or substitutions can be made to the embodiments of the invention by those skilled in the art without departing from the spirit and essence of the invention. Such modifications or substitutions should all fall within the scope of the invention, or any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the invention should be covered within the protection scope of the invention. Therefore, the protection scope of the invention should be determined by the scope of the claims.

Claims

1. A robot 3D point cloud localization method based on multi-view image matching, characterized in that, The method includes: Generate multi-view map images and build a database; Real-time acquisition of multi-view images of the robot; Feature extraction is performed on the acquired multi-view images of the robot to obtain a global descriptor vector; Based on global descriptor vectors, feature matching is performed between the multi-view images that have undergone feature extraction and the multi-view map images in the database. Based on the feature matching results, the pose of the query camera mounted on the robot is estimated; Based on the pose estimation results, the robot's relative pose in the map is optimized.

2. The method according to claim 1, characterized in that, The generation of multi-view map images and the construction of the database specifically include: First, set the global point cloud map M = {p i |p i R 3 } represents the baseline environment; V fixed viewing angles are defined, and for each viewing angle v, a virtual camera coordinate system is defined, wherein the virtual camera coordinate system is transformed by a transformation matrix. Connect to the robot's own coordinate system; utilize the robot's current pose. The transformation relationship to the map is used to spatially associate the virtual camera coordinate system with the global point cloud map M; Then, for each key location k in the map, starting from each viewpoint v, the following operations are performed to construct a multi-view image database: first, a three-dimensional local point cloud block Pk is cropped according to the viewpoint direction; then, the three-dimensional local point cloud block Pk is projected onto a two-dimensional plane to generate a corresponding two-dimensional image representation, where each pixel in the two-dimensional image corresponds to a tiny region in the environment.

3. The method according to claim 2, characterized in that, Furthermore, to express the density information of the point cloud in the two-dimensional image, a rule for calculating pixel values ​​is defined: for each pixel, the number of point clouds it contains is counted. If the number of point clouds exceeds 2250, the pixel value is set to 225; if the number of point clouds is less than 2250, the pixel value is obtained by dividing the number of points by 10 and rounding down. In this way, the point cloud density is linearly mapped within the grayscale range of 0 to 255 to generate a grayscale image with environmental features. ; Ultimately, an image with a grayscale range of 0 to 255 was obtained. The image Together with the transformation matrix used for viewpoint conversion And the robot's pose information on the map. These images are stored together in database D, forming a multi-view map image database; each image in database D... Both represent the environmental view of the key location k from a specific viewpoint v on the map.

4. The method according to claim 3, characterized in that, The real-time acquisition of multi-view images of the robot specifically includes: When the robot is in an unknown pose, it is represented by a homogeneous matrix Tq. By rotating the motor, it imitates the human's surround-view behavior and acquires images from different perspectives. Each image contains environmental features in a certain direction.

5. The method according to claim 4, characterized in that, The step of extracting features from the acquired multi-view images of the robot to obtain a global descriptor vector specifically includes: Using a pre-trained feature extraction network N The network processes each acquired image. N It contains two modules: a rotationally equivariant module and a rotationally invariant module; Among them, the rotation isomorphic module pre-processes the image by rotating it at different angles, and the rotation isomorphic features obtained after the rotation processing can correspond to the rotation angle of the image. The rotation-invariant module generates a rotated view of the input image at a preset set of angles, and then uses a convolutional backbone network with shared weights to extract features from the rotated view. Next, based on the rotation-invariant features, the features extracted at different rotation angles are adjusted to the same reference direction through an inverse rotation alignment operation. Finally, the features are aggregated in the angle dimension to obtain a rotation-invariant descriptor, and then the spatial information is compressed through global pooling. After the feature dimension is adjusted by the mapping layer, the output vector is L2 normalized to obtain a normalized global descriptor vector.

6. The method according to claim 5, characterized in that, The feature matching process, based on global descriptor vectors, between the feature-extracted multi-view images and the multi-view map images in the database specifically includes: When performing feature matching, the images acquired from each viewpoint are first calculated. The global descriptor vector is generated, and then the global descriptor vector is matched with the global descriptor vectors pre-calculated from the multi-view map images in the database. During the matching process, the similarity or distance between the two global descriptor vectors is calculated to measure their similarity. The higher the similarity or the smaller the distance, the better the match.

7. The method according to claim 6, characterized in that, The feature matching process is specifically divided into three levels: measurement, retrieval or acceleration, and reordering and consistency verification. 1) Measurement: Perform similarity or distance measurement on one or more global descriptor vectors and the set of global descriptor vectors of multi-view map images in the database; obtain candidate matches sorted by score through the measurement; 2) Retrieval or Acceleration: To avoid performing full pairwise comparisons of global descriptor vectors from multi-view map images in the database and to improve matching efficiency, an approximate nearest neighbor retrieval method is adopted, specifically in two ways: Graph indexing based on hierarchical small-world graphs: Global descriptor vectors from multi-view map images in the database are constructed into a hierarchical small-world graph structure. Candidate descriptors similar to the query descriptor are quickly located through the graph's edge relationships; or Vector indexing based on inverted lists and product quantization: First, an inverted index is performed on the global descriptor vectors of multi-view map images in the database, grouping descriptors with similar features into the same inverted list. Simultaneously, product quantization technology is used to compress and quantize the descriptors, reducing storage space and computational load. During querying, candidate descriptors are quickly located using the inverted list, and then precise distance recalculation is performed on the candidate set to obtain the final Top-k candidate matches. Here, Top-k means selecting the k candidates with the lowest similarity or distance, Top-1 means only the nearest neighbor is needed, and Top-k means selecting the k nearest neighbors, where k is an integer greater than or equal to 2. 3) Reordering and Consistency Verification: Perform one or more reordering and consistency verification operations on candidate matches. Specific methods include ratio testing, nearest neighbor filtering, multi-perspective consistency voting, and relevance verification or geometric consistency verification on angle alignment features. The ratio testing compares the score ratio of the best and second-best matches among the candidate matches to filter out reliable matches with significant score differences. The nearest neighbor filtering requires that the query descriptor and candidate descriptors are each other's nearest neighbors to eliminate some false matches. The multi-perspective consistency voting uses matching results from multiple perspectives for comprehensive judgment; only when multiple perspectives consider a pair of matches reliable is it adopted as the final matching result. The relevance verification or geometric consistency verification on angle alignment features further checks the geometric and feature-related rationality of the matching results to ensure the accuracy of the matching.

8. The method according to claim 7, characterized in that, The pose estimation of the query camera mounted on the robot based on the feature matching results specifically includes: After obtaining the feature matching results, the relative rotation of each candidate pair is first estimated based on the rotation equivariance feature, and the consistency screening is performed by weighted random sampling consistency or circular statistics method to obtain the global rotation. Given the rotation, two methods are used to determine the translation: 1) Determining the translation direction based on 2D–2D geometric constraints of local features: When there is no depth information, the translation direction between two images is calculated by using local feature points in the image and through 2D–2D geometric constraints. 2) When the map view is associated with depth or 3D points, the 2D–3D PnP algorithm is used to obtain the translation vector with scale: If the map view is associated with depth information or 3D points, the correspondence between feature points in the 2D image and 3D points is used to solve the position and pose of the camera through the PnP algorithm, thereby obtaining the translation vector with scale. Finally, robust fusion or joint optimization is performed on the pose results of multiple viewpoints to obtain the final pose of the query camera; among them, the position uses a weighted average method and the pose uses a quaternion weighted average method.

9. The method according to claim 8, characterized in that, The optimization of the robot's relative pose in the map based on the pose estimation results specifically includes: Multi-view images and multi-view map images will yield multiple similar rotations and displacements. By establishing an optimization equation, the optimized relative pose can be obtained. Then through the database D The transformation matrix of the map image and the robot's pose to the map are recorded, and the optimized relative pose is obtained. Multiplying these values ​​yields the correct robot pose on the map; the optimization equation employs robust least squares optimization, and its functional form is as follows: in, R For the final rotation, t For the final translation; The sample weights are denoted by , where a larger weight indicates a greater contribution of the sample to the final pose estimation. For robust kernel functions, their function is to resist outliers. When there are some matching point pairs with large errors, the robust kernel function reduces the impact of these points on the optimization results; ||·|| represents the norm; The parameters are used to balance the dimensions of rotation and translation, and to adjust the relative importance of rotation and translation in the optimization process.

10. A robot 3D point cloud localization system based on multi-view image matching, the system comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein, When the computer program instructions are executed by the processor, the system is triggered to execute the robot 3D point cloud localization method based on multi-view image matching as described in any one of claims 1 to 9.

Images