A fast initialization method for point cloud image cross-modal matching constraint

Abstract

The application discloses a kind of point cloud image cross-modal matching constraint fast initialization method, specifically relates to mobile robot positioning initialization field, for solving the problem that image feature is easy to be ambiguous when starting at any position under repeated texture scene, point cloud search range is big, rough positioning is easy to lose accuracy and initialization process is difficult to consider speed and reliability;The method is by projecting alignment to prior image sequence and prior point cloud data, constructs cross-modal training set and trains cross-modal semantic mapping model, generates the mixed feature map bound with visual key frame and prior pose node, then extracts current geometric perception visual descriptor according to real-time environment image, completes target key frame retrieval, matching discrimination and rough global initial pose solution, and combined with current real-time point cloud data, local rigid registration under limited search is implemented, and the output is accurate six-degree-of-freedom repositioning pose.

Description

Technical Field

[0001] This invention relates to the field of mobile robot localization initialization, and more specifically, to a fast initialization method for cross-modal matching constraints of point cloud images. Background Technology

[0002] Currently, mobile robots in large-scale industrial warehousing or complex corridor applications typically rely on simultaneous localization and mapping (SMR) technology for spatial perception and autonomous navigation. In actual operations, the robot first needs to use multiple sensors to collect environmental information and create a global spatial map of the target scene. After mapping, to achieve relocalization, human intervention is often required to return the robot to the initial mapping starting point, using the known spatial coordinates of the starting point as initial prior values ​​to initialize the subsequent localization module. However, when the application site is vast and contains a large number of repetitive textures such as shelves, load-bearing columns, or white walls with extremely similar appearances, if the robot is required to power on and start navigation directly at any unknown location on the map, the system must perform a full-range search and matching of the real-time environmental features extracted at the current location with the entire massive global map database. Due to the large scale of the operation scenario, remotely controlling the robot to return to a fixed starting point every time the system restarts would greatly limit the operational efficiency and deployment flexibility of the equipment. Therefore, exploring a technical solution that can adapt to complex environments and achieve rapid and accurate localization initialization at any location on the map has become a key fundamental requirement in the field of autonomous navigation and multi-source sensor fusion.

[0003] To address the need for arbitrary location activation in the aforementioned scenarios, existing relocalization techniques typically rely on traditional visual bag-of-words models for image similarity comparison. For example, patent document CN112509027A, titled "Relocalization Method, Robot, and Computer-Readable Storage Medium," provides an initial spatial pose estimation scheme based on pure visual feature retrieval. However, this existing technology suffers from a significant drawback in repetitive texture environments, easily falling into local feature ambiguity. This deficiency arises because traditional visual feature extraction heavily relies on local pixel grayscale gradient changes. When the robot is in an environment with highly homogenized features and lacking salient markers, the visual matching system, lacking the rigid constraints of three-dimensional spatial geometric topology, may incorrectly identify two visually similar but physically distant locations as the same position. Once such severe mismatches due to the lack of cross-modal structural information occur, the initial pose candidate calculated by the perspective multi-point localization algorithm will suffer catastrophic spatial coordinate jumps. When an initial pose with a large geometric coordinate deviation is used as prior information and fed into the subsequent laser and inertial joint optimization process, it not only fails to guide the joint optimization algorithm to quickly converge to the accurate real pose, but also completely destroys the original mathematical consistency of local ranging. This ultimately leads to the divergence of the system's underlying optimization, serious misalignment of the global point cloud map, and even causes the robot to completely lose control during navigation, making it impossible to land safely and reliably and achieve global relocalization at any location.

[0004] To address the aforementioned problems, a technical solution is provided. Summary of the Invention

[0005] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a rapid initialization method for cross-modal matching constraints of point cloud images. This method constructs a cross-modal training set and trains a cross-modal semantic mapping model by performing projection alignment on prior image sequences and prior point cloud data. It further generates a hybrid feature map bound to visual keyframes and prior pose nodes, extracts the current geometric perception visual descriptor based on the real-time environment image, completes target keyframe retrieval, matching discrimination, and coarse global initial pose solution, and performs local rigid registration under constrained search using the current real-time point cloud data. This outputs accurate six-DOF relocation pose and localization results, forming a complete initialization chain to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: S1: Obtain prior image sequences and prior point cloud data, perform inverse ray tracing and projection alignment, construct a cross-modal training set, train a cross-modal semantic mapping model, and construct a hybrid feature map; S2: Acquire real-time environment images, input the real-time environment images into the image encoder branch of the cross-modal semantic mapping model, and generate the current geometric perception visual descriptor by combining the attention weight matrix; S3: Map the current geometric perception visual descriptor to the hybrid feature map, lock the target keyframe, establish feature matching point pairs, and obtain the rough global initial pose after discrimination by the adaptive neural fuzzy inference system; S4: Obtain the current real-time point cloud data, use the coarse global initial pose as the input of the absolute reference initial value to iterate the nearest point joint optimization framework, complete the local rigid registration within the search and matching space limited by the hybrid feature map, and output the accurate six-degree-of-freedom relocalization pose.

[0007] Further, step S1 includes performing inverse ray tracing on the two-dimensional pixel features in the prior image sequence based on the camera intrinsic parameter matrix and the extrinsic parameter calibration matrix of the LiDAR to the camera, and projecting and aligning them with the three-dimensional spatial coordinates in the prior point cloud data to construct a cross-modal training set; inputting the cross-modal training set into a dual-branch encoder network to train a cross-modal semantic mapping model; and then binding the geometric perception visual descriptors output from the prior image sequence with the corresponding visual keyframes and prior pose nodes to construct a hybrid feature map.

[0008] Furthermore, in step S1, projection alignment starts with the two-dimensional pixel features in a single frame image, establishes the correspondence between pixel observations and three-dimensional spatial points according to a unified world coordinate reference, extracts grayscale texture fragments around the aligned pixel positions, extracts local point set fragments around the aligned three-dimensional spatial points, and forms cross-modal correspondence units together with grayscale texture fragments, local point set fragments and corresponding three-dimensional spatial points, and then gathers all cross-modal correspondence units to form a cross-modal training set.

[0009] Further, step S2 includes: after acquiring the real-time environment image, converting the real-time environment image into a normalized matrix consistent with the input dimension of the image encoder branch, and inputting the normalized matrix separately into the image encoder branch of the cross-modal semantic mapping model to perform forward inference, extracting the deep convolutional feature tensor and attention weight matrix, and then assigning the three-dimensional structural prior probability to the high gradient response region in the real-time environment image according to the attention weight matrix to generate the current geometric perception visual descriptor.

[0010] Furthermore, in step S2, the current geometric perception visual descriptor generation process includes: constructing a focus consistency map based on the attention weight matrix, constructing a gradient magnitude matrix based on the normalization matrix, constructing a three-dimensional structure prior probability map based on the focus consistency map and the gradient magnitude matrix, then using the three-dimensional structure prior probability map to perform structure modulation on the deep convolutional feature tensor, and performing channel convergence within the structure pixel region to obtain the current geometric perception visual descriptor.

[0011] Furthermore, step S3 includes mapping the current geometric perception visual descriptor to the high-dimensional semantic space corresponding to the hybrid feature map, locking the target keyframe from the candidate keyframe set based on Euclidean distance, extracting the structural anchor point set from the cross-modal corresponding units associated with the target keyframe, extracting the current local feature set on the real-time environment image, and filtering the feature matching point pair set based on bidirectional nearest neighbor matching and random sampling consensus algorithms.

[0012] Furthermore, in step S3, the mapping depth span range rate is calculated for the spatial depth values ​​corresponding to the feature matching point pair set. Laplacian high-pass filtering is performed on the pixel neighborhood corresponding to the feature matching point pair set in the real-time environment image. The root mean square of the pixel edge response is calculated. The mapping depth span range rate and the root mean square of the pixel edge response are input into the adaptive neural fuzzy inference system. The relocation benchmark adoption coefficient is output and then compared with the adoption threshold.

[0013] Furthermore, in step S3, when the relocation reference adoption coefficient reaches the adoption threshold, the prior pose node bound to the target keyframe is retrieved, and the 3D world coordinates of the feature matching point pair set and the pixel positions in the real-time environment image are used to construct a 3D to 2D perspective multi-point localization solution relationship. The perspective n-point localization solution process is used to obtain the coarse global initial pose, and the obtained rotation matrix is ​​subjected to singular value decomposition orthogonalization processing. The coarse global initial pose serves as the source of the absolute reference initial value in step S4.

[0014] Further, step S4 includes extracting the current real-time point cloud data with the same timestamp as the real-time environment image, converting the rough global initial pose into an absolute reference initial value under the current lidar coordinate reference, constructing a three-dimensional spatial truncated sphere bounding box around the absolute reference initial value, and then extracting the local prior point set falling within the three-dimensional spatial truncated sphere bounding box from the prior point cloud data associated with the hybrid feature map to generate a prior point cloud pixel set.

[0015] Furthermore, in step S4, a current point cloud patch is constructed from the current real-time point cloud data, and a local nearest neighbor matching relationship is established between the current point cloud patch and the prior point cloud pixel set. Then, a composite residual is constructed based on the normal distance residual and the normal direction residual. The residual objective function is formed by all composite residuals. The pose is iteratively updated using the Gauss-Newton method until the equivalent pose update length reaches the lower limit of the convergence tolerance. Finally, the accurate six-degree-of-freedom relocalization pose is output.

[0016] The technical effects and advantages of the fast initialization method for cross-modal matching constraints of point cloud images according to the present invention are as follows: This invention constructs a cross-modal training set using prior image sequences and prior point cloud data, and trains a cross-modal semantic mapping model. Then, it organizes geometrically perceptual visual descriptors, visual keyframes, and prior pose nodes into a hybrid feature map. This allows real-time environmental images to generate current geometrically perceptual visual descriptors with prior 3D structural attributes even without prior global point cloud registration, thereby avoiding simple texture confusion caused by repeated shelves, white wall edges, and corridor outlines.

[0017] After the target keyframe is locked, this invention introduces a continuous discrimination process of mapping depth span range rate, root mean square of pixel edge response and relocation reference adoption coefficient, so that the generation of coarse global initial pose is based on the simultaneous existence of matching relationship and spatial distribution. The processing chain is consistent from prior construction and online retrieval to local rigid registration of point cloud, and finally outputs accurate six-degree-of-freedom relocation pose, which is suitable for the localization initialization of mobile robots under arbitrary start conditions. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a fast initialization method for cross-modal matching constraints of point cloud images according to the present invention. Figure 2 This is a schematic diagram illustrating the projection alignment of the prior image and the prior point cloud to construct cross-modal corresponding units in this invention. Figure 3 This is a block diagram of the cross-modal semantic mapping model training and hybrid feature map construction in this invention; Figure 4 This is a schematic diagram of the process of generating the current geometric perception visual descriptor based on real-time environmental images in this invention; Figure 5 This is a flowchart illustrating the target keyframe locking, feature matching and discrimination, and coarse global initial pose calculation in this invention. Figure 6 This is a schematic diagram of local point cloud fine registration based on absolute reference initial value in the present invention. Detailed Implementation

[0019] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings and specific embodiments. The following embodiments are used to illustrate the technical principles, processing flow and implementation structure of the present invention, and are not intended to limit the scope of protection of the present invention.

[0020] Please see Figures 1-6 This invention provides a fast initialization method for cross-modal matching constraints of point cloud images, comprising: S1: Obtain prior image sequences and prior point cloud data, perform inverse ray tracing and projection alignment, construct a cross-modal training set, train a cross-modal semantic mapping model, and construct a hybrid feature map; S2: Acquire real-time environment images, input the real-time environment images into the image encoder branch of the cross-modal semantic mapping model, and generate the current geometric perception visual descriptor by combining the attention weight matrix; S3: Map the current geometric perception visual descriptor to the hybrid feature map, lock the target keyframe, establish feature matching point pairs, and obtain the rough global initial pose after discrimination by the adaptive neural fuzzy inference system; S4: Obtain the current real-time point cloud data, use the coarse global initial pose as the input of the absolute reference initial value to iterate the nearest point joint optimization framework, complete the local rigid registration within the search and matching space limited by the hybrid feature map, and output the accurate six-degree-of-freedom relocalization pose.

[0021] This invention specifically relates to the field of mobile robot localization and initialization, addressing the problems of ambiguous image features, large point cloud search range, inaccurate coarse localization, and difficulty in balancing speed and reliability during initialization in repetitive texture scenarios when starting at any location. It constructs a cross-modal training set and trains a cross-modal semantic mapping model by performing projection alignment on prior image sequences and prior point cloud data. This further generates a hybrid feature map bound to visual keyframes and prior pose nodes. Then, based on real-time environmental images, it extracts the current geometric perception visual descriptor, completes target keyframe retrieval, matching and discrimination, and coarse global initial pose calculation. Finally, it performs local rigid registration under constrained search using current real-time point cloud data, outputting accurate six-DOF relocalization pose and localization results, forming a complete initialization chain.

[0022] In industrial warehouses, within long aisles, repetitive shelving areas, and the intersection of corridors, after a mobile robot completes a global mapping, if only discrete point clouds or individual images are saved, the shelf uprights, load-bearing columns, white wall edges, and beam outlines in the scene will still exhibit highly similar appearances at different locations. This results in a lack of stable references that can be directly accessed when starting at any subsequent location. Therefore, step S1 is not simply a matter of stacking prior image sequences and prior point cloud data. Instead, it first unifies 2D pixel observations, 3D spatial coordinates, and prior pose nodes under the same world coordinate reference, and then solidifies the cross-modal correspondences that can simultaneously reflect texture and geometric relationships. This allows the prior scene to move beyond being stored in a scattered data storage state and enter a stage of organization that is trainable, searchable, and localizable.

[0023] Step S1 specifically includes the following: S101: Prior acquisition scenario establishment and cross-modal corresponding unit definition.

[0024] During the mapping process in long aisles, shelf corners, and intersections of corridors in industrial warehouses, the mobile robot continuously moves along the work path. The synchronous localization and mapping module outputs prior image sequences, prior point cloud data, and prior pose nodes corresponding to each sampling time in chronological order. The prior pose nodes are specifically used to characterize the spatial position and orientation of the LiDAR coordinate reference relative to the world coordinate reference. The camera intrinsic parameter matrix and the LiDAR to camera extrinsic parameter calibration matrix are used to merge the image observation relationship and the point cloud observation relationship under the same world coordinate reference. After the coordinate reference is unified, to avoid loose definitions of subsequent training samples, cross-modal correspondence units are introduced as the basic processing object of step S1. A cross-modal correspondence unit (CTU) is defined as a combination of a two-dimensional pixel feature, a three-dimensional spatial point aligned with that two-dimensional pixel feature, and local observations captured around that three-dimensional spatial point and the two-dimensional pixel feature at the same sampling time. The introduction of CTUs aims to fix two-dimensional texture information and three-dimensional geometric information into the smallest trainable, searchable, and traceable correspondence particles. For example, when a mobile robot passes through a double-row shelf aisle, the turning edge of the pillar in the image and the corresponding edge spatial point in the point cloud can jointly form a CTU, thereby ensuring that subsequent training no longer stays on abstract image similarity relationships, but directly establishes correspondences with real spatial structures.

[0025] S102: Two-dimensional pixel feature inverse ray tracing and projection alignment processing.

[0026] After the prior image sequence, prior point cloud data, and prior pose nodes have been merged into a unified world coordinate reference, two-dimensional pixel features are extracted for each frame of the prior image. In one embodiment, the two-dimensional pixel features are extracted using a scale-invariant feature transform algorithm. The processing order is as follows: first, a multi-scale Gaussian blur pyramid is constructed on the prior image; then, local extrema in adjacent scale difference images are filtered to remove low-contrast points and unstable edge response points, and the remaining key points are retained as two-dimensional pixel features; each two-dimensional pixel feature records at least the pixel coordinates, scale, and principal direction. Subsequently, the camera intrinsic parameter matrix is ​​used to restore each two-dimensional pixel feature to an observation line direction under the camera coordinate reference. The observation line direction is then transformed to the world coordinate reference by combining the prior pose node at the corresponding moment, thus obtaining the reverse ray starting from the camera optical center and pointing to the scene space. At the same time, each point in the prior point cloud data is also transformed to the world coordinate reference according to the prior pose node at the same moment. Then, the orthogonal distance from the spatial point to the reverse ray is used as the projection alignment criterion. The spatial point with the smallest geometric distance to the current two-dimensional pixel feature in the same frame of prior point cloud data is selected as the candidate alignment point. The axial depth of the candidate alignment point on the reverse ray is further checked to see if it falls within the reliable observation range.

[0027] To prevent false alignment caused by calibration errors and discrete sampling of point clouds, a projection alignment tolerance threshold and an axial depth reliable observation interval are set. The projection alignment tolerance threshold is defined as the maximum allowable geometric deviation between the inverse ray corresponding to the two-dimensional pixel feature and the candidate alignment point. It is obtained by: firstly, selecting a repeatable observable area on a continuous verification section with minimal occlusion at the edge of the shelf, and statistically analyzing the envelope of the residual of the manually verified point ray; then superimposing the half-diagonal discrete error introduced by the side length of the point cloud voxels; and finally adding the two to obtain the projection alignment tolerance threshold. The axial depth reliable observation interval is obtained by: firstly, collecting multiple sets of images and corresponding point cloud samples at a known distance calibration board, and statistically analyzing the depth error distribution along the direction of the observation ray; then taking the depth interval covering a predetermined proportion of manually verified samples as the base interval; and finally expanding the safety margin by the upper bound of the laser ranging error to obtain the axial depth reliable observation interval. Only candidate alignment points that simultaneously satisfy the projection alignment tolerance threshold constraint and the axial depth reliable observation interval constraint are considered valid alignment points.

[0028] When the candidate alignment point meets the projection alignment tolerance threshold constraint and the axial depth is reliable, grayscale texture fragments are extracted centered on the two-dimensional pixel features, and local point set fragments are aggregated centered on the candidate alignment point. The grayscale texture fragment is defined as a local image region that retains the central grayscale distribution and the continuous relationship of the neighborhood grayscale. The local point set fragment is defined as a local three-dimensional geometric neighborhood collected around the candidate alignment point. The two, together with the corresponding spatial points, form a cross-modal correspondence unit. All cross-modal correspondence units are aggregated to form a cross-modal training set. For example, in a passageway where white walls and shelves are parallel, although the texture of the white wall area is sparse, the corners of the walls and the boundaries of the door frames can still find stable spatial points through reverse rays. After the projection alignment tolerance threshold is filtered, only areas with clear geometric relationships can enter the cross-modal training set, reducing the probability of mismatched training from the source.

[0029] S103: Cross-modal semantic mapping model training and geometrically perceptual visual descriptor generation.

[0030] After the cross-modal training set is constructed, grayscale texture fragments are input into the image encoding branch, and local point set fragments are input into the point cloud encoding branch. Together, they constitute a cross-modal semantic mapping model. The image encoding branch outputs a common geometric vector and an image residual vector, while the point cloud encoding branch outputs a common geometric vector and a point cloud residual vector. The common geometric vector is used to carry the common information that allows mutual mapping between two-dimensional textures and three-dimensional structures. The image residual vector is used to retain image modality-specific information, and the point cloud residual vector is used to retain point cloud modality-specific information. As a construction example, grayscale texture fragments can be cropped into single-channel image blocks of 32 by 32 pixels, local point set fragments can be uniformly sampled into 128 three-dimensional points, the image encoding branch can adopt a four-level convolutional backbone network with the number of channels set to 32, 64, 128, and 256 respectively, and the point cloud encoding branch can adopt the PointNet++ hierarchical feature extraction structure with the hierarchical output dimensions set to 64, 128, 256, and 256 respectively. Then, two sets of fully connected mapping heads are connected respectively, with the common geometric vector dimension set to 128, the image residual vector dimension set to 64, and the point cloud residual vector dimension set to 64.

[0031] During the training phase, a joint constraint approach is employed to achieve feature decoupling and joint measurement. Firstly, orthogonal constraints are applied to common geometric vectors and their corresponding residual vectors to reduce the coupling between common information and modality-specific information. Secondly, contrastive loss constraints are applied to the common geometric vectors of the image and the common geometric vectors of the point cloud within the same cross-modal correspondence unit (CLU), ensuring they are close in the feature space and separated from other CLUs. Positive samples in the contrastive loss are taken from the same CLU, while negative samples are taken from other CLUs within the same batch, from different sampling times or spatial locations. As an example of training optimization, the cross-modal training set can be divided into training, validation, and test subsets in an 8:1:1 ratio. The AdamW optimizer is used, with a batch size of 64, an initial learning rate of 0.001, a weight decay coefficient of 0.0001, and a total of 120 training rounds. In rounds 40 and 80, the learning rate is decayed to 0.1 of its original value. The contrast loss temperature coefficient is set to 0.07, the orthogonality constraint loss coefficient is set to 0.2, and the gradient clipping threshold is set to 5. Training is stopped when the validation loss does not decrease for 12 consecutive rounds, and the model parameters with the minimum loss in the validation subset are retained.

[0032] After training convergence, the image encoding branch is retained as the forward inference channel directly called in step S2. Then, all valid common geometric vectors in the same frame of prior image are integrated into a frame-level representation according to the generalized mean convergence rule, resulting in a geometrically perceptual visual descriptor. As an example of parameter settings, the generalized mean convergence order can be set to three, performing channel convergence only on valid common geometric vectors that have passed projection alignment verification, and adding a length normalization process at the output to ensure consistent description scales between different frames. The geometrically perceptual visual descriptor is used to retain clues of continuous changes in scene depth and spatial topological differences without directly inputting real-time point cloud data. For example, when two shelf aisles both present repeated columns and beams, the aisle near the corner retains polyline aggregation and depth transition information in the common geometric vectors, while the aisle near the straight end retains depth truncation and end boundary information. The geometrically perceptual visual descriptor obtained after convergence has stable differences and can be directly used for subsequent retrieval.

[0033] S104: Visual keyframe binding and hybrid feature map construction.

[0034] After obtaining the geometric perception visual descriptors corresponding to each frame of prior image, the system continues to select frames from the prior image sequence that represent the local spatial observation state, have complete viewpoint coverage, and minimal occlusion as visual keyframes. Each visual keyframe is then bound one-to-one with the geometric perception visual descriptor, prior pose node, and adjacency relationship in the pose graph at the same time. The selection of visual keyframes is performed sequentially according to the translational changes, pose changes, and effective structural pixel coverage changes between adjacent retained frames: when the translational increment relative to the previous retained keyframe reaches a first displacement threshold, or the heading change reaches a first angle threshold, or the structural pixel coverage change reaches a first coverage threshold, the current frame is written into the visual keyframe set. In one embodiment, the first displacement threshold can be 0.5 meters to 1 meter, the first angle threshold can be 8 degrees to 15 degrees, and the first coverage threshold can be 0.15 to 0.25. To improve subsequent retrieval speed, locality-sensitive hashing is used to perform binary indexing on geometrically perceived visual descriptors. Specifically, multiple sets of hash hyperplanes are used to sequentially determine whether the geometrically perceived visual descriptor is located on the positive or negative side of the hyperplane. All determination results are then concatenated into a binary index key. Subsequently, the binary index key is used as the retrieval entry point, and geometrically perceived visual descriptors, visual keyframes, prior pose nodes, and adjacency relationships are written into a unified index structure to form a hybrid feature map. The hybrid feature map is defined as an information collection that simultaneously contains high-dimensional semantic retrieval information, keyframe image information, pose prior information, and graph structure topological relationships.

[0035] For example, when a mobile robot enters a crossroads from a corridor, although the visual keyframes of the crossroads and the visual keyframes of the straight sections before and after have some local texture overlap, they will fall into different index positions after being written into the hybrid feature map because the geometric perception visual descriptors are different and the adjacency relationships are different. In the subsequent step S2, only the real-time environment image needs to be input and the image encoding branch needs to be called to directly generate the current geometric perception visual descriptor of the same dimension as the hybrid feature map without reorganizing the prior point cloud data. This provides a unique entry point for the target keyframe locking and coarse global initial pose solution in step S3.

[0036] After step S1, a stable correspondence has been established between the two-dimensional pixel features in the prior image sequence and the three-dimensional spatial points in the prior point cloud data. All valid correspondences are organized into a cross-modal training set. The cross-modal training set further drives the cross-modal semantic mapping model to complete training. The prior image sequence is then used by the cross-modal semantic mapping model to generate a geometrically perceptual visual descriptor, which, together with visual keyframes, prior pose nodes, and graph structure topology, is solidified into a hybrid feature map. At this point, the target scene has been transformed from raw multi-source collected data into a priori representation with unified semantic expression and unified spatial indexing relationships.

[0037] When a mobile robot leaves the mapping starting point and is powered on directly at an unknown location, the immediate available information is a real-time environmental image. However, the current real-time point cloud data has not yet been spatially aligned with the hybrid feature map, so local registration of the point cloud cannot be directly relied upon for relocalization. If the pure visual texture comparison path is still used at this point, repeating shelves, white wall boundaries, and corridor lines will push multiple widely separated locations into the candidate range simultaneously. In this scenario, step S2 undertakes the single-frame entry conversion task, that is, without first calling the current real-time point cloud data for sampling, directly recovering the structural semantic expression consistent with step S1 from the real-time environmental image, so that the current observation has the same descriptive basis for entering the hybrid feature map retrieval process.

[0038] Step S2 specifically includes the following: S201: Real-time environmental image access and normalization matrix construction.

[0039] After receiving the global localization initialization command at any unknown location on the map, the mobile robot uses its onboard camera to capture real-time environmental images. The real-time environmental images maintain the same optical axis direction definition and timestamp recording method as the mapping stage in step S1, to ensure that the subsequent output objects can establish a unified semantic mapping with the hybrid feature map. Before the images enter the cross-modal semantic mapping model, size reshaping, channel sorting, and numerical normalization are performed. Size reshaping is used to adjust the real-time environmental images to the fixed input size used in the training stage of step S1. Channel sorting is used to maintain the consistency of the color channel order. Numerical normalization is used to eliminate the grayscale amplitude shift caused by lighting switching, exposure drift, and shadow occlusion at different startup times, and finally obtains the normalized matrix.

[0040] The normalization matrix is ​​defined as a tensor representation that is completely consistent with the input size, input channel order, and input value range of the image encoder branches. It is constructed as follows: for each color channel, the maximum and minimum gray values ​​in the real-time environmental image are read. Then, the minimum gray value of the current color channel is subtracted from the current pixel's gray value, and then divided by the difference between the maximum and minimum gray values ​​of the current color channel. A zero-value protection factor is added to the denominator of the division to obtain the normalized result of the current color channel. The zero-value protection factor is a fixed positive number, rounded to 10^-6 under single-precision floating-point processing conditions and to 10^-4 under half-precision floating-point processing conditions. This ensures that the normalization matrix can be stably generated even when the overall image brightness is nearly constant.

[0041] For example, when a mobile robot stops in the middle of a double-row shelf, the top lighting is fully turned on during the first startup, and only the end lighting of the aisle is turned on during the second startup. At this time, although the grayscale distribution of the column boundaries, shelf beams and ground seams in the two real-time environmental images is different, the texture transition relationship and edge transition relationship are still retained within a uniform amplitude range after the normalized matrix is ​​compressed channel by channel. Therefore, the image encoder branch can use the feature extraction habits formed during the training in step S1, without causing deep response misalignment due to the overall increase or compression of brightness. After the normalized matrix is ​​constructed, step S2 enters the forward inference stage of the image encoder branch.

[0042] S202: Image encoder branch forward inference and attention weight matrix construction.

[0043] After the normalized matrix is ​​constructed, it is input separately into the image encoder branch of the cross-modal semantic mapping model. The image encoder branch performs one forward inference along the fixed parameters trained in step S1, outputting a deep convolutional feature tensor and an attention weight matrix. The attention weight matrix is ​​directly output by the multi-head attention layer in the image encoder branch during the current forward inference process, and after normalization, it represents the response distribution relationship between different receptive field units. The deep convolutional feature tensor is defined as the set of spatial semantic responses formed by the convolutional layer group at the end of the image encoder branch to the entire real-time environment image. It simultaneously preserves the edge direction, corner combination, line-plane intersection, and local texture arrangement relationship, and is subsequently used to generate the current geometric perception visual descriptor. The attention weight matrix is ​​used to characterize which positions will point to each other, complement each other, and jointly form a stable spatial structure at the deep semantic level.

[0044] The attention weight matrix is ​​not constructed from the retraining in step S2, but directly from the attention mapping results that have been solidified during the training of the cross-modal semantic mapping model in step S1. It is then combined with the forward response of the current normalized matrix to form the attention weight matrix at the current moment. In order to transform the matrix-level relationship into structural cues that can be called on the image plane, step S2 further constructs a focus consistency map. The focus consistency map is defined as the intensity distribution map formed after projecting the bidirectional focus consistency relationship between each receptive field unit and the other receptive field units back onto the image plane. The construction method is as follows: for each receptive field unit, its outward focus distribution and the outward return focus distribution are read respectively. First, the sum of the products of the two types of distributions is calculated to represent the degree of bidirectional overlap. Then, the sum of the differences between the two types of distributions is calculated to represent the degree of bidirectional deviation. Finally, the degree of bidirectional overlap is divided by the degree of bidirectional deviation and a zero-value protection factor is added to obtain the focus consistency of the current receptive field unit. Then, the focus consistency of all receptive field units is backfilled onto the feature map plane according to the spatial position to form the focus consistency map.

[0045] The purpose of the focus consistency map is to elevate the question of whether a single edge exists to whether multiple receptive field units continuously focus around the same structure. Therefore, it can distinguish between column bends, door frame boundaries, and beam connection sections that truly have spatial organizational significance, and isolated bright spots caused simply by stains, reflections, or local lighting. For example, when a mobile robot starts facing the end of a shelf, the connection between the end frame edge and the column will form a back-and-forth pointing relationship among multiple receptive field units. The focus consistency map shows continuous enhancement in this area. Although the local reflection on the white wall surface may show abrupt changes in brightness, it will not form a stable bidirectional focus consistency relationship, and thus will not leave traces of the same structure in the focus consistency map. After the focus consistency map is constructed, step S2 enters the gradient abrupt change region screening and three-dimensional structure prior probability generation stage.

[0046] S203: Gradient mutation region screening and 3D structure prior probability generation.

[0047] Focus consistency maps alone are insufficient to guarantee complete structural location extraction, as shelf edges, corners, and floor seams in warehouse scenarios exhibit significant gray-level transitions. Therefore, step S2 further performs Schhaler operator convolution on the gray-level channels of the normalized matrix to obtain the gradient magnitude matrix. The Schhaler operator uses a 3x3 discrete convolution kernel to calculate the horizontal and vertical gradients, respectively. The horizontal kernel enhances the vertical boundary response, and the vertical kernel enhances the horizontal boundary response. Compared to ordinary first-order difference operators, the Schhaler operator is more stable to changes in rotation direction, making it suitable for extracting shelf edges, corners, and floor seams. The gradient magnitude matrix is ​​defined as a measure of the strength of local gray-level changes at each pixel position in the entire real-time environmental image. It is constructed by convolving the gray-level channels with horizontal Schhal convolution kernels and vertical Schhal convolution kernels respectively to obtain horizontal gradient components and vertical gradient components. Then, the horizontal gradient components and vertical gradient components are summed by squares and the square root is taken to obtain the gradient magnitude at each pixel position, thereby distinguishing the flat areas of the wall, the vertical areas of the shelf, and the boundary transition areas.

[0048] After the focus consistency map and gradient magnitude matrix are obtained, step S2 constructs a structural prior potential map. The structural prior potential map is defined as a geometric saliency distribution map formed by mapping deep semantic focus relationships and local gray-level transition relationships onto the same image plane. The construction method is as follows: first, the focus consistency map and gradient magnitude matrix are multiplied point by point at the corresponding pixel positions to obtain the joint structural response. Then, the median position value of all joint structural responses in the entire image is counted as the reference scale. Finally, the joint structural response at the current pixel position is divided by the reference scale and a zero-value protection factor is added to obtain the structural prior potential value at the current pixel position.

[0049] After generating the structural prior potential map, a three-dimensional structural prior probability map is constructed. This map is defined as the probability distribution of each pixel location belonging to a region capable of supporting spatial structural cues. The construction method involves inputting the structural prior potential map into a monotonically increasing exponential saturation mapping, making positions with larger structural prior potential values ​​closer to one, and positions with structural prior potential values ​​close to zero closer to zero, thereby compressing the saliency of unbounded structures to between zero and one. To clarify which positions can participate in the generation of the current geometric perception visual descriptor, step S2 sets a structural activation threshold. This threshold is defined as the number of pixels allowed to participate in the generation of the three-dimensional structural prior probability map. The minimum probability limit for elements to enter the structural pixel region is determined by the following method: Select several verified road sections that have been manually verified from the mapping stage in step S1, project the column edges, wall corner lines, shelf front edges, and door frame boundaries in the prior point cloud data back to the corresponding images to form a set of structural sample pixels. Then, form a set of non-structural sample pixels from the flat wall area, the reflective area of ​​the ground, the stain area, and the non-boundary repeating texture area. Calculate the probability distribution of the two types of samples in the three-dimensional structural prior probability map, and take the probability value corresponding to the intersection of the two distributions as the structural activation threshold, thereby constructing the structural pixel region.

[0050] For example, when a mobile robot starts facing the white wall of a corridor, the shadow band in the middle of the white wall may retain a certain gradient magnitude. However, due to insufficient focus consistency, the joint structural response cannot pass the structural activation threshold. Meanwhile, the corner line, the intersection of the top and the ground, and the boundary of the door frame have both a continuous focus relationship and a grayscale transition relationship. After entering the structural pixel region, they will be preferentially retained in the subsequent descriptor convergence process. After completing the construction of the three-dimensional structural prior probability map and the structural pixel region, step S2 enters the current geometric perception visual descriptor generation stage.

[0051] S204: Structural modulation feature generation and current geometric perception visual descriptor output.

[0052] Once the structural pixel region is clearly defined, step S2 does not call the current real-time point cloud data. Instead, it directly applies the 3D structural prior probability map to the deep convolutional feature tensor to generate a structural modulation feature tensor. The structural modulation feature tensor is defined as a set of deep visual features that have been embedded with 3D structural prior attributes. It is constructed by multiplying the probability value of the corresponding position in the 3D structural prior probability map by the total channel response of the deep convolutional feature tensor at that position at each spatial location. The closer the probability value is to one, the closer the current position is to a position that can carry spatial structural cues, and the more completely the corresponding channel response is preserved. The closer the probability value is to zero, the more the current position lacks stable structural meaning, and the more fully the corresponding channel response is suppressed. In this way, the original deep response that only depends on the 2D texture is rewritten into a structural modulation response that takes into account spatial structural cues.

[0053] After the structural modulation feature tensor is generated, step S2 follows the generalized mean convergence rule determined in step S1, performing convergence only within the structural pixel region channel by channel to obtain the current geometric perception visual descriptor. The current geometric perception visual descriptor is defined as a frame-level structural semantic vector that maintains the same dimension, channel order, and numerical organization method as the geometric perception visual descriptors of each frame in the hybrid feature map. The construction method is as follows: for each valid position in the structural pixel region, read the absolute value of the response of the structural modulation feature tensor in the corresponding channel, perform channel-by-channel convergence according to the generalized mean convergence order fixed in step S1, and then concatenate all channel results in a predetermined order to obtain the current geometric perception visual descriptor. Since the geometric perception visual descriptors used to generate the hybrid feature map in step S1 use the same image encoder branch, the same convergence rule, and the same channel organization method, the current geometric perception visual descriptor can be directly mapped to the high-dimensional semantic space of the hybrid feature map after generation, without the need for dimensional calibration or semantic alignment in step S3.

[0054] For example, when a mobile robot starts up and stops in the middle of two rows of shelves with the same height and similar appearance, ordinary visual descriptions often only repeatedly record the surface textures of the columns and beams, easily mistaking multiple channels for the same location. However, the structural modulation feature tensor prioritizes saving the deep responses that are closer to the intersection of real-world polylines, end connection segments, and ground seams. As a result, the current geometric perception visual descriptor is closer to the visual keyframes corresponding to the real location. Thus, step S2 completes the full-link transformation from the real-time environment image to the current geometric perception visual descriptor. Step S3 can directly read the current geometric perception visual descriptor and perform target keyframe locking, feature matching point pair filtering, relocation benchmark adoption coefficient discrimination, and coarse global initial pose solving in the hybrid feature map.

[0055] After step S2, the real-time environment image has been processed into a normalized matrix. Then, the deep convolutional feature tensor and attention weight matrix are extracted by the image encoder branch of the cross-modal semantic mapping model. Subsequently, by combining the focus consistency relationship, gradient change relationship, and 3D structure prior probability allocation process, structure modulation and channel convergence are completed, ultimately forming the current geometrically perceptual visual descriptor. At this point, the current observation is no longer a simple two-dimensional image input, but has been processed into a frame-level structural semantic object consistent with the geometrically perceptual visual descriptor in the hybrid feature map.

[0056] After the geometrically perceptual visual descriptor has been formed, the relocalization task shifts from the image reasoning stage to the semantic retrieval and spatial discrimination stage. The problem at this point is no longer how to extract structural information from real-time environmental images, but how to locate the visual keyframes corresponding to the true location in the hybrid feature map and eliminate spatially invalid pseudo-matches from multiple appearance-appearance similarity candidates. Especially in scenarios where warehouse shelves are arranged in rows and corridor boundaries extend repeatedly, the nearest neighbor results of a single descriptor may still contain misleading terms. Therefore, step S3 must link target keyframe locking, local matching establishment, geometric distribution verification, and pose calculation and discrimination into a single processing chain.

[0057] S301: Current geometric perception visual descriptor retrieval and target keyframe locking.

[0058] After the mobile robot completes step S2 at the unknown location, the current geometric perception visual descriptor has the same channel organization method and semantic scale as the hybrid feature map. Step S3 first uses the hash partitioning rule used when constructing the hybrid feature map in step S1 to map the current geometric perception visual descriptor into the current index key. The current index key is defined as the retrieval entry point of the current geometric perception visual descriptor in the hybrid feature map. The construction method is to read all the hash hyperplanes that have been fixed in step S1 in sequence, compare the projection result of the current geometric perception visual descriptor on each hash hyperplane with the hyperplane boundary direction, write a fixed binary value when it is on one side, write another fixed binary value when it is on the other side, and finally concatenate all the binary values ​​in sequence to form the current index key.

[0059] After obtaining the current index key, visual keyframes that are completely identical to the current index key are first read from the hybrid feature map to form the first candidate set. When the first candidate set is empty or the number of keyframes in the first candidate set is less than a preset lower limit, visual keyframes in the index buckets with a Hamming distance of one from the current index key are then read to form the second candidate set. Subsequently, visual keyframes that have a one-hop pose graph adjacency relationship with each visual keyframe in the first and second candidate sets are incorporated to form the candidate keyframe set. Each frame in the candidate keyframe set records the frame-level descriptor distance, index bucket distance, and graph topological adjacency relationship. After the candidate keyframe set is established, the Euclidean distance between the current geometrically perceived visual descriptor and each geometrically perceived visual descriptor in the candidate keyframe set is calculated one by one, and the first M keyframes are obtained in ascending order of Euclidean distance. Then, structural anchor point matching and reprojection interior point statistics are performed on each initial keyframe. The keyframe with the most interior points is selected as the target keyframe. When the number of interior points is the same, the keyframe with the smallest average reprojection error is selected. If the average reprojection error is still the same, the keyframe with the smallest Euclidean distance is selected. The target keyframe is defined as a unique keyframe determined after descriptor distance screening and local geometric consistency verification, and the subsequent structural anchor point set is extracted from this keyframe.

[0060] For example, when a mobile robot starts moving between two identical aisle shelves, multiple visual keyframes from the hybrid feature map often simultaneously enter the candidate keyframe set. Directly searching the entire map is not only time-consuming but also prone to including distantly similar regions in the comparison. By first narrowing the search entry point using the current index key, then supplementing with spatially continuous positions using graph topological adjacency relationships, and finally using Euclidean distance for a precise filtering, the target keyframe truly corresponding to the current location can be identified. Thus, step S3 has obtained the unique source of visual keyframes for subsequent matching.

[0061] S302: Construction of structural anchor point set and formation of feature matching point pair set.

[0062] After the target keyframe is locked, step S3 continues to expand the geometric carrying information within the target keyframe to construct a set of structural anchor points. The set of structural anchor points is defined as the collection of structural anchor points extracted from the cross-modal correspondence units corresponding to the target keyframe. Each structural anchor point simultaneously contains the pixel position in the target keyframe, the local description information extracted around that pixel position, and the corresponding 3D world coordinate point. The construction of the structural anchor point set involves traversing all cross-modal correspondence units bound and saved in the target keyframe, reading the pixel position and 3D world coordinate point in each cross-modal correspondence unit whose projection alignment relationship has been confirmed, and then, using that pixel position as the center, calling the scale-invariant feature transform algorithm on the target keyframe image to extract local description information. This reorganizes the 2D local observations and 3D world coordinate points in the target keyframe into matchable structural anchor points.

[0063] Synchronized with the structural anchor point set, step S3 extracts the current local feature set on the real-time environment image using the same scale-invariant feature transformation algorithm. The current local feature set is defined as the set of all pixel positions in the real-time environment image that can participate in local matching and their local descriptive information. Subsequently, step S3 performs bidirectional nearest neighbor matching on the structural anchor point set and the current local feature set. Specifically, for each current local feature, the structural anchor point with the shortest descriptive distance is first found in the structural anchor point set, and then for each structural anchor point, the current local feature with the shortest descriptive distance is found in the current local feature set. Only the sets of correspondences that are mutually pointed back by the bidirectional search results are retained to form an initial correspondence set. The initial correspondence set is defined as the set of candidate correspondences established only through the similarity of local descriptive information and that have not yet undergone spatial geometric verification.

[0064] After the initial correspondence set is formed, step S3 uses a three-point perspective localization algorithm and a random sampling consensus algorithm to perform geometric consistency elimination. The three-point perspective localization algorithm is used to generate candidate poses from a small number of 3D to 2D correspondences, and the random sampling consensus algorithm is used to eliminate outlier matches in multiple rounds of sampling. In each round of sampling, a set of minimum support correspondences is extracted from the initial correspondence set to generate a set of candidate poses. Then, all candidate correspondences are projected back to the real-time environmental image one by one, and the pixel deviation between the projected position and the current local feature pixel position is calculated. Step S3 introduces a reprojection residual threshold, which is defined as the maximum pixel deviation allowed when a single candidate correspondence is accepted. It is obtained by reading the reprojection error envelope of the calibration samples that have been manually verified during the camera calibration stage, and then collecting repeated image samples at known positions in the storage channel. The pixel drift boundary under the true matching conditions is statistically analyzed, and the upper limit of the common coverage of the two types of boundaries is taken as the reprojection residual threshold. Only correspondences with pixel deviations not exceeding the reprojection residual threshold are retained, ultimately forming a set of feature matching point pairs. The set of feature matching point pairs is defined as a set of three-dimensional to two-dimensional correspondences that simultaneously satisfy local description consistency and spatial reprojection consistency, and is also the only input set for all subsequent discriminant quantities and pose calculations in step S3.

[0065] For example, when a mobile robot stops at the entrance of an adjacent shelf aisle, the surface of the shelf column often shows multiple sets of similar descriptions in the initial correspondence set. After repeated verification by the random sampling consensus algorithm, only the correspondences that satisfy the real spatial arrangement relationship can enter the feature matching point pair set. False correspondences caused by repeated textures are cleared in the reprojection verification stage.

[0066] S303: Calculation of the root mean square of the mapping depth span range rate and pixel edge response.

[0067] After the feature matching point pair set is formed, step S3 does not directly perform a coarse global initial pose calculation. Instead, it first determines whether this batch of matches has reliable spatial unfolding capability and local texture carrying capability. To this end, step S3 first calculates the mapping depth span range rate. The mapping depth span range rate is defined as the ratio between the depth unfolding degree of all world coordinate 3D points in the feature matching point pair set in the imaging direction of the target keyframe and the average depth. Specifically, the calculation method is as follows: first, each world coordinate 3D point in the feature matching point pair set is transformed to the camera coordinate reference corresponding to the target keyframe, and the depth coordinates in the transformation result are read. Then, the maximum depth coordinate and the minimum depth coordinate are found among all depth coordinates, and the difference between the two is taken as the absolute depth span. Then, the absolute depth span is divided by the arithmetic mean of all depth coordinates, and a fixed stable quantity is added to the denominator to obtain the mapping depth span range rate. The fixed stable quantity is introduced here to prevent unstable fluctuations in the denominator when the number of feature matching point pairs is small or the depth distribution is concentrated. The significance of the mapping depth span range rate lies in determining whether this batch of matches covers sufficient depth variations to support perspective positioning. If all matches are concentrated in the same plane or the same narrow depth band, even if the appearance is consistent, it cannot provide sufficient spatial constraints for subsequent pose calculation.

[0068] Subsequently, step S3 calculates the root mean square (RMS) of the pixel edge response. The RMS of the pixel edge response is defined as a comprehensive measure of the high-frequency edge energy carried by the set of feature matching points within all local neighborhoods in the real-time environmental image. Specifically, a five-by-five pixel neighborhood is extracted centered on each current local feature pixel location. A Laplacian high-pass filter is applied to each five-by-five neighborhood to obtain the corresponding two-dimensional high-frequency intensity matrix. The Laplacian high-pass filter uses a 3x3 discrete convolution kernel with a positive center and negative peripheries to enhance the second-order changes in local grayscale, highlighting grayscale abrupt changes near edges and corners. The two-dimensional high-frequency intensity matrix is ​​defined as the set of grayscale abrupt change in intensity for each pixel location within the five-by-five pixel neighborhood after high-pass filtering. After obtaining all two-dimensional high-frequency intensity matrices, all elements are squared, summed, and averaged sequentially. The square root of the average result is then taken to obtain the RMS of the pixel edge response. The root mean square of pixel edge response maintains the dimension of gray level. Its function is to determine whether there is a clear edge organization around the set of feature matching point pairs. If the matching only falls within flat walls, blurred shadows, or low-contrast texture areas, the root mean square of pixel edge response cannot form boundary information that supports perspective positioning.

[0069] For example, when a mobile robot starts facing the intersection of a white wall and a shelf, if the set of feature matching point pairs is only concentrated in the narrow shadow strip on the wall, the mapping depth span range will show insufficient depth expansion, and the root mean square of pixel edge response can only retain limited edge information. Even if the number of such matches meets the requirements, it is not possible to directly enter the rough global initial pose calculation.

[0070] S304: Relocation reference adoption coefficient discrimination and coarse global initial pose solution.

[0071] After obtaining both the mapping depth span range rate and the root mean square of the pixel edge response, they are input into the adaptive neural fuzzy inference system to output the relocation baseline adoption coefficient. The relocation baseline adoption coefficient is defined as the final adoption scalar given for the current set of feature matching point pairs, and its value is limited to the range between zero and one. The adaptive neural fuzzy inference system is constructed based on offline sample training. The training samples consist of real corresponding samples saved during the mapping in step S1 and manually labeled pseudo-corresponding samples. Each training sample contains the mapping depth span range rate, the root mean square of the pixel edge response, and a label indicating whether the correct pose can be stably solved.

[0072] As a construction example, the adaptive neural fuzzy inference system can adopt a five-layer structure. The input layer has two input nodes, which receive the mapping depth span range rate and the root mean square of the pixel edge response, respectively. The fuzzification layer sets three membership subsets for each input quantity, defined as low, medium, and high, and uses Gaussian membership functions to complete the membership degree mapping, thus forming nine fuzzy rules. The rule layer is used to calculate the activation value of each fuzzy rule. The normalization layer is used to normalize all activation values. The output layer is used to generate the relocation benchmark adoption coefficient based on the conclusion value of each fuzzy rule. As a parameter setting example, the initial center values ​​of the three membership functions corresponding to the mapping depth span range rate can be set to 0.2, 0.5, and 0.8, the initial center values ​​of the three membership functions corresponding to the root mean square of the pixel edge response can be set to 10, 30, and 50, the initial width parameters of the membership functions can be uniformly set to 0.1 and 8, and the initial values ​​of the rule conclusion values ​​can be uniformly set from 0.1 to 0.9.

[0073] As an example of training optimization, all samples can be divided into training, validation, and test subsets in a 7:2:1 ratio. The number of training rounds is set to 100, the initial learning rate to 0.01, and the batch size to 32. A hybrid learning approach is used to update parameters: the antecedent parameters are updated using gradient descent, and the consequent parameters are updated using least squares. Training stops when the validation error does not decrease after ten consecutive rounds, and the set of parameters with the smallest validation error is retained. After training, the membership degree of each input variable under each fuzzy rule is calculated during the current runtime. The two membership degrees within the same fuzzy rule are then multiplied to obtain the rule activation value. Finally, all rule activation values ​​are merged with their respective rule conclusion values ​​to output the relocation baseline adoption coefficient.

[0074] To ensure the relocation reference adoption coefficient has a clear engineering discrimination boundary, an adoption threshold is set. The adoption threshold is defined as the minimum adoption limit that allows the current set of feature matching point pairs to enter the coarse global initial pose calculation. As an example, 500 sets of true matching samples and 500 sets of false matching samples can be repeatedly collected at known road segments, and then fed into the adaptive neural fuzzy inference system one by one. The distribution of the relocation reference adoption coefficients of the two types of samples is statistically analyzed, and the value corresponding to the intersection of the two distributions is taken as the adoption threshold, for example, it can be set to 0.65. Currently, during operation, the set of feature matching point pairs is only confirmed as eligible for pose calculation when the relocation reference adoption coefficient reaches the adoption threshold; when the relocation reference adoption coefficient is lower than the adoption threshold, step S3 returns to step S301 to expand the candidate keyframe set and re-execute the matching filter; when the coefficient remains lower than the adoption threshold after a preset number of retries, an initialization failure flag is output, and step S4 is not performed.

[0075] Once the adoption conditions are met, the world coordinate reference relationship is read from the prior pose node bound to the target keyframe. Then, all the 3D points in the world coordinate system of the feature matching point pair set and the current local feature pixel positions in the real-time environment image are fed into the perspective n-point localization solution process to obtain a coarse global initial pose. The perspective n-point localization solution process is executed in the following order: first, candidate pose initial values ​​are generated using the minimum support correspondence; then, all interior point correspondences retained by the random sampling consensus algorithm are fed into the least squares reprojection optimization to obtain the rotation matrix and translation vector; finally, singular value decomposition orthogonalization is performed on the obtained rotation matrix to ensure that the rotation matrix satisfies the orthogonality constraint. The coarse global initial pose is defined as the first set of global pose results of the current camera relative to the world coordinate reference of the hybrid feature map, which is composed of the rotation matrix and translation vector. After the perspective n-point localization solution is completed, the coarse global initial pose is directly used as the source of the absolute reference initial value of the spatial transformation matrix in step S4. For example, when a mobile robot stops after entering an intersection from a corridor and restarts, if the target keyframe is located at the entrance of the intersection, the 3D points in the world coordinates of the feature matching point pair set are distributed in the intersection area of ​​the corner line, the door frame edge and the ground seam. After being adopted and judged, the robot enters the pose solution and can obtain a rough global initial pose consistent with the current position.

[0076] After step S3 is completed, the current geometric perception visual descriptor has completed the hybrid feature map mapping, the target keyframe has been uniquely locked, and the correspondence between the structural anchor point set and the current local feature set has been filtered into a set of feature matching point pairs through bidirectional nearest neighbor matching and random sampling consensus algorithms. The mapping depth span range rate and the root mean square of the pixel edge response have also been calculated and output as relocalization benchmark adoption coefficients through the adaptive neural fuzzy inference system. The set of feature matching point pairs that meet the adoption conditions further participates in the 3D to 2D perspective multi-point localization solution to form a coarse global initial pose, so that the current position obtains a pose result that already has global spatial constraints.

[0077] The rough global initial pose has brought the mobile robot back to the real-world neighborhood, but this result is still at the visual guidance level and has not yet precisely matched the current real-time point cloud data with the prior point cloud data in the hybrid feature map. For industrial warehousing and complex corridor scenarios, the local geometric relationships of shelf corners, wall polylines, and column boundaries still need to be further corrected through local rigid registration of the point cloud; otherwise, translational and orientation offsets may still remain during the navigation execution phase. Therefore, step S4 starts from the rough global initial pose, incorporates the current real-time point cloud data into a unified world coordinate reference, and completes the coarse-to-fine geometric fitting within the constrained search space.

[0078] S401: Coarse global initial pose transformation and construction of absolute baseline initial values.

[0079] After the mobile robot completes step S3 at any unknown location, it has obtained a rough global initial pose with the current camera as the reference object. Step S4 first transforms the rough global initial pose from the current camera coordinate reference to the current lidar coordinate reference based on the extrinsic parameter calibration matrix of the lidar to the camera that has been fixed in step S1, thus obtaining the initial orientation and initial position of the current lidar. If the current real-time point cloud data and the real-time environment image do not have the same timestamp, then the frame or a segment of cumulative scan with the smallest absolute value of the time difference is selected from the point cloud scans with a time difference not exceeding a preset synchronization window as the current real-time point cloud data; in one embodiment, the synchronization window can be between 20 milliseconds and 50 milliseconds. The current initial orientation of the lidar is defined as the rotation relationship corresponding to the coarse global initial pose under the current lidar coordinate reference. It is obtained by combining the rotation relationship in the current camera's coarse global initial pose with the rotation extrinsic parameters of the lidar to the camera in the order of coordinate transformation. The current initial position of the lidar is defined as the translation relationship corresponding to the coarse global initial pose under the current lidar coordinate reference. It is obtained by rotating the translation extrinsic parameters of the lidar to the camera to the world coordinate reference and then superimposing them on the position of the current camera's coarse global initial pose. This ensures that the output of step S3 can be seamlessly transferred to the point cloud registration process.

[0080] After obtaining the initial orientation of the current lidar, step S4 continues to generate the initial rotation quaternion of the current lidar according to the standard conversion rules from rotation matrix to rotation quaternion. The generation order is as follows: first, read the main diagonal information of the rotation matrix, then determine the acquisition channels of scalar components and vector components according to the sum of the main diagonal and the size rules of each diagonal element, and finally perform quaternion normalization processing to ensure that the rotation quaternion meets the requirements of rigid body rotation representation. The introduction of the initial rotation quaternion of the current lidar is to provide a set of numerically stable orientation expressions that will not produce Euler angle singularities for the subsequent iterative nearest point joint optimization framework.

[0081] Subsequently, the initial orientation, initial position, and initial rotation quaternion of the current LiDAR are encapsulated together as an absolute reference initial value. This absolute reference initial value is defined as the starting reference pose for all local rigid registration iterations in step S4. The constructed result includes both a spatial transformation matrix expression and a rotation quaternion expression, where the spatial transformation matrix expression is used for point cloud coordinate transformation, and the rotation quaternion expression is used for odometry state transfer. For example, if a mobile robot stops near the entrance of an intersection and then restarts, step S3 has already compressed the current position to the vicinity of the intersection. After step S4 obtains the initial orientation and initial position of the current LiDAR through extrinsic parameter transformation, the current real-time point cloud data no longer needs to be blindly searched from the entire hybrid feature map; instead, local registration can begin directly around the absolute reference initial value.

[0082] S402: Current real-time point cloud data mapping, 3D spatial truncated sphere bounding box construction and prior point cloud pixel generation.

[0083] After the initial absolute reference value is constructed, step S4 extracts the current real-time point cloud data with the same timestamp as the real-time environmental image. The current real-time point cloud data is defined as the spatial point set formed by a complete scan or a series of continuous scans accumulated by the current LiDAR at the current startup time. Each point cloud point is represented by the current LiDAR coordinate reference. Subsequently, step S4 calls the spatial transformation matrix in the initial absolute reference value to map all the current real-time point cloud data to the world coordinate reference, obtaining the initial mapped point set. The purpose of the initial mapped point set is to directly put the current point cloud observation into the unified spatial reference used by the hybrid feature map, so that subsequent prior point cloud data retrieval has the same coordinate system basis.

[0084] After the initial mapping point set is formed, step S4 constructs a 3D truncated spherical bounding box around the current initial position of the LiDAR. The 3D truncated spherical bounding box is defined as a local 3D search space with the current initial position of the LiDAR as the center, a bounding box radius threshold as the radius, and simultaneously constrained by an upper and lower height limit. The construction sequence is as follows: First, based on the known location, verify the upper envelope of the translational deviation of the rough global initial pose relative to the closed-loop correction pose in the road segment, and superimpose the half-diagonal length corresponding to the side length of the prior point cloud voxels to obtain the bounding box radius threshold; then, obtain the scanning center height range based on the LiDAR installation height and installation tolerance; subsequently, extend the first height margin downwards according to the ground undulation envelope to obtain the lower height limit; then, extend the second height margin upwards according to the upper limit of the main structure height of the shelf to obtain the upper height limit. Through the above sequence, reflections from top beams, light fixtures, and abnormal ground echoes below the effective scanning surface are excluded from the local search range.

[0085] After the bounding box of the truncated sphere in 3D space is determined, step S4 extracts only the local prior point set falling inside the bounding box from the prior point cloud data associated with the hybrid feature map. Then, the local prior point set is organized by voxel unit to generate a prior point cloud pixel set. A prior point cloud pixel is defined as a local plane element obtained by fitting several prior point cloud points within the same voxel unit, comprising three parts: the pixel center, the pixel normal vector, and the pixel support radius. The pixel center is constructed by taking the arithmetic mean of the coordinates of all prior point cloud points within the voxel unit. The pixel normal vector is constructed by minimizing the sum of the squared signed distances from all prior point cloud points within the voxel unit to the candidate plane, given a fixed pixel center, to obtain the normal direction per unit length. The pixel support radius is constructed by calculating the distances from all prior point cloud points within the voxel unit to the pixel center and taking the maximum value as the pixel support radius. The role of the pixel support radius is not to participate in the retrieval, but to provide a length conversion scale for subsequent normal direction differences, so that the normal direction error can be unified with the position error in terms of dimensions. At this point, step S4 has compressed the large-scale prior point cloud data in the hybrid feature map into a local prior point cloud pixel set, establishing a stable carrier for local rigid registration.

[0086] S403: Current point cloud patch construction, composite residual generation, and residual objective function establishment.

[0087] After the local prior point cloud pixel set is generated, step S4 further constructs the current point cloud patch from the current real-time point cloud data. The current point cloud patch is defined as a three-point local planar unit with a center point in the current real-time point cloud data as its core, and composed of the first neighbor point located at the previous scan position and the second neighbor point located at the subsequent scan position of the center point. The construction order is as follows: first, select a center point in the current real-time point cloud data, then select one neighbor point forward and one neighbor point backward according to the scan order, while requiring that the distance between the two neighbor points and the center point satisfy the distance continuity condition to avoid stitching across object boundaries or occlusion boundaries; then map the center point, the first neighbor point, and the second neighbor point to the world coordinate reference; finally, construct the normal vector of the current point cloud patch using the cross product direction of the three points, and perform normalization processing. The introduction of the current point cloud patch is to elevate the current real-time point cloud data from a discrete point set to a local geometric patch, so that the subsequent registration no longer depends only on the point-to-point relationship, but is transformed into a geometric consistency relationship from patch to pixel.

[0088] After the current point cloud patch is formed, step S4 truncates the bounding box of the sphere in three-dimensional space, finds the nearest graph element center for the center point of each current point cloud patch, and additionally checks whether the angle between the normal vector of the current point cloud patch and the normal vector of the graph element is consistent. In one embodiment, when the angle between the two does not exceed 20 degrees, it is determined that the normal orientation is consistent. Only pairs that are close in position and have consistent normal orientation are retained. Subsequently, step S4 constructs the normal distance residual, the normal direction residual converted length, and the composite residual. The normal distance residual is defined as the signed normal distance from the center point of the current point cloud patch to the supporting plane of the prior point cloud pixel. It is obtained by subtracting the pixel's center point from the current point cloud patch center point, and then projecting it onto the pixel's normal vector direction. The projected length is the normal distance residual. The normal direction residual converted length is defined as the result of converting the directional difference between the current point cloud patch normal vector and the pixel's normal vector into length. It is obtained by first calculating the sine of the angle between the two normal vectors, and then multiplying the sine by the pixel's supporting radius. Since the pixel's supporting radius itself has a length dimension, after conversion, the normal direction residual converted length has the same length dimension as the normal distance residual.

[0089] After obtaining both types of residuals, step S4 adds the square of the normal distance residual to the square of the converted length of the normal direction residual, and then takes the square root of the sum to obtain the composite residual. The composite residual is defined as a single geometric error that simultaneously reflects the current point cloud patch position offset and normal offset. After obtaining the composite residual, step S4 continues to establish the residual objective function. The residual objective function is defined as the sum of all composite residuals after robust smoothing mapping. The construction method is as follows: for each composite residual, first calculate the sum of the square of the composite residual and the square of the robust kernel smoothing factor, then take the square root of the sum, and finally subtract the robust kernel smoothing factor. Add all the processed results to form the residual objective function. The robust kernel smoothing factor is defined as a positive constant used to weaken the influence of outlier residuals. It is obtained by first reading the upper bound of laser ranging repeatability and the prior point cloud voxel side length, and then taking the larger of the two as the base value. When there is a lot of dynamic interference in the environment, take 1.5 times the base value as the robust kernel smoothing factor. By using this value selection method, the residuals of normal plane matching retain a linear response, while the abnormal residuals formed by occlusion points, reflective points, and dynamic interference points enter the smoothing weakening interval. For example, when a mobile robot stops at the intersection of a shelf aisle and a white wall, the reflection of the forklift's metal surface and the edge of a temporary stack may appear in the current real-time point cloud data. After robust smoothing mapping, the matching of the real shelf plane and the corner plane will continue to dominate the iteration direction, and local anomalies will not pull the pose update to the wrong area.

[0090] S404: Gauss-Newton method iteration, convergence tolerance lower limit judgment and accurate six-DOF relocation pose output.

[0091] After establishing the residual objective function, step S4 uses the Gauss-Newton method to perform local rigid registration on the current iterative pose. In each iteration, the current iterative pose is used as the expansion center to perform first-order linearization on all composite residuals, generating a system of linear equations for the six-dimensional pose increment, which consists of rotation increment vectors and translation increment vectors. Then, the system of linear equations is solved to obtain the rotation increment vector and translation increment vector for this iteration, and then the rotation increment vector and translation increment vector are updated to the current iterative pose through exponential mapping. To ensure that each iteration proceeds along a unified spatial reference, step S4 re-establishes the nearest neighbor relationship between the current point cloud patch and the prior point cloud pixels after the pose update, and simultaneously refreshes the normal distance residual, the normal direction residual reduced length, the composite residual, and the residual objective function, until the pose increment meets the convergence criterion or reaches the preset maximum number of iterations; in one embodiment, the preset maximum number of iterations can be fifteen to thirty.

[0092] To unify the rotation increment and translation increment under the same dimension, step S4 introduces the equivalent pose update length. The equivalent pose update length is defined as the unified update scale obtained by jointly converting the rotation increment vector and the translation increment vector in the current iteration. Specifically, it is constructed by first reading the median length of the distance from the center point of all current point cloud patches to the corresponding image mass center in the current iteration, using the median length as the rotation conversion length, then multiplying the magnitude of the rotation increment vector by the rotation conversion length to convert the angle change into the tangential displacement length, and finally combining the tangential displacement length and the magnitude of the translation increment vector by taking the square root of the sum of their squares to form a single length value.

[0093] In the convergence determination phase, step S4 sets a lower limit for the convergence tolerance. This lower limit is defined as the minimum update scale allowed at the end of local rigid registration. It is obtained by adding the upper bound of laser ranging repeatability to half the side length of a priori point cloud voxels. The upper bound of laser ranging repeatability is obtained statistically from the envelope of repeated scanning errors of targets on the same plane, and half the side length of a priori point cloud voxels represents the minimum structural resolution scale of the discrete representation of the priori point cloud. Only when the equivalent pose update length does not exceed the lower limit for convergence tolerance is the current iterative pose determined to have entered a stable region resolvable by the priori point cloud data. Upon reaching this condition, step S4 terminates the Gauss-Newton method iteration and writes the final iterative pose as the precise six-DOF relocalization pose. If the equivalent pose update length is still higher than the lower limit for convergence tolerance after reaching the maximum number of iterations, a local registration non-convergence flag is output, and writing this result into the continuous localization state is prohibited. The precise six-DOF repositioning pose is defined as the final rigid body pose of the current LiDAR relative to the world coordinate reference. It includes the final rotation matrix, the final translation vector, and the equivalent final rotation quaternion, and is a unified spatial output shared by the navigation module, the repositioning module, and the subsequent continuous mapping module. If the subsequent processing requires the current camera pose, a rigid body coordinate transformation is performed using the LiDAR and camera extrinsic parameters fixed in step S1.

[0094] Specifically, the above are merely preferred embodiments of this application and are not intended to limit this application.

[0095] In this specification, the terms "example", "concept", "implementation method", etc. are used only to describe specific technical content and do not imply any limitation on the scope of protection.

[0096] The above embodiments are used to illustrate the technical concept, processing flow and implementation structure of the present invention; without changing the technical concept of the present invention, equivalent substitutions made to individual module names, processing order and engineering parameters will not affect the implementation of the technical solution of the present invention.

Claims

1. A fast initialization method for cross-modal matching constraints in point cloud images, characterized in that, Including the following steps: S1: Obtain prior image sequences and prior point cloud data, perform inverse ray tracing and projection alignment, construct a cross-modal training set, train a cross-modal semantic mapping model, and construct a hybrid feature map; S2: Acquire real-time environment images, input the real-time environment images into the image encoder branch of the cross-modal semantic mapping model, and generate the current geometric perception visual descriptor by combining the attention weight matrix; S3: Map the current geometric perception visual descriptor to the hybrid feature map, lock the target keyframe, establish feature matching point pairs, and obtain a rough global initial pose after discrimination by the adaptive neural fuzzy inference system; S4: Obtain the current real-time point cloud data, use the coarse global initial pose as the input of the absolute reference initial value to iterate the nearest point joint optimization framework, complete the local rigid registration within the search and matching space limited by the hybrid feature map, and output the accurate six-DOF relocalization pose.

2. The fast initialization method for cross-modal matching constraints of point cloud images according to claim 1, characterized in that, Step S1 includes performing inverse ray tracing on the two-dimensional pixel features in the prior image sequence based on the camera intrinsic parameter matrix and the extrinsic parameter calibration matrix of the LiDAR to the camera, and projecting and aligning them with the three-dimensional spatial coordinates in the prior point cloud data to construct a cross-modal training set. The cross-modal training set is input into the dual-branch encoder network to train the cross-modal semantic mapping model; then the geometric perception visual descriptors output from the prior image sequence are bound to the corresponding visual keyframes and prior pose nodes to construct a hybrid feature map.

3. The fast initialization method for cross-modal matching constraints of point cloud images according to claim 2, characterized in that, In step S1, projection alignment starts with the two-dimensional pixel features in a single frame image, establishes the correspondence between pixel observations and three-dimensional spatial points according to a unified world coordinate reference, extracts grayscale texture fragments around the aligned pixel positions, extracts local point set fragments around the aligned three-dimensional spatial points, and forms cross-modal correspondence units together with grayscale texture fragments, local point set fragments and corresponding three-dimensional spatial points, and then gathers all cross-modal correspondence units to form a cross-modal training set.

4. The fast initialization method for cross-modal matching constraints of point cloud images according to claim 1, characterized in that, Step S2 includes: after acquiring the real-time environment image, converting the real-time environment image into a normalized matrix consistent with the input dimension of the image encoder branch, and inputting the normalized matrix separately into the image encoder branch of the cross-modal semantic mapping model to perform forward inference, extracting the deep convolutional feature tensor and attention weight matrix, and then assigning the three-dimensional structural prior probability to the high gradient response region in the real-time environment image according to the attention weight matrix to generate the current geometric perception visual descriptor.

5. The fast initialization method for cross-modal matching constraints of point cloud images according to claim 4, characterized in that, In step S2, the current geometric perception visual descriptor generation process includes: constructing a focus consistency map based on the attention weight matrix, constructing a gradient magnitude matrix based on the normalization matrix, constructing a three-dimensional structure prior probability map based on the focus consistency map and the gradient magnitude matrix, then using the three-dimensional structure prior probability map to perform structure modulation on the deep convolution feature tensor, and performing channel convergence within the structure pixel region to obtain the current geometric perception visual descriptor.

6. The fast initialization method for cross-modal matching constraints of point cloud images according to claim 1, characterized in that, Step S3 includes mapping the current geometric perception visual descriptor to the high-dimensional semantic space corresponding to the hybrid feature map, locking the target keyframe from the candidate keyframe set based on Euclidean distance, extracting the structural anchor set from the cross-modal corresponding units associated with the target keyframe, extracting the current local feature set on the real-time environment image, and filtering the feature matching point pair set based on bidirectional nearest neighbor matching and random sampling consensus algorithms.

7. The fast initialization method for cross-modal matching constraints of point cloud images according to claim 6, characterized in that, In step S3, the mapping depth span range rate is calculated for the spatial depth values ​​corresponding to the feature matching point pair set. Laplacian high-pass filtering is performed on the pixel neighborhood corresponding to the feature matching point pair set in the real-time environment image. The root mean square of the pixel edge response is calculated. The mapping depth span range rate and the root mean square of the pixel edge response are input into the adaptive neural fuzzy inference system. The relocation benchmark adoption coefficient is output and then compared with the adoption threshold.

8. The fast initialization method for cross-modal matching constraints of point cloud images according to claim 7, characterized in that, In step S3, when the relocation reference adoption coefficient reaches the adoption threshold, the prior pose node bound to the target keyframe is retrieved, and the 3D world coordinates of the feature matching point pair set and the pixel positions in the real-time environment image are used to construct a 3D to 2D perspective multi-point localization solution relationship. The perspective n-point localization solution process is used to obtain the coarse global initial pose, and the obtained rotation matrix is ​​subjected to singular value decomposition orthogonalization processing. The coarse global initial pose serves as the source of the absolute reference initial value in step S4.

9. The fast initialization method for cross-modal matching constraints of point cloud images according to claim 1, characterized in that, Step S4 includes extracting current real-time point cloud data with the same timestamp as the real-time environment image, converting the rough global initial pose into an absolute reference initial value under the current lidar coordinate reference, constructing a three-dimensional spatial truncated sphere bounding box around the absolute reference initial value, and then extracting the local prior point set falling within the three-dimensional spatial truncated sphere bounding box from the prior point cloud data associated with the hybrid feature map to generate a prior point cloud pixel set.

10. A fast initialization method for cross-modal matching constraints of point cloud images according to claim 9, characterized in that, In step S4, the current point cloud patch is constructed from the current real-time point cloud data, and a local nearest neighbor matching relationship is established between the current point cloud patch and the prior point cloud pixel set. Then, the composite residual is constructed based on the normal distance residual and the normal direction residual. The residual objective function is formed by all composite residuals. The pose is iteratively updated using the Gauss-Newton method until the equivalent pose update length reaches the lower limit of the convergence tolerance. The accurate six-degree-of-freedom relocalization pose is then output.

Images