Monocular visual relocalization method and system based on WiFi fingerprint assistance and three-dimensional Gaussian scene representation

Abstract

The application discloses a monocular visual repositioning method and system based on WiFi fingerprint assistance and a three-dimensional Gaussian scene representation. The method comprises the following steps: image acquisition, acquiring a query image collected by a monocular camera of a robot; WiFi coarse positioning, generating an initial six-degree-of-freedom pose hypothesis and corresponding spatial uncertainty information; virtual camera rendering, non-uniformly setting a plurality of candidate virtual reference cameras in a pre-constructed three-dimensional Gaussian scene representation according to the spatial uncertainty information, and respectively rendering a plurality of virtual reference images and corresponding geometric information through the plurality of candidate virtual reference cameras; semantic perception selection, selecting an optimal virtual reference image from the plurality of virtual reference images; corresponding relationship construction, constructing a two-dimensional-three-dimensional corresponding relationship between two-dimensional pixels in the query image and three-dimensional points of a target scene; and then sequentially performing pose refinement, cross-modal verification and fingerprint library online updating. The application is suitable for robot monocular visual positioning.

Description

Technical Field

[0001] This invention relates to the field of monocular vision localization for robots, specifically to a monocular vision relocalization method and system based on WiFi fingerprint assistance and 3D Gaussian scene representation. Background Technology

[0002] When robots operate in environments such as indoors, underground, warehouses, factories, shopping malls, and office buildings, they need to continuously obtain their position and orientation in the scene's coordinate system. Global Navigation Satellite Systems (GNSS) are typically unavailable or insufficiently accurate in these environments. While dedicated external positioning facilities such as Ultra-Wideband (UWB) base stations, Bluetooth beacons, visual markers, and motion capture systems can provide auxiliary positioning information, they require additional deployment, calibration, and maintenance, increasing system costs and limiting the deployment flexibility of robots in large-scale commercial scenarios.

[0003] Monocular vision relocalization, relying solely on a standard RGB camera, offers advantages such as low hardware cost, easy deployment, and broad compatibility with various robot platforms. Existing monocular vision relocalization methods can be broadly categorized into image retrieval and local feature matching-based methods, learning-based scene coordinate regression methods, and localization methods based on neural networks or explicit 3D representations.

[0004] In recent years, 3D Gaussian Splatting (3DGS) has emerged as a new renderable 3D scene representation method, capable of generating realistic reference images from candidate viewpoints and providing geometric information such as depth or 3D points. Based on this, researchers have proposed technical solutions for visual relocalization using 3DGS. For example, Chinese patent application CN119963641B discloses a 3D Gaussian splatting visual relocalization method driven by a visual fundamental model, which achieves relocalization by constructing a 3DGS spatial representation, extracting high-level features, and performing 2D-3D feature matching and pose optimization. SplatHLoc (CVPR 2026) further proposes a method that uses feature-based 3D Gaussian splatting to generate virtual viewpoints and performs high-precision pose estimation through mixed coarse and fine feature matching. These solutions demonstrate the feasibility of 3DGS in visual relocalization, but the initial pose acquisition still relies on visual image retrieval or feature matching. In visually degraded scenes such as insufficient visual texture, lighting changes, dynamic occlusion, or repetitive structures, the initial pose estimation is prone to failure, leading to the failure of the entire relocalization process.

[0005] On the other hand, many indoor environments already have WiFi access points deployed for communication, and the distribution of their received signal strength can form fingerprint information related to spatial location. WiFi fingerprint positioning does not rely on image texture, lighting, viewpoint overlap, or visible feature points; its signal source is independent of the visual image appearance. Existing technologies already include solutions that combine WiFi fingerprint positioning with visual positioning. For example, Chinese patent CN113382376B discloses an indoor positioning method based on WiFi and visual integration, which uses WiFi coarse positioning to filter candidate point sets and then uses images for fine positioning. However, in this type of solution, WiFi positioning and visual positioning are loosely coupled serial relationships—WiFi is only used to provide initial position values, and visual fine positioning is completed independently, lacking deep information interaction and joint optimization between the two. Furthermore, its visual fine positioning part relies on traditional image matching or feature point positioning methods, failing to fully utilize the geometric information of the 3D scene, resulting in limited precision positioning accuracy in scenes with sparse or repetitive textures.

[0006] Chinese patent CN120655823A uses depth maps and normal maps as geometric priors in 3D reconstruction, but it is geared towards 3D reconstruction tasks rather than relocalization tasks and does not involve how to apply geometric priors to refine camera pose.

[0007] Therefore, there is an urgent need for a monocular robot relocalization scheme that combines global coarse localization using WiFi fingerprinting with fine localization using 3D scene geometry. This scheme should achieve deep and tight coupling between WiFi signals and 3D scene geometry without requiring additional dedicated positioning base stations, beacons, manual markers, or external tracking systems, while also considering global robustness, local positioning accuracy, and engineering deployability in scenarios with visual degradation. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the prior art and provide a monocular visual relocalization method and system based on WiFi fingerprint assistance and 3D Gaussian scene representation. This method addresses the problems in existing monocular relocalization methods, such as global initialization being easily affected by visual appearance degradation, lack of deep coupling between WiFi and visual positioning, insufficient local geometric accuracy, and lack of reliable initial values ​​for geometric refinement.

[0009] The present invention achieves the above-mentioned objectives by adopting the following technical solution: Firstly, the present invention provides a monocular visual relocalization method based on WiFi fingerprint assistance and 3D Gaussian scene representation, comprising:

[0010] Image acquisition: Acquire the query image captured by the robot's monocular camera;

[0011] WiFi coarse localization: Obtain WiFi fingerprint information scanned by the robot at the start or relocation time, and generate initial six-degree-of-freedom pose assumptions and corresponding spatial uncertainty information based on the matching results of the WiFi fingerprint information and the pre-built WiFi fingerprint database;

[0012] Virtual camera rendering: Based on the spatial uncertainty information, multiple candidate virtual reference cameras are non-uniformly set in the pre-constructed 3D Gaussian scene representation, i.e., a first sampling range and a first sampling density are used in the first degree of freedom direction, and a second sampling range and a second sampling density are used in the second degree of freedom direction; the uncertainty corresponding to the first degree of freedom direction is greater than the uncertainty corresponding to the second degree of freedom direction, the first sampling range is greater than the second sampling range, and the first sampling density is less than the second sampling density; multiple virtual reference images and their corresponding geometric information are rendered by the multiple candidate virtual reference cameras respectively;

[0013] Semantic-aware selection: The query image is semantically segmented to obtain a semantic layout, and the optimal virtual reference image is selected from the multiple virtual reference images based on a comprehensive score of semantic similarity and feature matching quantity between the semantic layout and the semantic layout of each candidate virtual reference image.

[0014] Correspondence construction: Establish an image feature matching relationship between the query image and the optimal virtual reference image; Based on the image feature matching relationship and the geometric information, construct a two-dimensional-three-dimensional correspondence between two-dimensional pixels in the query image and three-dimensional points in the target scene;

[0015] Pose refinement: The refined six-DOF pose of the robot camera is solved based on the joint cost function, which includes a geometric reprojection residual term and a WiFi signal likelihood residual term.

[0016] Cross-modal verification: Perform geometric consistency verification and WiFi-vision cross-modal consistency verification on the refined six-DOF pose, and output the refined six-DOF pose only if both pass;

[0017] Fingerprint database online update: After the refined six-DOF pose is verified, the refined six-DOF pose is associated with the current WiFi fingerprint information to generate a new fingerprint sample, and the WiFi fingerprint database is updated online.

[0018] Furthermore, the initial six-DOF pose assumption is obtained by WiFi fingerprint matching; wherein, the similarity between the access point identifier and its received signal strength obtained by the robot's current scan and the fingerprint samples in the pre-built WiFi fingerprint database is calculated, and one or more global coarse position candidates are obtained by sorting according to the similarity; each global coarse position candidate is combined with the robot's prior orientation, historical motion state, traversable area on the map or a preset heading angle set to generate the initial six-DOF pose assumption.

[0019] Furthermore, the spatial uncertainty information is jointly determined by the WiFi fingerprint matching score, the spatial distribution dispersion of candidate fingerprint samples, and the WiFi signal spatial attenuation model; the WiFi signal spatial attenuation model is constructed based on the variation law of signal strength with distance in the fingerprint database and is used to estimate the positioning uncertainty in each degree of freedom direction;

[0020] The sampling range and sampling density of the multiple candidate virtual reference cameras in each degree of freedom direction are determined according to the following rules:

[0021] Let the uncertainty in the direction d of the degree of freedom be σ_d, then the sampling range along this direction is R_d = k_d × σ_d, and the sampling density is ρ_d = λ_d / σ_d, where k_d and λ_d are preset scaling factors;

[0022] The degrees of freedom include at least translation along the camera's optical axis, translation perpendicular to the optical axis, and yaw rotation.

[0023] Furthermore, the 3D Gaussian scene representation contains multiple 3D Gaussian primitives, each of which includes at least position, scale, rotation, opacity, and color or radiation feature parameters; the 3D Gaussian scene representation also includes semantic label information, which is used to describe the category attributes of different regions in the scene;

[0024] The geometric information includes a depth map, a normal map, a 3D point map, or a scene coordinate map; the 2D-3D correspondence is obtained by backprojecting or indexing matching pixels in the virtual reference image to the geometric information;

[0025] The comprehensive score is calculated by weighting the semantic similarity, feature matching quantity, and reprojection error between the semantic layout of the candidate virtual reference image and the semantic layout of the query image; wherein, the semantic layout of the query image is obtained by semantic segmentation of the query image, and the semantic layout of the candidate virtual reference image is obtained by rendering the semantic label information pre-stored in the 3D Gaussian scene representation.

[0026] Furthermore, when establishing image feature matching relationships, a geometric consistency filter is constructed using the depth map, normal map, and 3D point map corresponding to the virtual reference image; the geometric consistency filter includes at least one of the following:

[0027] Based on the pose of the virtual reference camera and the initial six-degree-of-freedom pose assumption, the epipolar search range of the query image feature points in the virtual reference image is calculated, and the feature matching search is restricted to the epipolar search range.

[0028] The depth values ​​of candidate matching points in the virtual reference image are calculated based on the depth map, and the scale changes of the corresponding feature points are predicted accordingly. Feature extraction and matching are performed only at the feature pyramid level of the predicted scale matching.

[0029] Calculate the local 3D normal vector of the candidate matching point in the virtual reference image based on the normal map, and filter out candidate matches whose normal vector direction is inconsistent with the local gradient direction of the corresponding feature point in the query image;

[0030] The depth gradient is calculated based on the depth map, and candidate matches located near depth discontinuities are filtered out.

[0031] Furthermore, the likelihood residual term of the WiFi signal is constructed in the following manner:

[0032] A location-signal strength mapping model was pre-trained based on the WiFi fingerprint database;

[0033] Input the current candidate pose into the position-signal intensity mapping model to obtain the predicted signal intensity distribution;

[0034] The difference between the predicted signal intensity distribution and the currently scanned WiFi fingerprint information is calculated as the likelihood residual term of the WiFi signal;

[0035] In the joint cost function, the geometric reprojection residual term and the WiFi signal likelihood residual term are summed by a weighted factor, which is dynamically adjusted according to the WiFi signal quality and visual matching quality of the current environment.

[0036] Furthermore, the WiFi-visual cross-modal consistency verification includes:

[0037] The refined six-DOF pose is input into a pre-trained position-signal strength mapping model to obtain the predicted WiFi signal strength distribution.

[0038] Calculate the similarity between the predicted WiFi signal strength distribution and the currently scanned WiFi fingerprint information;

[0039] If the similarity is lower than a preset threshold, even if the geometric consistency verification passes, the refined six-DOF pose is still deemed unreliable.

[0040] Furthermore, the geometric consistency verification includes: when the number of interior points is lower than a preset threshold, the reprojection error is higher than a preset threshold, the spatial distribution of matching points does not meet preset conditions, or the difference between the refined pose and the initial pose exceeds a preset threshold, the refined six-degree-of-freedom pose is determined to be geometrically unreliable.

[0041] When the refined six-DOF pose is determined to be unreliable:

[0042] Obtain the robot's successful localization pose and motion model predicted pose from the previous moment;

[0043] The initial six-degree-of-freedom pose assumption is weighted and fused with the pose predicted by the motion model. The weighting coefficients are dynamically adjusted based on the WiFi fingerprint matching confidence and the proportion of visual matching inliers.

[0044] The fused pose is used as the new virtual reference camera center pose, and a candidate virtual reference camera is regenerated with a perturbation radius greater than the previous one. Rendering, feature matching, and PnP refinement are then re-executed.

[0045] If the robot is still deemed unreliable after a retry, its pose is propagated using Kalman filtering or extended Kalman filtering based on the previous state, motion model, and control variables. The propagated pose is then used as a temporary localization output and as the initial prior for rendering the virtual reference camera in the next frame.

[0046] Furthermore, the pre-constructed 3D Gaussian scene representation is obtained in the following manner:

[0047] Acquire multi-view images of the target scene and their corresponding camera poses or positional constraints;

[0048] Based on the multi-view images, 3D reconstruction is performed to obtain the initial camera pose and scene geometry;

[0049] Optimize the 3D Gaussian scene representation based on the initial camera pose and scene geometry;

[0050] Align the 3D Gaussian scene representation with the target scene coordinate system;

[0051] When optimizing the 3D Gaussian scene representation, a localization-oriented multi-task joint loss function is adopted; the multi-task joint loss function includes at least one of the following:

[0052] Photometric reconstruction loss between rendered and real-world captured images;

[0053] The feature consistency loss between the rendered image and the real-world captured image in the local feature description subspace is constructed based on the distance metric of known matching pixel pairs in the feature description subspace.

[0054] The geometric consistency loss between the rendered depth map and the depth reference generated by the motion-reconstructed structure;

[0055] The repeatability loss of feature points in the rendered image is used to encourage the pixels corresponding to the same 3D point in the rendered images from different viewpoints to have stable feature responses.

[0056] Secondly, the present invention provides a monocular vision relocalization system based on WiFi fingerprint assistance and three-dimensional Gaussian scene representation, comprising:

[0057] The image acquisition module is used to acquire the query image captured by the robot's monocular camera;

[0058] The WiFi coarse localization module is used to obtain the initial six-degree-of-freedom pose assumptions and corresponding spatial uncertainty information based on the WiFi fingerprint information obtained by the robot's current scan and the pre-built WiFi fingerprint database.

[0059] The virtual camera rendering module is used to non-uniformly set multiple candidate virtual reference cameras in a pre-constructed three-dimensional Gaussian scene representation based on the spatial uncertainty information, and render multiple virtual reference images and their geometric information.

[0060] The semantic awareness selection module is used to perform semantic segmentation on the query image to obtain a semantic layout, and select the optimal virtual reference image from the multiple virtual reference images based on a comprehensive score of semantic similarity and feature matching quantity between the semantic layout and the semantic layout of each candidate virtual reference image.

[0061] The correspondence construction module is used to perform feature matching between the query image and the optimal virtual reference image, and to construct a two-dimensional-three-dimensional correspondence based on the geometric information;

[0062] The pose refinement module is used to solve for the refined six-degree-of-freedom pose based on the joint cost function, which includes a geometric reprojection residual term and a WiFi signal likelihood residual term.

[0063] The cross-modal verification module is used to perform geometric consistency verification and WiFi-vision cross-modal consistency verification on the refined six-DOF pose, and outputs the refined six-DOF pose only when both pass.

[0064] The fingerprint database online update module is used to associate the refined six-DOF pose with the current WiFi fingerprint information to generate a new fingerprint sample after the refined six-DOF pose has been verified, and to update the WiFi fingerprint database online.

[0065] Compared with the prior art, the present invention has at least the following beneficial effects:

[0066] No additional positioning infrastructure required: During the online positioning phase, no additional positioning facilities such as GNSS, UWB, Bluetooth beacons, QR codes, reflective markers, or motion capture systems are needed; the system reuses existing WiFi access points in the target environment as coarse positioning signal sources, and completes fine positioning through monocular vision and 3D scene geometry, which greatly reduces the system deployment and maintenance costs.

[0067] WiFi and vision are deeply tightly coupled: Unlike the loosely coupled architecture of existing WiFi + vision fusion solutions where WiFi only provides initial values, in this invention, WiFi fingerprint information participates in the positioning closed loop throughout the entire process. It not only provides the initial pose, but also guides the virtual camera sampling strategy through spatial uncertainty information, participates in joint optimization through the signal likelihood residual term, participates in the result reliability verification through cross-modal consistency verification, and realizes online adaptive updating of the fingerprint database through refined result feedback, forming a positive feedback loop of deep tight coupling between "WiFi and vision".

[0068] Geometric prior depth fusion improves matching quality: By utilizing the depth map, normal map and 3D point map obtained synchronously from the rendering of the 3D Gaussian scene representation, multiple geometric constraints such as epipolar constraints, scale consistency, normal vector consistency and depth gradient filtering are introduced in the feature matching stage, so that feature matching is upgraded from pure appearance-driven to "appearance + geometry" joint-driven, which significantly improves the matching inlier ratio in weak texture and repetitive texture scenes.

[0069] Optimization of 3D scene representation for localization: In the offline training of 3D Gaussian scene representation, a multi-task loss function is introduced, including feature consistency loss, geometric consistency loss, and feature point repeatability loss. This not only optimizes the rendering visual quality of the 3D scene representation, but also ensures the stability of subsequent localization matching from the source, which is different from the conventional 3DGS training method that only optimizes the photometric quality.

[0070] Semantic-aware virtual camera selection: By introducing semantic label information, the virtual reference image with the most similar semantic layout to the query image is selected from multiple candidate virtual reference images for matching and refinement. High-level semantic information is used to guide the selection of virtual viewpoints, reducing the impact of low-level visual ambiguity on the localization results.

[0071] Cross-modal redundancy verification improves reliability: WiFi-vision cross-modal consistency verification is introduced as a redundancy verification mechanism independent of geometric consistency verification. In visually ambiguous scenarios (such as symmetrical long corridors and repetitive shelves), it can effectively identify refined results that are "geometrically correct but actually incorrect in location" and avoid erroneous positioning output.

[0072] Dynamic fingerprint database adaptive update: The WiFi fingerprint database is updated online through feedback from visual fine positioning results, enabling the fingerprint database to adapt to environmental changes (such as device movement, personnel movement, AP restart, etc.), thereby improving the long-term robustness of WiFi coarse positioning.

[0073] Active failure recovery and state propagation: When PnP refinement fails, a complete failure recovery closed loop is formed through mechanisms such as weighted fusion to predict pose, amplified perturbation retry, and Kalman filter state propagation. This avoids system interruption caused by single-frame visual failure and provides reliable pose priors for subsequent frames, ensuring the stability of continuous tracking. Attached Figure Description

[0074] Figure 1 This is a flowchart of a monocular vision relocalization method based on WiFi fingerprint assistance and 3D Gaussian scene representation provided by an embodiment of the present invention;

[0075] Figure 2 This is a block diagram of a monocular vision relocalization system based on WiFi fingerprint assistance and 3D Gaussian scene representation provided in an embodiment of the present invention. Detailed Implementation

[0076] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0077] like Figure 1 As shown, this invention provides a monocular vision relocalization method based on WiFi fingerprint assistance and 3D Gaussian scene representation, comprising:

[0078] Image acquisition: Acquire the query image captured by the robot's monocular camera;

[0079] WiFi coarse localization: Obtain WiFi fingerprint information scanned by the robot at the start or relocation time, and generate initial six-degree-of-freedom pose assumptions and corresponding spatial uncertainty information based on the matching results of the WiFi fingerprint information and the pre-built WiFi fingerprint database;

[0080] Virtual camera rendering: Based on the spatial uncertainty information, multiple candidate virtual reference cameras are non-uniformly set in the pre-constructed 3D Gaussian scene representation, i.e., a first sampling range and a first sampling density are used in the first degree of freedom direction, and a second sampling range and a second sampling density are used in the second degree of freedom direction; the uncertainty corresponding to the first degree of freedom direction is greater than the uncertainty corresponding to the second degree of freedom direction, the first sampling range is greater than the second sampling range, and the first sampling density is less than the second sampling density; multiple virtual reference images and their corresponding geometric information are rendered by the multiple candidate virtual reference cameras respectively;

[0081] Semantic-aware selection: The query image is semantically segmented to obtain a semantic layout, and the optimal virtual reference image is selected from the multiple virtual reference images based on a comprehensive score of semantic similarity and feature matching quantity between the semantic layout and the semantic layout of each candidate virtual reference image.

[0082] Correspondence construction: Establish an image feature matching relationship between the query image and the optimal virtual reference image; Based on the image feature matching relationship and the geometric information, construct a two-dimensional-three-dimensional correspondence between two-dimensional pixels in the query image and three-dimensional points in the target scene;

[0083] Pose refinement: The refined six-DOF pose of the robot camera is solved based on the joint cost function, which includes a geometric reprojection residual term and a WiFi signal likelihood residual term.

[0084] Cross-modal verification: Perform geometric consistency verification and WiFi-vision cross-modal consistency verification on the refined six-DOF pose, and output the refined six-DOF pose only if both pass;

[0085] Fingerprint database online update: After the refined six-DOF pose is verified, the refined six-DOF pose is associated with the current WiFi fingerprint information to generate a new fingerprint sample, and the WiFi fingerprint database is updated online.

[0086] The present invention will now be described in detail with reference to specific embodiments.

[0087] Example 1: Offline Scenario Construction and WiFi Fingerprint Database Construction

[0088] During the offline phase, robots, handheld acquisition devices, or mobile acquisition platforms acquire multi-view RGB images in the target environment. These multi-view RGB images can be ordinary pinhole camera images, or converted from panoramic images, fisheye images, or multi-camera images. During acquisition, wheeled odometer data, robot navigation coordinates, manual measurement points, photogrammetric poses, or other positional constraints can be recorded to align the scene representation to the target scene coordinate system.

[0089] After acquisition, the camera pose and initial scene geometry of the acquired images are estimated using Structure from Motion (SfM), visual SLAM, photogrammetry software, or other 3D reconstruction methods.

[0090] Furthermore, the 3D Gaussian scene representation (3DGS) is trained or optimized using the aforementioned camera pose and initial scene geometry, enabling it to render a reference image of the target scene from given camera intrinsic and extrinsic parameters. Each Gaussian primitive in the 3D Gaussian scene representation may include a center position, covariance or scale rotation parameter, opacity, color parameter, spherical harmonic coefficients, or other radiative features.

[0091] When optimizing the 3D Gaussian scene representation, a localization-oriented multi-task joint loss function is employed. Conventional 3DGS training only optimizes the photometric reconstruction loss (such as a combination of L1 and SSIM loss) between the rendered image and the ground truth image. In this embodiment, in addition to the photometric reconstruction loss, the following auxiliary loss is introduced:

[0092] Feature consistency loss: Select image pairs with overlapping viewpoints from the training images, and use SuperPoint or similar local feature extractors to extract known matching pixel pairs from the image pairs. For each pair of matching points (p_i, q_i), calculate their descriptor distance in the feature space of the rendered image. The feature consistency loss is defined as the mean of the descriptor distances of all matching point pairs, which is used to encourage pixels corresponding to the same 3D point to have similar feature descriptors in images rendered by 3DGS from different viewpoints.

[0093] Geometric consistency loss: The sparse depth map or depth reference generated by SfM or visual SLAM is compared with the depth map rendered by 3DGS, and the difference between the two is calculated as the geometric consistency loss.

[0094] Feature point repeatability loss: Training images from multiple viewpoints are selected, and the feature response intensity of the corresponding pixel in the rendered image for the same 3D point is calculated at each viewpoint. Repeatability loss encourages the feature response intensity to remain stable across different viewpoints.

[0095] The joint loss function for multiple tasks is expressed as:

[0096] L_total = L_photometric + α×L_feature + β×L_geometric + γ×L_repeatability;

[0097] Here, α, β, and γ are the weighting coefficients of each loss term, which can be determined experimentally based on the specific scenario.

[0098] Through the optimization of the multi-task joint loss function, the 3D Gaussian scene representation not only faithfully reproduces the real scene in terms of visual appearance, but also provides favorable conditions for subsequent localization tasks in terms of feature matching, geometric accuracy and viewpoint repeatability.

[0099] Simultaneously, during the offline phase, a WiFi fingerprint database is constructed using WiFi scanning data collected as the robot moves within the target scene. Specifically, the identification information (such as BSSID) and received signal strength (RSSI) of observable access points are recorded at multiple collection locations to form fingerprint samples. Each fingerprint sample is associated with the three-dimensional coordinates (or 2.5D coordinates: x, y, floor) of the collection location. This location coordinate is mapped to the coordinate system of the three-dimensional Gaussian scene representation through offline calibration, ensuring spatial alignment between the WiFi fingerprint database and the three-dimensional scene representation.

[0100] The access points in the WiFi fingerprint database are preferably existing communication access points in the target environment, eliminating the need to deploy additional wireless base stations or beacons for the positioning task.

[0101] Example 2: WiFi coarse positioning and uncertainty estimation

[0102] During the online localization phase, the robot acquires query images using a monocular RGB camera and obtains WiFi scan information at the time of startup or relocation via a WiFi module. This WiFi scan information includes the identification information of observable access points and their received signal strength.

[0103] The system matches the current WiFi scan information with an offline WiFi fingerprint database. Matching methods can include K-Nearest Neighbors (KNN), Weighted K-Nearest Neighbors (WKNN), Gaussian process regression, or neural network-based methods. Specifically, it calculates the similarity between the current scanned fingerprint and each fingerprint sample in the database (e.g., a weighted sum of Euclidean distance, cosine similarity, or signal strength differences), and sorts them by similarity to obtain one or more global coarse position candidates.

[0104] For each coarse position candidate, one or more initial six-degree-of-freedom pose assumptions are generated by combining the robot's prior orientation, historical motion state, traversable area on the map, or a preset set of heading angles (such as 0°, 90°, 180°, 270°). Among them, the roll angle and pitch angle are set to fixed values ​​(such as roll 0°, pitch 0°) based on the ground normal vector of the corresponding position in the 3D Gaussian scene representation or the robot's motion plane constraints.

[0105] Simultaneously, the system generates spatial uncertainty information corresponding to each coarse position candidate. This uncertainty information is determined by a combination of the following factors:

[0106] WiFi fingerprint matching score: The higher the matching score (i.e., the more similar the current scan is to the fingerprint sample), the lower the uncertainty.

[0107] Spatial distribution dispersion of candidate fingerprint samples: If multiple fingerprint samples with high matching scores are spatially dispersed, the uncertainty is high; if they are concentrated in one place, the uncertainty is low.

[0108] WiFi signal spatial attenuation model: A signal attenuation model is constructed based on the variation of signal strength with distance in the fingerprint database, which is used to estimate the positioning uncertainty in each degree of freedom direction.

[0109] Specifically, based on the spatial attenuation model of WiFi signals, the positioning uncertainty along different directions can be estimated. Since WiFi signals are affected by obstacles such as walls and furniture in indoor environments, the signal attenuation characteristics are different in different directions. Therefore, the positioning uncertainty along different degrees of freedom exhibits anisotropic characteristics.

[0110] The initial six-DOF pose assumption does not require achieving the final navigation accuracy. Its role is to provide the subsequent virtual reference camera with a candidate view that is close to the real query view, while the spatial uncertainty information is used to guide the sampling strategy of the subsequent virtual reference camera.

[0111] Example 3: Uncertainty-Driven Non-Uniform Sampling of Virtual Reference Camera

[0112] The system sets up multiple candidate virtual reference cameras based on the initial six-degree-of-freedom pose assumptions and spatial uncertainty information.

[0113] The intrinsic parameters of the virtual reference camera can be consistent with those of the robot's monocular camera, or they can be adjusted based on the query image resolution, field of view, or distortion correction results. The extrinsic parameters of the virtual reference camera generate multiple candidate extrinsic parameters around the initial six-DOF pose assumption.

[0114] The multiple candidate virtual reference cameras are generated non-uniformly based on spatial uncertainty information. Specifically:

[0115] Let the uncertainty in the direction d be σ_d. Then, the sampling range along this direction is R_d = k_d × σ_d, and the sampling density is ρ_d = λ_d / σ_d, where k_d and λ_d are preset scaling factors. In other words, a larger sampling range and a sparser sampling density are used in the direction of the degree of freedom with larger uncertainty (such as along the camera optical axis, where the depth estimation uncertainty is usually large); a smaller sampling range and a denser sampling density are used in the direction of the degree of freedom with smaller uncertainty (such as the heading angle, where WiFi fingerprint matching can distinguish directions well).

[0116] The degrees of freedom include at least translation along the camera's optical axis, translation perpendicular to the optical axis, and yaw rotation. For example, a sampling range of ±2m and a sampling density of one candidate per 0.5m can be used in the forward-backward direction (optical axis direction) around the WiFi candidate location, a sampling range of ±0.5m and a sampling density of one candidate per 0.25m can be used in the left-right direction (perpendicular to the optical axis), and a sampling range of ±30° and a sampling density of one candidate per 15° can be used in the yaw direction.

[0117] By employing this non-uniform sampling strategy, the system can cover the largest pose uncertainty space with the fewest number of virtual reference cameras, effectively addressing the issue of inaccurate WiFi coarse positioning.

[0118] Each candidate virtual reference camera renders a virtual reference image and its depth map, normal map, and 3D point map from a 3D Gaussian scene representation.

[0119] The 3D Gaussian scene representation also includes semantic label information. When building 3DGS offline, semantic category labels (such as "corridor," "shelf," "doors and windows," "corner," etc.) can be assigned to different regions in the scene through manual annotation, automatic annotation by a semantic segmentation network, or a combination of both. This semantic label information is stored in association with 3D Gaussian primitives, and a semantic layout map can be rendered synchronously when rendering the virtual reference image.

[0120] Example 4: Semantic-Aware Virtual Reference Camera Selection

[0121] Semantic segmentation is performed on the query image to obtain its semantic layout. Semantic segmentation can be implemented using known semantic segmentation networks such as DeepLab, PSPNet, and SegFormer. The semantic layout of the query image describes the category attributes of different pixel regions in the query image (e.g., "there are shelves on the left and an aisle in front").

[0122] Simultaneously, the semantic layout of each virtual reference image is rendered from the virtual reference images rendered by the multiple candidate virtual reference cameras using the semantic label information stored in the three-dimensional Gaussian scene representation.

[0123] For each candidate virtual reference camera, calculate the semantic similarity between its semantic layout and the semantic layout of the query image. The semantic similarity can be calculated by: calculating the ratio of corresponding pixel categories in the two semantic layouts to be consistent, or using the intersection-over-union (IoU) metric, or calculating the cosine similarity of the semantic category histograms.

[0124] Simultaneously, local features are extracted from the query image and each candidate virtual reference image, and preliminary matching is performed, recording the number of feature matches. The reprojection error between the virtual reference camera pose corresponding to each candidate virtual reference image and the initial six-DOF pose assumption is calculated.

[0125] A comprehensive score is calculated based on a weighted average of semantic similarity, the number of feature matches, and reprojection error. Specifically:

[0126] Score = w_s×S_semantic + w_f×N_matches - w_r×E_reprojection

[0127] Where S_semantic is the semantic similarity, N_matches is the number of feature matches, E_reprojection is the reprojection error, and w_s, w_f, and w_r are the corresponding weight coefficients.

[0128] The virtual reference image with the highest overall score is selected as the optimal virtual reference image for subsequent feature matching and pose refinement.

[0129] By introducing semantic information, the system can first select the virtual viewpoint that is most similar to the spatial layout of the query image at the high-level semantic level before matching the underlying visual features, thereby reducing the risk of matching failure due to improper viewpoint selection.

[0130] Example 5: Feature Matching Based on Geometric Prior Deep Fusion

[0131] The system extracts local image features from the query image and the optimal virtual reference image, respectively. These local image features can be manually designed features (such as SIFT, ORB) or learned local features (such as SuperPoint, D2-Net, LoFTR).

[0132] During feature extraction and matching, the system utilizes the depth map, normal map, and 3D point map corresponding to the virtual reference image to construct a geometric consistency filter, integrating geometric prior depth into the feature matching process. Specifically, this includes the following four geometric constraints:

[0133] Epipolar search constraint: Based on the pose of the virtual reference camera and the initial six-DOF pose assumptions, the epipolar lines of the query image feature points in the virtual reference image are calculated. This narrows the search range for feature matching from the entire image to a band-shaped region near the epipolar line (bandwidth can be dynamically adjusted according to pose uncertainty), significantly reducing the number of candidate matches, improving matching efficiency, and filtering out false matches that do not conform to the epipolar geometric relationship.

[0134] Scale consistency constraint: The depth values ​​of candidate matching points in the virtual reference image are calculated based on the depth map. Under perspective projection, the larger the depth value, the smaller the pixel scale corresponding to the same physical size in the image. Based on this, the scale change of the corresponding feature point in the query image is predicted, and feature extraction and matching are performed only at the feature pyramid level of the predicted scale matching. For example, if the depth of a feature in the virtual reference image is 5 meters, and the depth of the corresponding feature in the query image is 2 meters, then the scale of that feature in the query image should be approximately 2.5 times the feature scale in the virtual reference image. This constraint filters out candidate matches whose scale changes do not conform to geometric relationships.

[0135] Normal vector consistency constraint: Calculate the local 3D normal vectors of candidate matching points in the virtual reference image based on the normal map. For the corresponding feature point in the query image, its surface orientation can be estimated from its local gradient direction. Candidate matches whose normal vector directions are inconsistent with the local gradient directions of the corresponding feature points in the query image are filtered out. This constraint effectively filters matches with inconsistent surface orientations caused by changes in viewpoint.

[0136] Depth gradient constraint: Calculate the depth gradient based on the depth map to identify depth discontinuities (such as object edges and contour boundaries). Filter out candidate matches located near depth discontinuities. Feature points in depth discontinuities may experience occlusion or abrupt parallax changes under different viewpoints, which is detrimental to establishing stable 2D-3D correspondences.

[0137] Through the above four-fold geometric consistency screening, the system eliminates a large number of outliers during the feature matching stage, which significantly improves the proportion of interior points and the accuracy of pose estimation in subsequent PnP solutions.

[0138] Example 6: Construction of Two-Dimensional-Three-Dimensional Correspondence

[0139] For each pair of matching points retained after geometric consistency filtering, let the matching pixel in the query image be u_q and the matching pixel in the virtual reference image be u_r.

[0140] The system obtains the target scene 3D point X_r corresponding to u_r based on the depth map, 3D point map, or scene coordinate map corresponding to the virtual reference image. Specifically, if a depth map is used, u_r is back-projected into 3D space using the intrinsic parameters and depth values ​​of the virtual reference camera; if a 3D point map or scene coordinate map is used, the 3D coordinates corresponding to the position of u_r are directly indexed.

[0141] Therefore, the system obtains the two-dimensional-three-dimensional correspondence between the queried pixel u_q and the three-dimensional point X_r: u_q X_r.

[0142] Compared to directly matching between the query image and the sparse 3D map, this embodiment first establishes a 2D match using a virtual reference image, and then uses rendered geometric information to convert it into 3D constraints. This can increase the number of available matches in scenarios with sparse textures, large changes in viewpoint, or map points that are not easy to describe directly.

[0143] Example 7: Pose Refinement Based on Joint Cost Function

[0144] The system inputs the 2D-3D correspondence into a perspective n-point (PnP) solver with a Random Sample Consensus (RANSAC) mechanism to estimate the initial refined pose of the query camera in the target scene coordinate system. The RANSAC mechanism is used to iteratively remove outliers during the solution process.

[0145] Furthermore, this embodiment constructs a joint cost function to perform nonlinear optimization on the initial refined pose. The joint cost function includes a geometric reprojection residual term and a WiFi signal likelihood residual term.

[0146] The geometric reprojection residual term is constructed based on the reprojection error of the PnP algorithm:

[0147] ;

[0148] Where T represents the current candidate camera pose, and Proj(·) is the projection function that projects the 3D points... Projected onto the query image plane, Match pixels for the corresponding query image.

[0149] The likelihood residual term of the WiFi signal is constructed in the following way:

[0150] A location-signal strength mapping model is pre-trained based on a WiFi fingerprint database. This model can be implemented using methods such as Gaussian Process Regression, random forest, or neural networks, and is used to predict the signal strength distribution of each access point at the input location coordinates.

[0151] Input the position component t of the current candidate pose T into the position-signal intensity mapping model to obtain the predicted signal intensity distribution. Calculate the predicted signal strength distribution and the currently actually scanned WiFi fingerprint information. Differences between them:

[0152] ;

[0153] The joint cost function is expressed as:

[0154] ;

[0155] Here, η is a weighting factor that is dynamically adjusted based on the current WiFi signal quality (such as the number of observable access points and signal strength variance) and visual matching quality (such as the number of interior points and average reprojection error). When the WiFi signal quality is high, the value of η is increased, allowing WiFi constraints to play a greater role in optimization; when the visual matching quality is high, the value of η is decreased, allowing visual geometric constraints to dominate the optimization.

[0156] By minimizing the aforementioned joint cost function, and using the Gauss-Newton method or the Levenberg-Marquardt algorithm for iterative optimization, a refined six-DOF pose is obtained.

[0157] This joint optimization mechanism transforms WiFi signals from merely a "one-off" information source providing initial values ​​into a constraint term that participates in the entire pose refinement optimization process alongside visual geometric constraints, thus achieving deep and tight coupling between WiFi and vision.

[0158] Example 8: Geometric Consistency Verification and Cross-Modal Consistency Verification

[0159] After obtaining the refined six-DOF pose, the system performs dual verification.

[0160] Geometric consistency verification includes at least one of the following criteria:

[0161] (1) The number of PnP in-points is lower than the preset threshold (e.g., less than 15 in-points);

[0162] (2) The average reprojection error is higher than the preset threshold (e.g., greater than 5 pixels).

[0163] (3) The spatial coverage of the matching points in the query image does not meet the preset conditions (e.g., the matching points are concentrated in a local area of ​​the image).

[0164] (4) The translational change of the refined pose relative to the initial pose exceeds the preset threshold (e.g., greater than 1 meter) or the rotational change exceeds the preset threshold (e.g., greater than 30°).

[0165] If any of the above conditions are met, the refined six-DOF pose geometry is deemed unreliable.

[0166] WiFi-visual cross-modal conformance verification includes:

[0167] The position components of the refined six-DOF pose are input into a pre-trained position-signal strength mapping model to obtain the predicted WiFi signal strength distribution. .

[0168] Calculate the similarity between the predicted WiFi signal strength distribution and the currently scanned WiFi fingerprint information S_scan. Similarity calculation can use metrics such as cosine similarity, the reciprocal of Euclidean distance, or mutual information.

[0169] If the similarity is lower than a preset threshold (e.g., cosine similarity is less than 0.7), even if all geometric consistency verifications pass, the refined six-DOF pose is still deemed unreliable.

[0170] Cross-modal consistency verification leverages the independence of WiFi signals and visual images in terms of information source. In visually ambiguous scenarios (such as symmetrical long corridor environments, where visual geometric constraints may refine the pose to a mirror position, with reprojection errors still being small), geometric consistency verification may fail to identify erroneous localization. However, the correspondence between WiFi signals and spatial positions is unaffected by visual ambiguity and can serve as an independent verification source to identify such errors. Only when both geometric consistency verification and cross-modal consistency verification pass will the system output the refined six-DOF pose as the final robot localization result.

[0171] Example 9: Online Update of Dynamic WiFi Fingerprint Database

[0172] Once the refined six-DOF pose passes dual verification, the system uses it as a reliable positioning result. Simultaneously, the position components of the refined six-DOF pose are associated with the currently scanned WiFi fingerprint information to generate a new fingerprint sample.

[0173] The new fingerprint sample was added to the WiFi fingerprint database for subsequent coarse WiFi localization. A time-decrease weighted strategy was used to manage the fingerprint database.

[0174] (1) Newly added fingerprint samples have the highest initial weight. ;

[0175] (2) The weight of existing fingerprint samples decays exponentially over time: , where t is the sample existence time and τ is a preset time constant (e.g., 7 days);

[0176] (3) When the weight of a fingerprint sample is lower than a preset threshold, the sample is removed from the database.

[0177] Through this online update mechanism, the WiFi fingerprint database can adapt to environmental changes. For example, when some access points are moved or reconfigured, old fingerprint samples gradually decay and are removed, while new fingerprint samples are continuously added, ensuring that the fingerprint database always reflects the current WiFi signal distribution and effectively solving the performance degradation problem of static fingerprint databases during long-term deployment.

[0178] Meanwhile, based on the updated fingerprint database, the location-signal strength mapping model can be retrained or fine-tuned to make the calculation of the WiFi signal likelihood residual term more accurate, forming a positive feedback loop of "visual refinement → WiFi database update → WiFi coarse positioning improvement → more accurate visual refinement".

[0179] Example 10: Failure Active Recovery and State Propagation

[0180] When the refined six-DOF pose is determined to be unreliable (whether due to geometric consistency verification failure or cross-modal consistency verification failure), the system does not immediately abandon localization, but instead executes the following proactive failure recovery procedure:

[0181] (a) Obtain the robot's successful localization pose T_prev and motion model predicted pose T_motion from the previous moment. The motion model can infer the predicted pose at the current moment based on the robot's kinematic model (such as differential drive model, Ackerman model) and control variables (linear velocity, angular velocity).

[0182] (b) The initial six-DOF pose assumption T_init and the motion model predicted pose T_motion are weighted and fused: T_fused = α · T_init + (1-α) · T_motion, where the weighted fusion coefficient α is dynamically adjusted based on the WiFi fingerprint matching confidence and the proportion of inliers in the visual matching. If the WiFi matching confidence is high, α is larger, indicating greater trust in the initial value given by WiFi; if the proportion of inliers in the visual matching is low, α is smaller, indicating greater trust in the motion model prediction.

[0183] (c) Using the fused pose T_fused as the new virtual reference camera center pose, and regenerating the candidate virtual reference camera with a perturbation radius greater than that of the previous attempt (e.g., the perturbation radius was ±0.5m in the previous attempt, and this time it is increased to ±1.0m), and re-performing rendering, feature matching and PnP refinement are performed.

[0184] (d) If the retry is still deemed unreliable, the robot pose is propagated using Kalman filtering or Extended Kalman filtering (EKF) based on the previous state, motion model, and control variables. The state vector of the Kalman filter includes the robot's position, orientation, velocity, and angular velocity. When PnP update fails, a prediction step is performed: , , where F is the state transition matrix, u is the control variable, and Q is the process noise covariance.

[0185] The pose obtained from the propagation is used as a temporary localization output and as the initial prior for rendering the virtual reference camera in the next frame.

[0186] Through this failure active recovery mechanism, the system can maintain continuous localization capability when visual degradation causes single-frame PnP refinement failure, avoid system interruption, and quickly reconverge in subsequent frames when visual conditions improve.

[0187] Example 11: System Structure

[0188] like Figure 2 As shown, the present invention also provides a monocular vision relocalization system based on WiFi fingerprint assistance and three-dimensional Gaussian scene representation, including an image acquisition module, a WiFi coarse localization module, a virtual camera rendering module, a semantic perception selection module, a correspondence construction module, a pose refinement module, a cross-modal verification module, and a fingerprint database online update module.

[0189] The image acquisition module is used to receive query images captured by the robot's monocular camera.

[0190] The WiFi coarse localization module receives the robot's current WiFi scan information and estimates one or more initial six-degree-of-freedom pose assumptions and corresponding spatial uncertainties based on a pre-built WiFi fingerprint database. This module includes a WiFi scanning unit, a WiFi fingerprint database, a fingerprint matching and sorting unit, and an uncertainty estimation unit.

[0191] The virtual camera rendering module is used to non-uniformly set up multiple candidate virtual reference cameras in a pre-constructed 3D Gaussian scene representation based on the spatial uncertainty information, and render multiple virtual reference images and their geometric information. This module includes a 3D Gaussian renderer.

[0192] The semantic awareness selection module is used to perform semantic segmentation on the query image to obtain a semantic layout, and select the optimal virtual reference image from multiple virtual reference images based on a comprehensive score of semantic similarity and feature matching quantity between the semantic layout and the semantic layout of each candidate virtual reference image.

[0193] The correspondence construction module is used to perform feature matching between the query image and the optimal virtual reference image, and to construct a two-dimensional-three-dimensional correspondence based on geometric information. This module includes a geometric consistency filtering unit, which uses depth maps, normal maps, and three-dimensional point maps to perform geometric consistency filtering on candidate feature matches.

[0194] The pose refinement module is used to solve for refined six-DOF poses based on a joint cost function, which includes geometric reprojection residuals and WiFi signal likelihood residuals. This module includes a robust perspective n-point solver and a position-signal strength mapping model.

[0195] The cross-modal verification module is used to perform geometric consistency verification and WiFi-vision cross-modal consistency verification on the refined six-DOF pose. The refined six-DOF pose is output only when both pass.

[0196] The fingerprint database online update module is used to generate new fingerprint samples by associating the refined six-DOF pose with the current WiFi fingerprint information after the refined six-DOF pose has been verified, and to update the WiFi fingerprint database online using a time decay weighting strategy.

[0197] The modules described above can be deployed on a robot body computing platform, edge computing devices, or cloud servers. When the robot body's computing power is limited, compressed 3D Gaussian scene representation, low-resolution virtual reference images, lightweight feature extractors, or keyframe-triggered localization strategies can be used to reduce computational load and memory usage.

[0198] Example 12: Optional Variation

[0199] Without departing from the concept of the invention, the invention can also be modified in the following ways:

[0200] First, the representation of a 3D scene is not limited to a 3D Gaussian scene representation; it can also be a neural radiation field (NeRF), an explicit mesh, a point cloud, a voxel map, a textured 3D model, or a combination of multiple representations, as long as it can generate a reference image based on a virtual reference camera and provide geometric information that can be used for 2D-3D correspondence. When using a neural radiation field, the depth map and normal map can be obtained through cumulative opacity weighted calculation in volumetric rendering.

[0201] Second, WiFi fingerprint matching methods are not limited to KNN or WKNN. Gaussian process regression, random forest, convolutional neural network, metric learning or other machine learning methods can also be used for fingerprint matching and location estimation.

[0202] Third, local feature matching methods are not limited to a single feature extractor or matcher. They can employ traditional local features (SIFT, ORB, AKAZE), deep learning local features (SuperPoint, D2-Net, LoFTR, Swin-Transformer), optical flow tracing, dense matching, or semantic-assisted matching.

[0203] Fourth, semantic segmentation is not limited to DeepLab, PSPNet, and SegFormer; it can also use any known semantic segmentation network or large-scale vision foundation model (such as SAM).

[0204] Fifth, Kalman filtering is not limited to standard Kalman filtering or extended Kalman filtering; it can also employ unscented Kalman filtering (UKF), particle filtering, sliding window optimization, or factor graph optimization.

[0205] Sixth, this invention can be used in mobile robots, service robots, inspection robots, unmanned vehicles, drones, augmented reality devices, or other intelligent terminals that require monocular vision positioning.

[0206] This invention achieves a closed-loop process for robots, from global coarse localization via WiFi to precise geometric localization via monocular vision, by combining coarse WiFi fingerprint localization with precise monocular vision localization based on 3D Gaussian scene representation. Unlike existing loosely coupled architectures where WiFi only provides initial values, this invention involves WiFi signals throughout the localization closed loop. It guides the virtual camera's non-uniform sampling strategy through spatial uncertainty information, optimizes the joint cost function through the likelihood residual term of the WiFi signal, verifies the reliability of localization results through WiFi-vision cross-modal consistency verification, and enables online adaptive updates of the WiFi fingerprint database through refined localization result feedback. Simultaneously, through feature matching via geometric prior deep fusion, semantically aware virtual camera selection, localization-oriented multi-task 3D Gaussian scene representation optimization, and failure proactive recovery and state propagation mechanisms, the system's localization accuracy, robustness, and long-term deployability are comprehensively improved in visually degraded scenarios. This solution reuses existing WiFi signals in the environment, eliminating the need for additional dedicated localization base stations, beacons, manual markers, or external tracking systems, demonstrating significant engineering application value.

[0207] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A monocular visual relocalization method based on WiFi fingerprint assistance and 3D Gaussian scene representation, characterized in that, include: Image acquisition: Acquire the query image captured by the robot's monocular camera; WiFi coarse localization: Obtain WiFi fingerprint information scanned by the robot at the start or relocation time, and generate initial six-degree-of-freedom pose assumptions and corresponding spatial uncertainty information based on the matching results of the WiFi fingerprint information and the pre-built WiFi fingerprint database; The spatial uncertainty information is jointly determined by the WiFi fingerprint matching score, the spatial distribution dispersion of candidate fingerprint samples, and the WiFi signal spatial attenuation model; the WiFi signal spatial attenuation model is constructed based on the variation law of signal strength with distance in the fingerprint database and is used to estimate the positioning uncertainty in each degree of freedom direction; Virtual camera rendering: Based on the spatial uncertainty information, multiple candidate virtual reference cameras are non-uniformly set in the pre-constructed three-dimensional Gaussian scene representation, that is, a first sampling range and a first sampling density are adopted in the first degree of freedom direction, and a second sampling range and a second sampling density are adopted in the second degree of freedom direction. The uncertainty corresponding to the first degree of freedom direction is greater than the uncertainty corresponding to the second degree of freedom direction, the first sampling range is greater than the second sampling range, and the first sampling density is less than the second sampling density; Multiple virtual reference images and their corresponding geometric information are obtained by rendering the multiple candidate virtual reference cameras respectively; Semantic-aware selection: The query image is semantically segmented to obtain a semantic layout, and the optimal virtual reference image is selected from the multiple virtual reference images based on a comprehensive score of semantic similarity and feature matching quantity between the semantic layout and the semantic layout of each candidate virtual reference image. Correspondence construction: Establish an image feature matching relationship between the query image and the optimal virtual reference image; Based on the image feature matching relationship and the geometric information, a two-dimensional-three-dimensional correspondence relationship is constructed between two-dimensional pixels in the query image and three-dimensional points in the target scene; Pose refinement: The refined six-DOF pose of the robot's monocular camera is solved based on the joint cost function, which includes a geometric reprojection residual term and a WiFi signal likelihood residual term. Cross-modal verification: Perform geometric consistency verification and WiFi-vision cross-modal consistency verification on the refined six-DOF pose, and output the refined six-DOF pose only if both pass; Fingerprint database online update: After the refined six-DOF pose is verified, the refined six-DOF pose is associated with the current WiFi fingerprint information to generate a new fingerprint sample, and the WiFi fingerprint database is updated online.

2. The monocular visual relocalization method based on WiFi fingerprint assistance and 3D Gaussian scene representation according to claim 1, characterized in that, The initial six-degree-of-freedom pose assumption is obtained by WiFi fingerprint matching; wherein, the similarity between the access point identifier and its received signal strength obtained by the robot's current scan and the fingerprint samples in the pre-built WiFi fingerprint database is calculated, and one or more global coarse position candidates are obtained by sorting according to the similarity; each global coarse position candidate is combined with the robot's prior orientation, historical motion state, traversable area on the map or a preset heading angle set to generate the initial six-degree-of-freedom pose assumption.

3. The monocular vision relocalization method based on WiFi fingerprint assistance and 3D Gaussian scene representation according to claim 1, characterized in that, The sampling range and sampling density of multiple candidate virtual reference cameras in each degree of freedom direction are determined according to the following rules: Let the uncertainty in the direction d of the degree of freedom be σ_d, then the sampling range along this direction is R_d = k_d × σ_d, and the sampling density is ρ_d = λ_d / σ_d, where k_d and λ_d are preset scaling factors; The degrees of freedom include at least translation along the camera's optical axis, translation perpendicular to the optical axis, and yaw rotation.

4. The monocular vision relocalization method based on WiFi fingerprint assistance and 3D Gaussian scene representation according to claim 1, characterized in that, The 3D Gaussian scene representation contains multiple 3D Gaussian primitives, each of which includes at least position, scale, rotation, opacity, and color; the 3D Gaussian scene representation also includes semantic labeling information, which is used to describe the category attributes of different regions in the scene; The geometric information includes a depth map, a normal map, a 3D point map, or a scene coordinate map; the 2D-3D correspondence is obtained by backprojecting or indexing matching pixels in the virtual reference image to the geometric information; The comprehensive score is calculated by weighting the semantic similarity, feature matching quantity, and reprojection error between the semantic layout of the candidate virtual reference image and the semantic layout of the query image; wherein, the semantic layout of the query image is obtained by semantic segmentation of the query image, and the semantic layout of the candidate virtual reference image is obtained by rendering the semantic label information pre-stored in the 3D Gaussian scene representation.

5. The monocular vision relocalization method based on WiFi fingerprint assistance and 3D Gaussian scene representation according to claim 1, characterized in that, When establishing image feature matching relationships, a geometric consistency filter is constructed using the depth map, normal map, and 3D point map corresponding to the virtual reference image; the geometric consistency filter includes at least one of the following: Based on the pose of the virtual reference camera and the initial six-degree-of-freedom pose assumption, the epipolar search range of the query image feature points in the virtual reference image is calculated, and the feature matching search is restricted to the epipolar search range. The depth values ​​of candidate matching points in the virtual reference image are calculated based on the depth map, and the scale changes of the corresponding feature points are predicted accordingly. Feature extraction and matching are performed only at the feature pyramid level of the predicted scale matching. Calculate the local 3D normal vector of the candidate matching point in the virtual reference image based on the normal map, and filter out candidate matches whose normal vector direction is inconsistent with the local gradient direction of the corresponding feature point in the query image; The depth gradient is calculated based on the depth map, and candidate matches located near depth discontinuities are filtered out.

6. The monocular visual relocalization method based on WiFi fingerprint assistance and 3D Gaussian scene representation according to claim 1, characterized in that, The likelihood residual term of the WiFi signal is constructed in the following way: A location-signal strength mapping model was pre-trained based on the WiFi fingerprint database; Input the current candidate pose into the position-signal intensity mapping model to obtain the predicted signal intensity distribution; The difference between the predicted signal intensity distribution and the currently scanned WiFi fingerprint information is calculated as the likelihood residual term of the WiFi signal; In the joint cost function, the geometric reprojection residual term and the WiFi signal likelihood residual term are summed by a weighted factor, which is dynamically adjusted according to the WiFi signal quality and visual matching quality of the current environment.

7. The monocular vision relocalization method based on WiFi fingerprint assistance and 3D Gaussian scene representation according to claim 1, characterized in that, The WiFi-visual cross-modal consistency verification includes: The refined six-DOF pose is input into a pre-trained position-signal strength mapping model to obtain the predicted WiFi signal strength distribution. Calculate the similarity between the predicted WiFi signal strength distribution and the currently scanned WiFi fingerprint information; If the similarity is lower than a preset threshold, even if the geometric consistency verification passes, the refined six-DOF pose is still deemed unreliable.

8. The monocular vision relocalization method based on WiFi fingerprint assistance and 3D Gaussian scene representation according to claim 1, characterized in that, The geometric consistency verification includes: when the number of interior points is lower than a preset threshold, the reprojection error is higher than a preset threshold, the spatial distribution of matching points does not meet preset conditions, or the difference between the refined pose and the initial pose exceeds a preset threshold, the refined six-degree-of-freedom pose is determined to be geometrically unreliable. When the refined six-DOF pose is determined to be unreliable: Obtain the robot's successful localization pose and motion model predicted pose from the previous moment; The initial six-degree-of-freedom pose assumption is weighted and fused with the pose predicted by the motion model. The weighting coefficients are dynamically adjusted based on the WiFi fingerprint matching confidence and the proportion of visual matching inliers. The fused pose is used as the new virtual reference camera center pose, and a candidate virtual reference camera is regenerated with a perturbation radius greater than the previous one. Rendering, feature matching, and PnP refinement are then re-executed. If the robot is still deemed unreliable after a retry, its pose is propagated using Kalman filtering or extended Kalman filtering based on the previous state, motion model, and control variables. The propagated pose is then used as a temporary localization output and as the initial prior for rendering the virtual reference camera in the next frame.

9. The monocular vision relocalization method based on WiFi fingerprint assistance and 3D Gaussian scene representation according to claim 1, characterized in that, The pre-constructed 3D Gaussian scene representation is obtained in the following way: Acquire multi-view images of the target scene and their corresponding camera poses or positional constraints; Based on the multi-view images, 3D reconstruction is performed to obtain the initial camera pose and scene geometry; Optimize the 3D Gaussian scene representation based on the initial camera pose and scene geometry; Align the 3D Gaussian scene representation with the target scene coordinate system; When optimizing the 3D Gaussian scene representation, a localization-oriented multi-task joint loss function is adopted; the multi-task joint loss function includes at least one of the following: Photometric reconstruction loss between rendered and real-world captured images; The feature consistency loss between the rendered image and the real-world captured image in the local feature description subspace is constructed based on the distance metric of known matching pixel pairs in the feature description subspace. The geometric consistency loss between the rendered depth map and the depth reference generated by the motion-reconstructed structure; The repeatability loss of feature points in the rendered image is used to encourage the pixels corresponding to the same 3D point in the rendered images from different viewpoints to have stable feature responses.

10. A monocular vision relocalization system based on WiFi fingerprint assistance and 3D Gaussian scene representation, used to implement the monocular vision relocalization method based on WiFi fingerprint assistance and 3D Gaussian scene representation as described in any one of claims 1-9, characterized in that, include: The image acquisition module is used to acquire the query image captured by the robot's monocular camera; The WiFi coarse localization module is used to obtain the initial six-degree-of-freedom pose assumptions and corresponding spatial uncertainty information based on the WiFi fingerprint information obtained by the robot's current scan and the pre-built WiFi fingerprint database. The virtual camera rendering module is used to non-uniformly set multiple candidate virtual reference cameras in a pre-constructed three-dimensional Gaussian scene representation based on the spatial uncertainty information, and render multiple virtual reference images and their geometric information. The semantic awareness selection module is used to perform semantic segmentation on the query image to obtain a semantic layout, and select the optimal virtual reference image from the multiple virtual reference images based on a comprehensive score of semantic similarity and feature matching quantity between the semantic layout and the semantic layout of each candidate virtual reference image. The correspondence construction module is used to perform feature matching between the query image and the optimal virtual reference image, and to construct a two-dimensional-three-dimensional correspondence based on the geometric information; The pose refinement module is used to solve for the refined six-degree-of-freedom pose based on the joint cost function, which includes a geometric reprojection residual term and a WiFi signal likelihood residual term. The cross-modal verification module is used to perform geometric consistency verification and WiFi-vision cross-modal consistency verification on the refined six-DOF pose, and outputs the refined six-DOF pose only when both pass. The fingerprint database online update module is used to associate the refined six-DOF pose with the current WiFi fingerprint information to generate a new fingerprint sample after the refined six-DOF pose has been verified, and to update the WiFi fingerprint database online.

Images