A Passive Passenger Localization Method for Railway Passenger Stations Based on Multimodal Image Fusion

Abstract

This invention provides a passive passenger localization method for railway passenger stations based on multimodal image fusion. The method includes: performing data augmentation processing on images acquired by cameras and inertial sensors within the high-speed railway station area, extracting local and semantic features, and constructing an adaptive bag-of-words tree; using a graphics processing unit (GPU) to divide the bag-of-words tree into subsets to generate a global 3D point cloud model; using multiple sets of locality-sensitive hash functions to generate hash codes, and using the global 3D point cloud model as a physical space mapping benchmark, selecting high-similarity candidate images through a voting mechanism to obtain a preliminary pose estimate from the passenger's perspective; and smoothing the preliminary pose estimate from the passenger's perspective using Kalman filtering based on the camera intrinsic parameter matrix and rotation matrix from the passenger's perspective, outputting the passenger's location coordinates. This invention applies passive localization, reducing costs. It can provide accurate navigation support for passengers, provide passenger flow data for stations, and optimize operational efficiency.

Description

Technical Field

[0001] This invention relates to the field of passenger indoor positioning technology, and in particular to a passive positioning method for passengers in railway passenger stations based on multimodal image fusion. Background Technology

[0002] With the rapid development of high-speed rail networks, railway passenger stations, as transportation hubs, directly impact passenger travel efficiency and experience through their service capacity and operational management. Within large high-speed rail stations, due to complex spatial layouts and high passenger flow, passengers often face navigation difficulties, getting lost, or being stranded. Therefore, accurate indoor positioning technology has become a key element in improving station service quality and operational efficiency. Currently, passenger navigation and passenger flow management within stations face the following challenges:

[0003] Spatial complexity: The multi-layered structure and densely packed functional areas (ticket gates, waiting halls, commercial areas) make it difficult for traditional signage systems to meet real-time navigation needs;

[0004] Dynamic interference: During peak hours, the population density is >5 people / ㎡, and metal structures (steel frames, elevators) cause multipath attenuation of wireless signals;

[0005] Accuracy requirements: Passengers need sub-meter (<1 meter) positioning accuracy to accurately locate the ticket gate or shop, while existing active positioning technologies (such as Wi-Fi fingerprint) often have an error of more than 3 meters.

[0006] Against this backdrop, passive visual positioning technology has become a research hotspot. The core of passive visual positioning is to achieve positioning through environmental visual information (rather than external signals), but the following technical bottlenecks need to be overcome:

[0007] ① Large-scale scene reconstruction efficiency: The SfM (Structure from Motion) algorithm takes tens of hours to process 100,000 images;

[0008] ② Robustness of image retrieval: Single features (such as SIFT (Scale-invariant feature transform) have a high mismatch rate under lighting conditions and complex scene layouts;

[0009] ③ Real-time performance and resource consumption: The algorithm needs to be adapted to edge devices (such as mobile phones) to achieve lightweight deployment and application in passenger service scenarios within railway stations.

[0010] Currently, existing passive visual positioning methods for railway station passenger transport include:

[0011] ① Bluetooth beacon positioning: The principle is to deploy Bluetooth beacons (such as iBeacon) within the station, calculate the distance by receiving signal strength, and estimate the location by combining it with triangulation. This method has drawbacks such as high hardware cost (requiring one beacon to be deployed every 20 meters, more than 5,000 beacons are needed for a 100,000 square meter station, with deployment costs exceeding 2 million yuan), susceptibility to environmental signal interference, and dependence on user equipment (passengers need to enable Bluetooth and install a specific APP).

[0012] ② Wi-Fi fingerprint positioning: This method works by pre-collecting Wi-Fi signal strength (fingerprint database) at various locations within the station and then matching the real-time signal with the fingerprint database to determine the location. However, it also suffers from high maintenance costs and poor adaptability to dynamic environments with significant changes in passenger density within the station. (Patent No.: CN202411691394.X)

[0013] The disadvantages of the aforementioned passive visual positioning methods for railway station passengers include: these methods mainly rely on active positioning schemes, such as deploying Bluetooth beacons, wireless Wi-Fi access points, or RFID devices, utilizing the signal interaction between passengers' mobile devices and these infrastructures to achieve positioning. However, these methods have significant drawbacks: firstly, the deployment and maintenance costs of the equipment are high, especially in large stations where a large number of sensors are needed to cover a wide area; secondly, the complex station environment, with metal structures and dense crowds causing signal interference and attenuation, often prevents positioning accuracy from meeting sub-meter level requirements; furthermore, active positioning depends on passenger device support, and some passengers may not carry compatible devices or have related functions disabled, resulting in limited positioning coverage.

[0014] For high-speed railway passenger stations with complex spatial layouts and large scale, these methods still have limitations: they often only target a single object or a limited range of indoor areas, making it difficult to apply them directly to large-scale scenarios such as the entire railway station; at the same time, the existence of large-scale image datasets will significantly reduce the efficiency of image matching and retrieval, and existing visual positioning schemes are still insufficient in terms of real-time performance and robustness to dynamic interference under massive data.

[0015] Therefore, there is an urgent need for an efficient, accurate, and passive positioning method that does not require external equipment, in order to meet the needs of large-scale and complex scenarios in high-speed railway stations, provide convenient navigation services for passengers, and improve station management efficiency. Summary of the Invention

[0016] The embodiments of the present invention provide a passive localization method for passengers in railway passenger stations based on multimodal image fusion, so as to effectively locate passengers in railway passenger stations.

[0017] To achieve the above objectives, the present invention adopts the following technical solution.

[0018] A passive passenger localization method for railway passenger stations based on multimodal image fusion includes:

[0019] A multimodal image dataset was acquired and constructed using RGB-D cameras and inertial sensor devices within the functional area of ​​a high-speed railway station. Data augmentation processing was performed on the images in the multimodal image dataset using random occlusion and affine transformation.

[0020] Local and semantic features are extracted from the data-enhanced image to construct an adaptive bag-of-words tree;

[0021] The bag-of-words tree is divided into subsets using a graphics processing unit (GPU), key frames are selected using a key frame dynamic filtering mechanism, projection errors are optimized using the sparse light speed adjustment method, and a global 3D point cloud model is generated.

[0022] Multiple sets of locality-sensitive hash functions are used to generate hash codes. The global 3D point cloud model is used as the physical space mapping benchmark. A voting mechanism is used to filter high-similarity candidate images to obtain a preliminary pose estimate from the passenger's perspective.

[0023] Based on the camera intrinsic parameter matrix and rotation matrix from the passenger's perspective, the preliminary pose estimation from the passenger's perspective is smoothed by Kalman filtering, and the passenger's positioning coordinates are output.

[0024] Preferably, the process of acquiring and constructing a multimodal image dataset within the functional area of ​​a high-speed railway station using an RGB-D camera and inertial sensor equipment, and performing data augmentation processing on the images in the multimodal image dataset through random occlusion and affine transformation, includes:

[0025] RGB cameras and inertial measurement units (IMUs) are deployed within the functional area of ​​railway passenger stations. Hardware trigger signals are used to ensure that the timestamps of RGB images, depth maps and IMU data are aligned. The spatial coordinate system is based on the station center as the origin, with the z-axis pointing vertically upward and the x and y axes parallel to the station ground. Each RGB camera and IMU captures data synchronously to construct a multimodal image dataset.

[0026] Data augmentation is performed on the image data in the multimodal image dataset by randomly generating rectangular occlusion regions in the images. The texture of the occlusion objects is simulated with Gaussian noise to represent dynamic interference. Within the random occlusion regions, the noise value of each pixel follows a Gaussian distribution. Where the mean μ = 0, the pixel values ​​I(x,y) of the image within the occluded area are perturbed:

[0027]

[0028] Applying a random affine transformation matrix A to each image yields the data-enhanced image. The random affine transformation matrix A is defined as follows:

[0029]

[0030] Where the scaling factor s∈[0.8,1,2], the rotation angle θ∈[-30°,30°], and the translation amount t x ,t y ∈[-50,50] pixels.

[0031] Preferably, the step of extracting local and semantic features from the data-enhanced image and constructing an adaptive bag-of-words tree includes:

[0032] For each data-enhanced image, detect the corner points of the FAST spherical radio telescope, calculate the BRIEF descriptor, and generate a 256-dimensional binary feature vector using the rotation-invariant descriptor. The formula for compensating the direction angle θ of the rotation-invariant descriptor is as follows:

[0033]

[0034] I(x,y) is the image pixel intensity value, where x and y represent the two-dimensional coordinate index of the pixel in the image or neighborhood;

[0035] A semantic model was used to extract a 128-dimensional semantic feature vector from the original RGB image. The 256-dimensional binary feature vectors of the N images are combined with the 128-dimensional semantic feature vectors. Concatenate into a 384-dimensional hybrid feature vector

[0036] The HDBSCAN algorithm is used to process the hybrid feature vector. Hierarchical clustering is performed, and a core distance is constructed based on k-nearest neighbors to generate a weighted adjacency graph. Then, the minimum spanning tree is generated using the Prim algorithm, and pruning is performed based on the reach distance. The clustering levels in the tree are extracted, the cluster stability λ is calculated, and the ratio of the number of samples in the cluster to the cluster diameter is calculated. Clusters with λ < 0.7 are removed, and clusters with high stability are merged to obtain a hierarchical bag-of-words tree. Each leaf node in the bag-of-words tree stores the cluster center feature vector and the spatial coordinates of the corresponding image.

[0037] Preferably, the step of using a graphics processing unit (GPU) to divide the bag-of-words tree into subsets, selecting keyframes using a keyframe dynamic filtering mechanism, optimizing projection errors using the sparse light speed adjustment method, and generating a global 3D point cloud model includes:

[0038] The hierarchical cluster structure of the bag-of-words tree and the hybrid feature vectors The spatial coordinates of the corresponding image, as well as the camera intrinsic matrix K and the initial estimate of the shooting pose T. i =[R i |t iThe input is fed into the GPU, which divides the bag-of-words tree into C subsets {a1, a2, ..., a...} based on spatial proximity and feature similarity. c Each subset contains local features and semantic features;

[0039] The GPU processes each subset pair (a i ,a j Extract matching feature points {(p i ,p j The essential matrix E is calculated using the five-point method. ij The essential matrix E ij Decomposition yields the relative rotation matrix The objective function is to minimize the difference in rotation matrices between all subset pairs. The optimized objective function is:

[0040]

[0041] Among them, R i ,R j Let be the rotation matrices for cameras i and j, respectively. Let ||·|| be the estimated value of the rotation matrix between the two cameras. F Representing the Frobenius norm, we obtain the optimized global rotation matrix R. i ;

[0042] The optimized global rotation matrix R′ i With the initial translation t i Combined, the keyframe pose T is obtained. i ′=[R′ i |t i Based on keyframe pose T i ′ and matching feature point pairs {(p i ,p j The sparse three-dimensional point X is calculated. l Camera intrinsic parameters K and feature point matching relationship O = {(k, l, x)} kl )}, where x kl Represents a three-dimensional point X l At keyframe T k The pixel coordinate observations in ' are used to construct the objective function by minimizing the sum of reprojection errors, while simultaneously optimizing the poses and 3D points of all keyframes:

[0043]

[0044] Where π([x,y,z]) T = [x / z, y / z] T For perspective projection function, π(KT) k ′·X l )-xkl Defined as the projection error e between each observation (k,l) kl ρ is the Huber robust kernel function, defined as:

[0045]

[0046] Where δ is a hyperparameter;

[0047] The three-dimensional point set {X l} and keyframe {T k The data is divided into independent blocks and allocated to different thread blocks of the GPU. Then, the 3D point coordinates and keyframe poses are updated to the final optimized value, and the optimized and updated pose T is output. k ′ and optimized updated sparse 3D point cloud X l It includes vertex coordinates, color, and normal vectors. Using triangulation positioning method, it performs 3D point cloud reconstruction on the spatial 3D point set, generates a global 3D point cloud model covering the functional area of ​​railway passenger station, and realizes 3D scene reconstruction.

[0048] Preferably, the step of generating hash codes using multiple sets of locality-sensitive hash functions, using the global 3D point cloud model as a physical space mapping benchmark, filtering candidate images with high similarity to the passenger's viewpoint image through a voting mechanism, and obtaining a preliminary pose estimate from the passenger's viewpoint includes:

[0049] Use 4 sets of locality-sensitive hash functions Each group generates a 12-bit hash code:

[0050] h i (x)=sign(w i x+b i (7)

[0051] in, w is a 384-dimensional blended feature vector of the input image. i b is a random projection vector that follows a standard normal distribution, with dimensions 384×1. i The uniformly distributed random bias term is used to adjust the separation threshold of the hash function, and sign(·) is the sign function, which maps real values ​​to binary hash codes of ±1.

[0052] Images captured in real time by passengers q Feature F is extracted using the DenseNet-121 architecture. q Principal component analysis was used to reduce the dimensionality of the eigenvectors to 384 dimensions, and F was calculated. q The 4 sets of hash codes {h i (F q )}, Statistical hash table and F qReference images for hash code matching, sorted by the number of matching hash tables:

[0053]

[0054] in, The function is an indicator function; it is 1 if the hash codes are equal, and 0 otherwise. It retains the set C = {(F...} of candidate images with scores S ≥ 3. j ,T j For the candidate image set C, calculate its relationship with F. q The cosine similarity of the feature vectors is used to select a set number of images with the highest similarity as the final matching result. The pose of the final matched image {T} is then calculated. j The spatial positions of the passenger's current viewpoint and the viewpoint image in the global 3D point cloud model are uniformly aligned. The cosine similarity results between the viewpoints of multiple known high-similarity candidate reference images and the passenger's viewpoint image pose are used as weights. Weights are generated using the Softmax function based on the cosine similarity results and then weighted and averaged to obtain the weighted pose T. f :

[0055]

[0056] The weighted pose is used as a preliminary pose estimate T from the passenger's perspective. f =[R f |t f ], where R f The rotation matrix of the matched image is converted into a quaternion q. j The result of weighted averaging and conversion back to a rotation matrix is The result is a weighted average of the translation vectors. A voting mechanism is then used to filter the results.

[0057] Preferably, the step of smoothing the preliminary pose estimation of the passenger's perspective using Kalman filtering based on the camera intrinsic parameter matrix and rotation matrix from the passenger's viewpoint, and outputting the passenger's positioning coordinates, includes:

[0058] Position p init =[x init ,y init ,z init ] T and the initial rotation quaternion q init =[q w ,q x ,q y ,q z ] T As the passenger's initial position and attitude state, combined with the velocity component vectors in three directions v = [v x ,v y ,v z ] TThe state vector x = [p, v, q] is formed. T Assuming the passenger moves at a constant speed or remains stationary for a short period of time, the state at the next moment a+1 will be updated as follows:

[0059] p a+1 =p a +v a Δt (9)

[0060] v a+1 =v a (10)

[0061]

[0062] Where ω is the angular velocity, and the weighted pose T f As observed value z k The predicted and observed values ​​are obtained through Kalman gain fusion and the state update formula:

[0063] K a+1 =P a+1|a +H T HP a+1|a H T +R a+1 ) -1 (12)

[0064] x a+1 =x a+1|a +K a+1 (z k -Hx a+1|a (13)

[0065] P a+1 =(IK a+1 H)P a+1|a (14)

[0066] Where z k -Hx a+1 K represents the difference between observation and prediction. f R is used to balance the confidence levels between predicted and observed values. a+1 To observe the noise, after multiple iterations, P a+1 The system gradually converges, the state estimation stabilizes, and the final output state update includes the passenger's final position p. a+1 and the state vector x of the attitude a+1 .

[0067] As can be seen from the technical solutions provided by the embodiments of the present invention above, the method of the present invention applies passive positioning, reducing costs: it does not require the deployment of external beacon equipment, and positioning can be achieved relying on visual information within the station, significantly reducing deployment and maintenance costs and having broad promotion potential. It can provide accurate navigation support for passengers, while providing passenger flow data for stations, optimizing operational efficiency.

[0068] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of the invention. Attached Figure Description

[0069] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0070] Figure 1 This is a flowchart illustrating a passive passenger localization method for railway passenger stations based on multimodal image fusion, provided as an embodiment of the present invention.

[0071] Figure 2 This is an overall implementation block diagram of the passive passenger localization method for railway passenger stations based on multimodal image fusion and parallelized 3D reconstruction according to the present invention.

[0072] Figure 3 This is a graph feature coding network structure diagram provided by the present invention. Detailed Implementation

[0073] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0074] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or couplings. The term “and / or” as used herein includes any and all combinations of one or more of the associated listed items.

[0075] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0076] To facilitate understanding of the embodiments of the present invention, the following will provide further explanation and description with reference to the accompanying drawings and several specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0077] The passive positioning system for passengers in railway passenger stations provided in this embodiment of the invention consists of a multimodal image data acquisition module, an image hybrid feature extraction and adaptive bag-of-words tree construction module, a parallelized 3D reconstruction module, a multi-hash table joint retrieval module, and a dynamic error compensation positioning module. The connectivity relationships between the modules are as follows:

[0078] The multimodal image data acquisition module is deployed in key areas of the station (such as ticket gates and waiting halls). It synchronously acquires RGB images, depth maps and inertial data through RGB (red (R), green (G), blue (B))-D cameras and IMU (Inertial Measurement Unit) sensors. After spatiotemporal alignment, the data is output to the hybrid feature extraction module.

[0079] The hybrid feature extraction module extracts local and semantic features from the input image, constructs an adaptive bag-of-words tree, and outputs the bag-of-words tree to the parallelized 3D reconstruction module.

[0080] The parallelized 3D reconstruction module divides the bag-of-words tree into subsets and uses GPU (Graphics Processing Unit) to accelerate the generation of a global 3D point cloud model, which serves as the physical space mapping benchmark and is input into the multi-hash table joint retrieval module.

[0081] The multi-hash table joint retrieval module receives real-time images captured by passengers, filters candidate images based on physical space mapping benchmarks through hash encoding and lightweight matching, and outputs the matching pose to the dynamic error compensation module.

[0082] The dynamic error compensation positioning module uses Kalman filtering to dynamically smooth the pose, ultimately outputting passenger coordinates with sub-meter accuracy.

[0083] The flowchart of a passive passenger localization method for railway passenger stations based on multimodal image fusion provided in this embodiment of the invention is as follows: Figure 1 As shown, the processing steps include the following;

[0084] Step S1: Collect and construct a multimodal image dataset within the functional area of ​​a high-speed railway station using an RGB-D camera and inertial sensor. Enhance data diversity through random occlusion and affine transformation. Simultaneously record the shooting parameters and latitude and longitude coordinates of each image in the station coordinate system to provide high-quality input for subsequent feature extraction and 3D reconstruction.

[0085] The method of this invention deploys multiple hardware components within the functional areas of a railway passenger station (entrance, waiting hall, etc.), including multiple sets of RGB-D cameras (such as Intel RealSense D455), spaced 5-10 meters apart, covering a 360° field of view; an IMU (such as BMI160) is integrated into the cameras, with a sampling frequency of 100Hz. Each RGB camera and IMU simultaneously captures data to construct a multimodal image dataset.

[0086] Hardware triggering signals ensured the timestamp alignment of the RGB image, depth map, and IMU data. The spatial coordinate system had the station center as its origin, with the z-axis pointing vertically upwards and the x and y axes parallel to the station ground. Data augmentation was performed on the image data in the multimodal image dataset by randomly generating rectangular occlusion regions in the images, with occlusion areas ranging from 10% to 30%. Gaussian noise was used to simulate dynamic interference in the texture of the occlusion objects. Within the random occlusion regions (rectangular areas), the noise value of each pixel followed a Gaussian distribution. The mean μ = 0, and the standard deviation is dynamically adjusted according to the station's lighting conditions. The pixel values ​​I(x,y) of the image within the occluded area are perturbed:

[0087]

[0088] Simultaneously, a random affine transformation matrix A is applied to each image to obtain the data-enhanced image, defined as:

[0089]

[0090] Where the scaling factor s∈[0.8,1,2], the rotation angle θ∈[-30°,30°], and the translation amount t x ,t y ∈[-50,50] pixels.

[0091] Step S2: Extract local and semantic features from the data-enhanced image and construct an adaptive bag-of-words tree.

[0092] A directional fast feature keypoint detection algorithm and rotation-invariant descriptors are used to extract local features from images, generating two types of binary feature vectors. A self-supervised model is then used to extract semantic features from these vectors, and these features are concatenated to obtain a hybrid feature. A spatial clustering algorithm is used to classify and cluster the hybrid features, adaptively removing unstable branches to reduce redundant computation and improve the adaptability of the bag-of-words tree to complex scenes.

[0093] For each data-augmented image, detect FAST (Five-hundred-meter Aperture Spherical radio Telescope) corner points, calculate the BRIEF (Binary Robust Independent Elementary Features) descriptor, and combine it with the rotation-invariant descriptor to generate a 256-dimensional binary feature vector. The formula for compensating the orientation angle θ of the above rotation-invariant descriptor is as follows:

[0094]

[0095] In the formula, since the image used in this invention is a color image, I(x,y) is the image pixel intensity value, and x,y represent the two-dimensional coordinate index of the pixel in the image or neighborhood. The direction component is weighted and summed using the intensity values ​​of all pixels in the neighborhood, and then the principal direction angle θ of the neighborhood is obtained through the binary arctangent function.

[0096] Subsequently, a pre-trained SuperPoint semantic model is used to extract 128-dimensional floating-point vectors from the input original RGB image. The semantic model's decoder contains four convolutional blocks (each block includes a convolutional layer, ReLU (Rectified Linear Unit), and pooling). The keypoint detection head in the decoder outputs 65 channels, and a keypoint probability map is generated using Softmax. The descriptor generation head outputs a 128-dimensional floating-point descriptor after Min-Max normalization. Through these steps, the feature map resolution is ultimately downsampled to 1 / 8 of the original image. For each image, a 128-dimensional semantic feature vector is extracted. The 256-dimensional binary feature vectors of the N images are concatenated with the SuperPoint features to form a 384-dimensional hybrid feature vector.

[0097]

[0098] In the adaptive bag-of-words tree construction stage, the HDBSCAN algorithm is used to process the above-mentioned mixed feature vectors. Hierarchical clustering is performed by first constructing a core distance based on k-nearest neighbors to generate a weighted adjacency graph. Then, a minimum spanning tree is generated using the Prim algorithm, and pruning is performed based on reach distance. The clustering levels in the tree are extracted, and the cluster stability λ is calculated (determined through cross-validation). The ratio of the number of samples within a cluster to the cluster diameter is calculated, and clusters with λ < 0.7 are removed, merging them to form clusters with high stability to balance clustering purity and coverage. Finally, a hierarchical bag-of-words tree is obtained, where each leaf node stores the cluster center feature vector and the spatial coordinates of the corresponding image.

[0099] Step S3: The bag-of-words tree is divided into subsets using a GPU (Graphics Processing Unit) to accelerate the calculation of the global rotation matrix. Combined with the keyframe dynamic filtering mechanism, keyframes with high matching quality and wide coverage are selected. The projection error is optimized by the sparse light speed adjustment method to generate a global 3D point cloud model. The global 3D point cloud model is used as the physical space mapping benchmark and input into the multi-hash table joint retrieval module.

[0100] The hierarchical cluster structure of the bag-of-words tree and the hybrid feature vectors The spatial coordinates of the corresponding image, as well as the camera intrinsic matrix K and the initial estimate of the shooting pose T. i =[R i |t i The input is fed into the GPU. The GPU then determines the input based on spatial proximity (Euclidean distance threshold d). max =10m) and feature similarity (cosine similarity greater than 0.8) divide the bag-of-words tree into C subsets {a1, a2, ..., a c Each subset contains local features and semantic features.

[0101] In the global rotation matrix optimization stage, the relative rotation is first estimated from the local camera pairs. The GPU then partitions each bag-of-words tree into subset pairs (a i ,a j Extract matching feature points {(p i ,p j The essential matrix E is calculated using the five-point method. ij The relative rotation matrix is ​​obtained by decomposition. The objective function is to minimize the difference in rotation matrices between all subset pairs to ensure that the rotation matrices of multiple camera views remain consistent in the global coordinate system. The optimized objective function is as follows:

[0102]

[0103] Among them, R i ,R j Let be the rotation matrices for cameras i and j, respectively. Let ||·|| be the estimated value of the rotation matrix between the two cameras. F This represents the Frobenius norm. In practice, subset pairs are assigned to GPU thread blocks, with each thread block computing one. This involves parallelizing the computation. The final optimized global rotation matrix R is obtained. i And save it as an NPY format file.

[0104] In the keyframe dynamic selection and initialization phase, the number of matching image points is set to no less than 50 to ensure sufficient geometric constraints. Scene coverage is no less than 60%, covering the main structure of the current subset. The frame with the highest feature distribution entropy is selected to avoid redundancy. Then, the optimized global rotation matrix R obtained in the previous step is... i ′ and initial translation t i Combined, the keyframe pose T is obtained. i ′=[R′ i |t i ].

[0105] After completing the above process, the keyframe pose T i ′ and through multi-view geometric triangulation from the matched feature point pairs {(p i ,p j The sparse three-dimensional point X calculated in )} l Camera intrinsic parameters K and feature point matching relationship O = {(k, l, x)} kl x is taken as input. kl Represents a three-dimensional point X l At keyframe T k The pixel coordinates observed in ' are used. An objective function is constructed to minimize the sum of reprojection errors, while simultaneously optimizing the poses and 3D points across all keyframes:

[0106]

[0107] Where π([x,y,z]) T = [x / z, y / z] T For perspective projection function, π(KT) k ′·X l )-x kl Defined as the projection error e between each observation (k,l) kl ρ is the Huber robust kernel function, defined as:

[0108]

[0109] Where δ is a hyperparameter, and its optimal value was determined to be 2 through cross-validation.

[0110] The three-dimensional point set {X l} and keyframe {T k The data is divided into independent blocks and allocated to different thread blocks on the GPU. For example, each thread block handles the Jacobian calculation for 128 3D points, and each thread block handles the pose update for 64 keyframes. Specifically, the initial pose {T} is loaded. k ′} and the three-dimensional point set {X l}, pre-calculate the projection error e for all observations. kl All error terms e are computed in parallel. kl and its Jacobian matrix The H and g values ​​from each thread block are accumulated into the global matrix, where H and g are defined as the second-order approximate Hessian matrix and gradient vector of the objective function. The CUDA library is used to solve the linear equation HΔξ = -g, where Δξ represents a 6-dimensional vector of camera pose increments, with the first 3 dimensions representing rotation increments and the last 3 dimensions representing translation increments. If the relative error change is less than 1 × 10⁻⁶... -6 Then the process terminates. Subsequently, the 3D point coordinates and keyframe poses are updated to the final optimized values, and the optimized and updated pose T is output. k (CSV file) and the optimized and updated sparse 3D point cloud X l (PLY format), containing vertex coordinates, color, and normal vectors. A 3D point cloud is reconstructed from a spatial 3D point set using triangulation positioning methods, generating a global 3D point cloud model covering the functional areas of a railway passenger station, thus achieving 3D scene reconstruction.

[0111] Step S4: Use multiple sets of locality-sensitive hash functions to generate hash codes, filter high-similarity candidate images through a voting mechanism, and calculate the preliminary localization result based on the weighted result of the candidate image set pose.

[0112] First, image features are extracted from the multimodal image dataset using the DenseNet-121 architecture. The DenseNet-121 architecture consists of three dense blocks with three convolutional layers, connected by transition layers consisting of 1×1 convolutions and 2×2 average pooling layers. The multimodal image dataset acquired in step S1 is used as input, and image features are encoded using DenseNet-121. Each image is ultimately processed by global average pooling to generate a 1024-dimensional feature vector. This 1024-dimensional feature vector extracted based on the DenseNet-121 model is then input as input. And an image dataset D = {(F i ,T i )}.

[0113] Using 4 sets of Locality Sensitive Hash (LSH) functions Each group generates a 12-bit hash code:

[0114] h i (x)=sign(w i x+b i (7)

[0115] in, w is a 384-dimensional blended feature vector of the input image. i b is a random projection vector that follows a standard normal distribution, with dimensions 384×1. i The uniformly distributed random bias term is used to adjust the separating threshold of the hash function. `sign(·)` is the sign function, which maps real values ​​to binary hash codes of ±1.

[0116] Using the same method as image feature extraction in multimodal image datasets, the images I captured in real time by passengers were processed. q The feature vector F is extracted using DenseNet-121. q Principal component analysis was used to reduce the dimensionality of the eigenvectors to 384 dimensions, and F was calculated. q The 4 sets of hash codes {h i (F q Then, the hash table is counted in relation to F. q Reference images for hash code matching, sorted by the number of matching hash tables:

[0117]

[0118] in, The function is an indicator; it is 1 if the hash codes are equal, and 0 otherwise. The set of candidate images with scores S≥3 is retained as C={(F j ,T j Further precise matching is performed, calculating the match between the candidate image set C and F.q The cosine similarity of the feature vectors is used to select the top-5 images with the highest similarity as the final matching results. The pose of the top-5 matched images {T} is then calculated. j The spatial positions of the passenger's current viewpoint and the viewpoint image in the global 3D point cloud model are uniformly aligned. The cosine similarity results between the viewpoints of multiple known high-similarity candidate reference images and the passenger's viewpoint image pose are used as weights. Weights are generated using the Softmax function based on the cosine similarity results and then weighted and averaged to obtain the weighted pose T. f :

[0119]

[0120] The above weighted pose is used as a preliminary pose estimate T from the passenger's perspective. f =[R f |t f ], where R f The rotation matrix of the matched image is converted into a quaternion q. j The result of weighted averaging and conversion back to a rotation matrix is This is the weighted result of the translation vector.

[0121] To improve the accuracy and robustness of localization, a global 3D point cloud model is introduced as a mapping benchmark for physical space, and the pose is optimized by utilizing the correspondence between the 3D point cloud and image feature points. For Top-5 matching images, the correspondence between their 2D feature points and 3D points is obtained. For passenger images, the SIFT algorithm is used to extract their feature points. By matching with feature points in the candidate image, the corresponding 3D point p is indirectly found. l The correspondence between feature points in the passenger's perspective image and three-dimensional points is obtained. And optimize the pose by minimizing the reprojection error.

[0122]

[0123] In the formula, π is the Huber kernel function. The Levenberg-Marquardt algorithm is used to solve the nonlinear least squares problem to obtain the optimized pose T. refined =[R refined |t refined ].

[0124] Step S5: Based on the camera intrinsic parameter matrix and rotation matrix from the passenger's perspective, the initial pose estimation from the passenger's perspective is smoothed by Kalman filtering based on the shooting pose obtained from the previous weighted calculation, and the passenger coordinates are output.

[0125] The pose estimate T representing the passenger's initial viewpoint obtained from the preceding steps refined For input. Position pinit =[x init ,y init ,z init ] T and the initial rotation quaternion q init =[q w ,q x ,q y ,q z ] T As the passenger's initial position and attitude state, combined with the velocity component vectors in three directions v = [v x ,v y ,v z ] T The state vector x = [p, v, q] is formed. T Assuming the passenger moves at a constant speed or remains stationary for a short period of time, the state update at the next moment a+1 is:

[0126] p a+1 =p a +v a Δt (11)

[0127] v a+1 =v a (12)

[0128]

[0129] Where ω is the angular velocity. The weighted pose from step S4 is then used as the observation value z. k The predicted and observed values ​​obtained by Kalman gain fusion using the state update formula are:

[0130] K a+1 =P a+1|a +H T HP a+1|a H T +R a+1 ) -1 (14)

[0131] x a+1 =x a+1|a +K a+1 (z k -Hx a+1|a (15)

[0132] P a+1 =(IK a+1 H)P a+1|a (16)

[0133] Where z k -Hx a+1 K represents the difference between observation and prediction. fR is used to balance the confidence levels between predicted and observed values. a+1 To observe noise, if it is small, the gain K f As the value increases, the system trusts the observations more and quickly corrects prediction biases. After multiple iterations, P... a+1 The system gradually converges, the state estimation stabilizes, and the final output state update includes the passenger's final position p. a+1 and the state vector x of the attitude a+1 .

[0134] In summary, the passive positioning method of this invention reduces costs: it eliminates the need to deploy external beacon equipment and achieves positioning based on visual information within the site, significantly reducing deployment and maintenance costs and possessing broad potential for widespread application.

[0135] High-efficiency 3D reconstruction: By using a parallelized hierarchical motion recovery structure algorithm and GPU acceleration, the efficiency bottleneck of traditional reconstruction methods in large-scale scenes is overcome, the reconstruction time is shortened, and it can adapt to the complex environment of high-speed railway stations.

[0136] High precision and robustness: Multimodal image fusion enhances feature representation capabilities, and multi-hash table retrieval combined with Kalman filtering controls positioning errors to the sub-meter level, improving matching accuracy;

[0137] High real-time performance: The single location time of lightweight retrieval is reduced compared to traditional image retrieval algorithms, meeting the needs of real-time navigation;

[0138] Dual improvement in service and management: providing passengers with accurate navigation support, while providing stations with passenger flow data to optimize operational efficiency.

[0139] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0140] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.

[0141] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for apparatus or system embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The apparatus and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0142] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A passive passenger localization method for railway passenger stations based on multimodal image fusion, characterized in that, include: A multimodal image dataset was acquired and constructed using RGB-D cameras and inertial sensor devices within the functional area of ​​a high-speed railway station. Data augmentation processing was performed on the images in the multimodal image dataset using random occlusion and affine transformation. Local and semantic features are extracted from the data-enhanced image to construct an adaptive bag-of-words tree; The bag-of-words tree is divided into subsets using a graphics processing unit (GPU), key frames are selected using a key frame dynamic filtering mechanism, projection errors are optimized using the sparse beam adjustment method, and a global 3D point cloud model is generated. Multiple sets of locality-sensitive hash functions are used to generate hash codes. The global three-dimensional point cloud model is used as the physical space mapping benchmark. A set number of candidate images with cosine similarity exceeding a set threshold are selected through a voting mechanism to obtain a preliminary pose estimate from the passenger's perspective. Based on the camera intrinsic parameter matrix and rotation matrix from the passenger's perspective, the preliminary pose estimation from the passenger's perspective is smoothed by Kalman filtering, and the passenger's positioning coordinates are output. The process of smoothing the preliminary pose estimation from the passenger's perspective using Kalman filtering based on the camera intrinsic parameter matrix and rotation matrix, and outputting the passenger's location coordinates, includes: Position p init =[x init ,y init ,z init ] T and the initial rotation quaternion q init =[q w ,q x ,q y ,q z ] T As the passenger's initial position and attitude state, combined with the velocity component vectors in three directions v = [v x ,v y ,v z ] T The state vector x = [p, v, q] is formed. T Assuming the passenger moves at a constant speed or remains stationary for a short period of time, the state at the next moment a+1 will be updated as follows: p a+1 =p a +v a Δt v a+1 =v a Where ω is the angular velocity, and the weighted pose T f As observed value z k The predicted and observed values ​​are obtained by Kalman filtering gain fusion and then using the state update formula: K a+1 =P a+1|a +H T (HP a+1|a H T +R a+1 ) -1 x a+1 =x a+1|a +K a+1 (z k -Hx a+1|a ) P a+1 =(I-K a+1 H)P a+1|a Where z k -Hx a+1 K represents the difference between observation and prediction. f R is used to balance the confidence levels between predicted and observed values. a+1 To observe the noise, after multiple iterations, P a+1 The system gradually converges, the state estimation stabilizes, and the final output state update includes the passenger's final position p. a+1 and the state vector x of the attitude a+1 .

2. The method according to claim 1, characterized in that, The method described above involves acquiring and constructing a multimodal image dataset within the functional area of ​​a high-speed railway station using an RGB-D camera and inertial sensor equipment, and performing data augmentation processing on the images in the multimodal image dataset through random occlusion and affine transformation, including: RGB cameras and inertial measurement units (IMUs) are deployed within the functional area of ​​railway passenger stations. Hardware trigger signals are used to ensure that the timestamps of RGB images, depth maps and IMU data are aligned. The spatial coordinate system is based on the station center as the origin, with the z-axis pointing vertically upward and the x and y axes parallel to the station ground. Each RGB camera and IMU captures data synchronously to construct a multimodal image dataset. Data augmentation is performed on the image data in the multimodal image dataset by randomly generating rectangular occlusion regions in the images. Gaussian noise is used to simulate dynamic interference in the texture of the occlusion objects. Within the random occlusion regions, the noise value of each pixel follows a Gaussian distribution. Where the mean μ = 0, the pixel values ​​I(x,y) of the image within the occluded area are perturbed: Applying a random affine transformation matrix A to each image yields the data-enhanced image. The random affine transformation matrix A is defined as follows: Where the scaling factor s∈[0.8,1,2], the rotation angle θ∈[-30°,30°], and the translation amount t x ,t y ∈[-50,50] pixels.

3. The method according to claim 2, characterized in that, The process of extracting local and semantic features from the data-enhanced image and constructing an adaptive bag-of-words tree includes: For each data-enhanced image, detect the corner points of the FAST spherical radio telescope, calculate the BRIEF descriptor, and generate a 256-dimensional binary feature vector using the rotation-invariant descriptor. The formula for compensating the direction angle θ of the rotation-invariant descriptor is as follows: I(x,y) is the image pixel intensity value, where x and y represent the two-dimensional coordinate index of the pixel in the image or neighborhood; A semantic model was used to extract a 128-dimensional semantic feature vector from the original RGB image. The 256-dimensional binary feature vectors of the N images are combined with the 128-dimensional semantic feature vectors. Concatenate into a 384-dimensional hybrid feature vector The HDBSCAN algorithm is used to process the hybrid feature vector. Hierarchical clustering is performed, and a core distance is constructed based on k-nearest neighbors to generate a weighted adjacency graph. Then, the minimum spanning tree is generated using the Prim algorithm, and pruning is performed based on the reach distance. The clustering levels in the tree are extracted, the cluster stability λ is calculated, and the ratio of the number of samples in the cluster to the cluster diameter is calculated. Clusters with λ < 0.7 are removed, and clusters with high stability are merged to obtain a hierarchical bag-of-words tree. Each leaf node in the bag-of-words tree stores the cluster center feature vector and the spatial coordinates of the corresponding image.

4. The method according to claim 3, characterized in that, The process of using a graphics processing unit (GPU) to divide the bag-of-words tree into subsets, selecting keyframes using a dynamic keyframe filtering mechanism, optimizing projection errors through sparse bundle adjustment, and generating a global 3D point cloud model includes: The hierarchical cluster structure of the bag-of-words tree and the hybrid feature vectors The spatial coordinates of the corresponding image, as well as the camera intrinsic matrix K and the initial estimate of the shooting pose T. i =[R i |t i The input is fed into the GPU, which divides the bag-of-words tree into C subsets {a1, a2, ..., a...} based on spatial proximity and feature similarity. c Each subset contains local features and semantic features; The GPU processes each subset pair (a i ,a j Extract matching feature points {(p i ,p j The essential matrix E is calculated using the five-point method. ij The essential matrix E ij Decomposition yields the relative rotation matrix The objective function is to minimize the difference in rotation matrices between all subset pairs. The optimized objective function is: Among them, R i ,R j Let be the rotation matrices for cameras i and j, respectively. Let ||·|| be the estimated value of the rotation matrix between the two cameras. F Representing the Frobenius norm, we obtain the optimized global rotation matrix R. i ; The optimized global rotation matrix R′ i With initial translation t i Combined, the keyframe pose T is obtained. i ′=[R′ i |t i Based on keyframe pose T i ′ and matching feature point pairs {(p i ,p j The sparse three-dimensional point X is calculated. l Camera intrinsic parameters K and feature point matching relationship O = {(k, l, x)} kl )}, where x kl Represents a three-dimensional point X l At keyframe T k The pixel coordinate observations in ' are used to construct the objective function by minimizing the sum of reprojection errors, while simultaneously optimizing the poses and 3D points of all keyframes: Where π([x,y,z]) T = [x / z, y / z] T For perspective projection function, π(KT) k ′·X l )-x kl Defined as the projection error e between each observation (k,l) kl ρ is the Huber robust kernel function, defined as: Where δ is a hyperparameter; The three-dimensional point set {X l } and keyframe {T k The data is divided into independent blocks and allocated to different thread blocks of the GPU. Then, the 3D point coordinates and keyframe poses are updated to the final optimized value, and the optimized and updated pose T is output. k ′ and optimized updated sparse 3D point cloud X l It includes vertex coordinates, color, and normal vectors. Using triangulation positioning method, it performs 3D point cloud reconstruction on the spatial 3D point set, generates a global 3D point cloud model covering the functional area of ​​railway passenger station, and realizes 3D scene reconstruction.

5. The method according to claim 4, characterized in that, The process involves generating hash codes using multiple sets of locality-sensitive hash functions, using the global 3D point cloud model as a physical space mapping benchmark, filtering a set number of candidate images whose cosine similarity exceeds a set threshold through a voting mechanism, and obtaining a preliminary pose estimate from the passenger's perspective, including: Use 4 sets of locality-sensitive hash functions Each group generates a 12-bit hash code: h i (x)=sign(w i x+b i ) in, w is a 384-dimensional blended feature vector of the input image. i b is a random projection vector that follows a standard normal distribution, with dimensions 384×1. i The uniformly distributed random bias term is used to adjust the separation threshold of the hash function, and sign(·) is the sign function, which maps real values ​​to binary hash codes of ±1. Images captured in real time by passengers q Feature F is extracted using the DenseNet-121 architecture. q Principal component analysis was used to reduce the dimensionality of the eigenvectors to 384 dimensions, and F was calculated. q The 4 sets of hash codes {h i (F q )}, Statistical hash table and F q Reference images for hash code matching, sorted by the number of matching hash tables: in, The function is an indicator function; it is 1 if the hash codes are equal, and 0 otherwise. It retains the set C = {(F...} of candidate images with scores S ≥ 3. j ,T j For the candidate image set C, calculate its relationship with F. q The cosine similarity of feature vectors is used to select a set number of images with a similarity exceeding a set threshold as the final matching result. The pose {T} of the final matched images is then considered. j The spatial positions of the passenger's current viewpoint and the viewpoint image in the global 3D point cloud model are uniformly aligned. The cosine similarity results between the viewpoints of multiple known high-similarity candidate reference images and the passenger's viewpoint image pose are used as weights. Weights are generated using the Softmax function based on the cosine similarity results and then weighted and averaged to obtain the weighted pose T. f : The weighted pose is used as a preliminary pose estimate T from the passenger's perspective. f =[R f |t f ], where R f The rotation matrix of the matched image is converted into a quaternion q. j The result of weighted averaging and conversion back to a rotation matrix is This is the weighted result of the translation vector.

Images