A shielding space positioning method, system and terminal based on multi-source data fusion

Abstract

This invention belongs to the field of occluded spatial positioning technology, and relates to an occluded spatial positioning method based on multi-source data fusion. This method simultaneously collects multi-source data from occluded spaces and preprocesses each type of multi-source data separately. The preprocessed multi-source data is then fused to obtain standard fused data. A trained deep learning model is used to extract features and identify scenes from the standard fused data. The weights of each type of data are dynamically adjusted based on the scene identification results. Preliminary solutions are then calculated for each type of data to obtain preliminary positioning. Finally, the preliminary positioning and weights of each type of data are fused to obtain the final positioning result. This invention also provides an occluded spatial positioning system based on multi-source data fusion and a terminal. This invention solves the technical problems of low positioning accuracy, poor adaptability, and difficulty in terminal adaptation in occluded scenarios by fusing multi-source data, deep learning optimization, and dynamic weight adjustment.

Description

Technical Field

[0001] This invention relates to the field of occult space positioning technology, and in particular to an occult space positioning method, system and terminal based on multi-source data fusion. Background Technology

[0002] In the field of positioning, while the Global Navigation Satellite System (GNSS) is the core means of outdoor positioning, its signal is easily affected by obstruction and multipath reflection in obstructed scenarios such as urban canyons, under elevated roads, and port operation areas, leading to a significant decrease in positioning accuracy or even failure. To compensate for this deficiency, existing technologies attempt to fuse data from multiple sensors; however, they face many challenges: sampling timing deviations between different sensors can easily lead to data mismatch; the redundancy and complementarity of multimodal features are difficult to balance; and there is a lack of adaptive strategies for specific obstructed scenarios, making it difficult to stably leverage the advantages of each sensor.

[0003] In recent years, deep learning technology has been introduced into the field of localization for feature extraction, scene recognition, and localization optimization. However, existing models are mostly designed for specific scenarios, with limited generalization ability, making them difficult to adapt to diverse occlusion scenarios such as urban canyons and tree-lined roads. Furthermore, the high complexity and computational demands of these models make them difficult to deploy on outdoor terminals with limited computing power. In addition, the differences in the characteristics of different hardware devices lead to difficulties in model adaptation and poor cross-device compatibility, while retraining requires a large amount of labeled data, resulting in high costs. Regarding the critical issue of multipath error, traditional methods, based on empirical models or filtering algorithms, struggle to cope with the complex and ever-changing multipath signals in dynamically occluded scenarios. Moreover, existing solutions often result in excessively high energy consumption due to high sampling rates and complexity, failing to meet the long-term continuous operation requirements of outdoor terminals. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing a method, system, and terminal for occluded spatial positioning based on multi-source data fusion. By fusing multi-source data, deep learning optimization, and dynamic weight adjustment, this invention solves the technical problems of low positioning accuracy, poor adaptability, and difficulty in terminal adaptation in occluded scenarios.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a method for occlusion spatial localization based on multi-source data fusion, comprising the following steps: S1. Synchronously collect multi-source data of the obscured space, including: GNSS signal data, environmental image data, INS inertial measurement data, and lidar point cloud data; S2. Preprocess each type of data, merge the preprocessed multi-source data, and obtain standard fused data; As one possible implementation, preprocessing is performed on each type of data, including: Invalid values ​​in the GNSS signal data are removed, and the carrier phase smoothing pseudorange algorithm is used to optimize the GNSS signal data after removing invalid values; zero bias compensation is performed on the INS inertial measurement data based on extended Kalman filtering; voxel filtering downsampling is performed on the lidar point cloud data, and the point cloud density and plane fitting error are calculated; visual feature entropy and edge density features of the environmental image data are extracted.

[0006] S3. Use a trained deep learning model to extract features and identify scenes from the standard fusion data, and dynamically adjust the weights of each data category based on the scene identification results; As one possible implementation, the deep learning model includes: a feature extraction network and a scene recognition unit; the trained deep learning model is used to perform feature extraction and scene recognition on standard fused data, including the following steps: S30. The feature extraction network extracts local features at each time step of the standard fusion data, obtains the long-term temporal dependencies of the standard fusion data, and obtains the global feature vector through global average pooling; S31. Perform nonlinear mapping on the global feature vector to obtain feature discrimination results, and identify scene categories based on the feature discrimination results. The scene categories include: urban canyons, open areas, tunnels, construction sites, and port work areas. S32. Based on the activation function included in the scene recognition unit, the scene recognition result is converted into a probability distribution of scene classes, and the probability values ​​of the current standard fusion data belonging to each scene class are obtained. The scene corresponding to the maximum probability value is the scene recognition result.

[0007] As one possible implementation, the weights of each data category are dynamically adjusted based on the scene recognition results, including: Configure a scene probability threshold. When the probability of a tunnel scene is greater than the scene probability threshold, adjust the data weights to: INS inertial measurement data weight > LiDAR point cloud data weight > GNSS signal data weight. When the probability of an open field scene is greater than the scene probability threshold, the data weights are adjusted to: GNSS signal data weight > INS inertial measurement data weight > lidar point cloud data weight. When the probability of a construction site scenario is greater than the scenario probability threshold, the data weights are adjusted to: LiDAR point cloud data weight > GNSS signal data weight > INS inertial measurement data weight. When the probability of an urban canyon scene is greater than the scene probability threshold, the data weights are adjusted as follows: LiDAR point cloud data weight = INS inertial measurement data weight > GNSS signal data weight. When the probability of a port work area scenario is greater than the scenario probability threshold, the data weights are adjusted to: LiDAR point cloud data weight > INS inertial measurement data weight > GNSS signal data weight.

[0008] S4. Perform preliminary solution calculations for each type of multi-source data to obtain the preliminary solution location for each type of data; As one possible implementation, preliminary solution calculations are performed on the multi-source data, including: The PPP-RTK model was used to solve the preprocessed GNSS signal data. A positioning equation was constructed based on carrier phase double-difference observations. Atmospheric correction parameters were introduced during the solution process to compensate for ionospheric and tropospheric delays, resulting in an initial positioning solution for the GNSS signal data. Dead reckoning was used to solve the preprocessed INS inertial measurement data. The solution results of the INS inertial measurement data were corrected by combining the weights of the lidar point cloud data, resulting in an initial positioning solution for the INS inertial measurement data. The spatial transformation relationship between the point cloud in the preprocessed lidar point cloud data and the corresponding features in the preset map was calculated to obtain the initial positioning solution for the lidar point cloud data.

[0009] S5. The initial location and weight of each type of data are fused and calculated to obtain the location result.

[0010] As one possible implementation, S5 includes the following sub-steps: S50. The state prediction equation is constructed using the EKF master filter method, and the current position is predicted based on the EKF master filter state and INS inertial measurement data of the previous moment. S51. Calculate the GNSS deviation between the initial solution of GNSS signal data and the predicted current position, and the point cloud deviation between the initial solution of lidar point cloud data and the predicted current position. S52. The observation residual is obtained by weighted summation of GNSS bias, GNSS signal data weight, point cloud bias, and lidar point cloud data weight; S53. Calculate the Kalman gain based on the covariance of GNSS bias, the covariance of point cloud bias, the observation matrix, and the covariance of observation noise; S54. The positioning result is calculated based on the predicted current position, observation residual, and Kalman gain.

[0011] As one possible approach, when the scene recognition result is a construction site, the following method can be used to optimize the localization result: S55. Initialize N particles, the position of each particle follows a Gaussian distribution with the mean of the positioning result obtained in step S54, and the particle weights are initialized to equal values; S56. Construct a likelihood function based on point cloud density and visual feature entropy, and calculate the matching degree between each particle and the current environment features; S57. Assign weights to each particle based on its matching degree with the current environment features and normalize the weights. Then, resample the normalized weights using the roulette wheel algorithm. S58. Calculate the weighted average value of the particles after resampling, which is the optimized localization result.

[0012] Secondly, the present invention provides a shading spatial positioning system based on multi-source data fusion, comprising: The multi-source data acquisition module synchronously acquires multi-source data of the obscured space, including at least: GNSS signal data, environmental image data, INS inertial measurement data, and lidar point cloud data; The preprocessing module preprocesses each type of data in the multi-source data separately, and merges the preprocessed multi-source data to obtain standard fused data; The deep learning module performs feature extraction and scene recognition on standard fusion data, and dynamically adjusts the weight of each type of data based on the scene recognition results; The initial solution localization calculation module performs initial solution calculations on multi-source data to obtain the initial solution localization for each type of data. The system also includes a positioning result acquisition module, which integrates the initial positioning and weights of each type of data to obtain the positioning result.

[0013] As one possible implementation, the occlusion space positioning system also includes: The localization result optimization module initializes N particles that follow a normal distribution, constructs a likelihood function based on point cloud density and visual feature entropy, calculates the matching degree between each particle and the current environment features, assigns weights to each particle based on the matching degree of the current environment features and performs weight normalization, resamples the normalized weights using the roulette wheel algorithm, and calculates the weighted average of the particles after resampling, which is the optimized localization result. The lightweight deployment module performs lightweight processing on the occluded spatial positioning system and optimizes the performance of the occluded spatial positioning system using the knowledge distillation method.

[0014] Thirdly, the present invention provides a terminal including a processor and a communication interface coupled to the processor, the processor being used to run computer programs or instructions to implement the occlusion spatial positioning method based on multi-source data fusion provided in the first aspect.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. The occluded space localization method and system proposed in this invention, based on multi-source data fusion, synchronously collects multi-source data of the occluded space, performs preprocessing, feature extraction and scene recognition on the multi-source data, and dynamically adjusts the weight of various types of data based on the scene recognition results, which can give full play to the advantages of different types of data during localization and improve localization accuracy.

[0016] 2. The occlusion spatial localization method and system proposed in this invention based on multi-source data fusion trains a deep learning model using five common occlusion scenarios. For new scenarios not included in the training set, a transfer learning mechanism is used for rapid adaptation. Training with a small number of new scenario samples enables the deep learning model to maintain stable recognition performance in new scenarios without the need for a large amount of labeled data, ensuring that the deep learning model can play an effective role in diverse occlusion environments.

[0017] 3. This invention proposes a occlusion spatial positioning method and system based on multi-source data fusion, which does not require a large amount of computation, adopts a lightweight deployment scheme, is compatible with a variety of terminal devices, and can run efficiently on resource-constrained platforms such as in-vehicle embedded devices. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings: Figure 1 A flowchart of the occlusion spatial localization method based on multi-source data fusion provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the deep learning model structure in an embodiment of the present invention. Detailed Implementation

[0019] To facilitate a clear description of the technical solutions in the embodiments of the present invention, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. For example, the first threshold and the second threshold are merely used to distinguish different thresholds and do not limit their order. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" are not necessarily different.

[0020] It should be noted that in this invention, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0021] In this invention, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one" or similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, "at least one of a, b, or c" can represent: a, b, c, a combination of a and b, a combination of a and c, a combination of b and c, or a, b, and c, where a, b, and c can be single or multiple.

[0022] The present invention aims to provide a method, system and terminal for occluded spatial positioning based on multi-source data fusion. By fusing multi-source data, deep learning optimization and dynamic weight adjustment, it solves the technical problems of low positioning accuracy, poor adaptability and difficulty in terminal adaptation in occluded scenarios.

[0023] In a first aspect, embodiments of the present invention provide a method for occlusion spatial localization based on multi-source data fusion, see [link to previous document]. Figure 1 It includes the following steps: S1. Synchronously collect multi-source data of the obscured space, including: GNSS signal data, environmental image data, INS inertial measurement data, and lidar point cloud data; Taking the urban canyon scenario as an example, the dense high-rise buildings in this scenario cause severe GNSS signal obstruction. The building height is mostly between 50 and 150 meters, and the street width is mostly between 10 and 30 meters. The number of visible satellites is often less than 5, resulting in a significant multipath effect. Traditional positioning methods have an error of 5 to 15 meters, which is difficult to meet the positioning requirements.

[0024] Exemplarily, the multi-source data acquisition method of the present invention includes: selecting a GNSS receiver supporting GPS L1 / L5 and BDSB1 / B3 frequency bands, capable of real-time acquisition of pseudorange, carrier phase, and signal strength information. Communication between the GNSS receiver and external devices is completed via a serial port, with a data output frequency of 1Hz. A MEMS device integrating a three-axis accelerometer and a three-axis gyroscope is used as the INS inertial measurement data acquisition device. The accelerometer range is set to ±16g, and the gyroscope range is set to ±2000° / h, capable of acquiring the acceleration and angular velocity data of the carrier. It is connected to a computer terminal via an SPI interface, with a data output frequency of 100 Hz, and also has attitude angle calculation capabilities, capable of outputting pitch, roll, and heading angle information. A 16-line lidar is used as the lidar point cloud data acquisition device, with a scanning frequency of 10 Hz and a ranging range covering 0.5–100 meters. Point cloud data is transmitted via an Ethernet interface, with each frame of the point cloud containing information on the distance and reflection intensity of environmental objects. A monocular camera is used to acquire environmental image data to extract visual features. The monocular camera has a resolution of 1280×720 and a frame rate of 30fps. It is connected to a computer terminal via a USB interface.

[0025] Hardware-level synchronization is achieved using the second pulse signal output by the GNSS receiver, ensuring that the timestamp alignment accuracy of the GNSS receiver, INS inertial measurement data acquisition device, LiDAR point cloud data acquisition device, and monocular camera does not exceed 1ms. The synchronized multi-source data is then transmitted to a computer terminal for processing. For example, the computer terminal is an in-vehicle embedded device equipped with a quad-core processor and 4GB of memory, supporting GPU-accelerated computing to meet the real-time requirements of subsequent deep learning and positioning calculations.

[0026] In practical applications, GNSS signal data is parsed in NMEA format, INS inertial measurement data is converted into physical quantities according to the manufacturer's defined protocol, lidar point cloud data is received in the form of point cloud frames, and environmental image data is acquired in the form of image streams.

[0027] As an example of synchronous acquisition of multi-source data, the GNSS receiver continuously acquires pseudorange, carrier phase, and signal strength information at a frequency of 1Hz. This information is transmitted to the computer terminal via a serial port, strictly maintaining the integrity of the data format during transmission to lay the foundation for accurate analysis in subsequent processing. The INS inertial measurement data acquisition device works in parallel with the GNSS receiver, acquiring triaxial acceleration and angular velocity data at a frequency of 100Hz. This data is transmitted to the computer terminal in real time via the SPI interface. The output data is converted into easily processed physical parameters through a protocol. Simultaneously, based on the built-in attitude calculation algorithm, the device outputs the vehicle's pitch, roll, and yaw angles in real time, ensuring that these data are precisely aligned with the GNSS signal data on the time axis. While the INS inertial measurement data is being transmitted, the lidar point cloud data acquisition device generates one frame of point cloud data every 100ms and transmits it to the computing terminal via Ethernet using the UDP protocol. The point cloud data contains the three-dimensional coordinates and reflection intensity information of each lidar point. During transmission, timestamps are used to maintain time consistency with the GNSS signal data and INS inertial measurement data, ensuring the coordination of multi-source data in the time dimension. The monocular camera captures environmental images at a frame rate of 30fps and transmits the image data to a computer terminal via a USB interface. Each frame includes a precise timestamp, allowing it to be integrated into the main time synchronization system. This achieves synchronous acquisition of multi-source data.

[0028] S2. Preprocess each type of data, merge the preprocessed multi-source data, and obtain standard fused data; As one possible implementation, preprocessing is performed on each type of data, including: Invalid values ​​in the GNSS signal data are removed, and the carrier phase smoothing pseudorange algorithm is used to optimize the GNSS signal data after removing invalid values.

[0029] As an example, when the signal strength C / N0 is less than 25 dB-Hz, the pseudorange and carrier phase data at that moment are deemed invalid and discarded, while valid observations are retained for subsequent processing. A carrier phase smoothing pseudorange algorithm is used to optimize the GNSS signal data after removing invalid values. The smoothing formula is as follows: in, Represents the original pseudorange. Indicates carrier phase, This indicates the carrier wavelength of the corresponding frequency band.

[0030] Optimizing GNSS signal data after removing invalid values ​​using a carrier phase smoothing pseudorange algorithm can reduce the impact of multipath effects on pseudorange measurements, improve data stability, and provide reliable input for subsequent positioning calculations.

[0031] The preprocessing of INS inertial measurement data specifically involves zero-bias compensation based on extended Kalman filtering.

[0032] As an example, the state equation is set as follows: ,in, It is a state vector, which includes position, velocity, attitude, and sensor bias; Represents the state transition matrix. Represents the noise driving matrix. This represents process noise. The observation equation is set as follows: ,in, The observation vector is derived from the position and velocity information of the GNSS signal data. Represents the observation matrix. This indicates observation noise.

[0033] Through iterative calculations using Kalman filtering, the accelerometer and gyroscope biases in the INS inertial measurement data are estimated and compensated in real time, and the corrected velocity and position increment data are output. These data complement the optimized GNSS signal data above, jointly supporting the stability and accuracy of subsequent positioning.

[0034] The preprocessing of lidar point cloud data specifically involves: performing voxel filtering downsampling on the lidar point cloud data, and calculating the point cloud density and plane fitting error.

[0035] As an example, a voxel filtering method is employed to divide the 3D space into a 0.1m × 0.1m × 0.1m voxel grid. Within each voxel, the point closest to the voxel center is retained. This operation reduces the number of point clouds while maintaining the integrity of environmental features, thus improving the efficiency of subsequent processing. Simultaneously, the density features of the point cloud for each frame are calculated, and the number of laser points per unit volume is counted to reflect the openness of the environment. For areas with obvious planar features, such as indoor walls and floors, a planar fitting algorithm is used to calculate the planar fitting error, providing feature support for scene recognition. The preprocessed LiDAR point cloud data is then aggregated with other preprocessed data to form a feature set.

[0036] The preprocessing of environmental image data specifically involves extracting the visual feature entropy and edge density features of the environmental image data.

[0037] As an example, the color image is first converted to a grayscale image. The SIFT algorithm is then used to extract feature points from the image, and the distribution density and orientation information of the feature points are calculated to obtain the feature entropy. The formula for calculating the feature entropy is as follows: in, The probability distribution of different gray values ​​in an image is represented by the feature entropy value, which reflects the complexity of the environment. Image edges are detected using the Canny operator, and the proportion of edge pixels to the total number of pixels in the image is used as the edge density feature. This edge density feature exhibits high values ​​in scenes such as metal walls, and can serve as an important basis for scene recognition.

[0038] S3. Perform feature extraction and scene recognition on the standard fusion data, and dynamically adjust the weight of each type of data based on the scene recognition results; As one possible implementation, deep learning models include: feature extraction networks and scene recognition units; As an example, see Figure 2 The feature extraction network includes convolutional layers, a first activation function, an LSTM layer, and a global average pooling layer; the scene recognition unit adopts an ML-LSTM structure, which includes a first fully connected layer, a second activation function, a second fully connected layer, and a third activation function connected in sequence.

[0039] The trained deep learning model is used to extract features and identify scenes from standard fused data, including the following steps: S30. The feature extraction network extracts local features at each time step of the standard fusion data, obtains the long-term temporal dependencies of the standard fusion data, and obtains the global feature vector through global average pooling; S31. Perform nonlinear mapping on the global feature vector to obtain feature discrimination results, and identify scene categories based on the feature discrimination results. The scene categories include: urban canyons, open areas, tunnels, construction sites, and port work areas. S32. Based on the activation function included in the scene recognition unit, the scene recognition result is converted into a probability distribution of scene classes, and the probability values ​​of the current standard fusion data belonging to each scene class are obtained. The scene corresponding to the maximum probability value is the scene recognition result.

[0040] As an example, the feature extraction network receives standard fused data, organizes the data according to a 7-second time window, and samples at a frequency of 10Hz. Each time window contains 70 time steps, and the feature vector for each time step is 16-dimensional, covering multiple dimensions such as pseudorange residuals of GNSS signal data, signal strength, velocity and attitude angle change rate of INS inertial measurement data, point cloud density and plane fitting error of LiDAR point cloud data, and visual feature entropy and edge density features of environmental image data. Specifically, the first layer of the feature extraction network is a 1D convolutional layer with a kernel size of 3×3, 32 output channels, and a stride of 1. This convolutional layer extracts local features at each time step, capturing the signal change trend within three consecutive time steps. Next, the ReLU activation function is used to enhance the nonlinear expressive power of the feature extraction network, with an output dimension of (70, 32). The network then proceeds to an LSTM layer with 64 hidden units and a dropout rate of 0.2 to prevent overfitting. This layer effectively captures long-term temporal dependencies in multi-source data. For example, in a tunnel environment, the cumulative trend of INS inertial measurement data error over time is observed, and the output dimension of the LSTM layer is (70, 64). The output of the LSTM layer enters the global average pooling layer. After global average pooling, the features of 70 time steps are compressed into a 64-dimensional global feature vector, which is then smoothly transitioned to the input of the scene recognition unit. For instance, the first fully connected layer of the scene recognition unit contains 128 neurons, and the second activation function is ReLU, which performs further nonlinear mapping on the global feature vector to improve the feature discriminative power. The second fully connected layer contains 5 neurons, corresponding to five types of occluded scenes: urban canyons, open areas, tunnels, construction sites, and port work areas. The output of the second fully connected layer is converted into a probability distribution for each scene through the third activation function Softmax. The sum of the probability values ​​is 1, which means that the probability values ​​of the current standard fused data belonging to each scene are obtained. The scene corresponding to the highest probability value is the scene recognition result.

[0041] As a model training example, a dataset containing more than 100,000 samples was used, with no less than 20,000 samples for each scene. The cross-entropy loss function was used during training, the Adam optimizer was selected, and the initial learning rate was set to 0.001. The model converged after 100 rounds of iterative training, and the scene recognition accuracy on the validation set reached more than 92%, providing an accurate basis for scene judgment for subsequent dynamic weight adjustment.

[0042] As one possible implementation, the weights of each data category are dynamically adjusted based on the scene recognition results, including: Configure a scene probability threshold. When the probability of a tunnel scene is greater than the scene probability threshold, adjust the data weights to: INS inertial measurement data weight > LiDAR point cloud data weight > GNSS signal data weight. When the probability of an open field scene is greater than the scene probability threshold, the data weights are adjusted to: GNSS signal data weight > INS inertial measurement data weight > lidar point cloud data weight. When the probability of a construction site scenario is greater than the scenario probability threshold, the data weights are adjusted to: LiDAR point cloud data weight > GNSS signal data weight > INS inertial measurement data weight. When the probability of an urban canyon scene is greater than the scene probability threshold, the data weights are adjusted as follows: LiDAR point cloud data weight = INS inertial measurement data weight > GNSS signal data weight. When the probability of a port work area scenario is greater than the scenario probability threshold, the data weights are adjusted to: LiDAR point cloud data weight > INS inertial measurement data weight > GNSS signal data weight.

[0043] As an example, with a scenario probability threshold of 80%, when the probability of a tunnel scenario is greater than 80%, the weighting is adjusted as follows: INS inertial measurement data weight 50%, LiDAR point cloud data weight 40%, and GNSS signal data weight 10%. This fully utilizes the short-term stability of INS inertial measurement data and the environmental constraints of LiDAR point cloud data. When the probability of an open area scenario is greater than 80%, the weighting is adjusted as follows: GNSS signal data weight 70%, INS inertial measurement data weight 20%, and LiDAR point cloud data weight 10%. This prioritizes the high-precision positioning results of GNSS. When the probability of a construction site scenario is greater than 80%, the weighting is adjusted as follows: LiDAR point cloud data weight 50%, GNSS signal data weight 30%, and INS inertial measurement data weight 20%. This fully utilizes the anti-interference capability of LiDAR point cloud data in metallic environments. When the probability of an urban canyon scenario is greater than 80%, the weighting is adjusted as follows: LiDAR point cloud data weight 40%, INS inertial measurement data weight 40%, and GNSS signal data weight 20%. When the probability of a port work area scenario is greater than 80%, the weights are adjusted as follows: LiDAR point cloud data weight 50%, INS inertial measurement data weight 30%, and GNSS signal data weight 20%.

[0044] It should be noted that for new scenes not included in the training set, a transfer learning mechanism is used for rapid adaptation. The parameters of the convolutional and LSTM layers in the feature extraction network are frozen, and only the fully connected layers in the scene recognition unit are fine-tuned. The learning rate is reduced to 0.0001, and 500 new scene samples are used for training. This allows the deep learning model to maintain stable recognition performance in new scenes, ensuring that the deep learning model can play an effective role in diverse occlusion environments.

[0045] S4. Perform preliminary solution calculations for each type of multi-source data to obtain the preliminary solution location for each type of data; As one possible implementation, preliminary solution calculations are performed on the multi-source data, including: The PPP-RTK model was used to solve the preprocessed GNSS signal data. A positioning equation was constructed based on carrier phase double-difference observations. Atmospheric correction parameters were introduced during the solution process to compensate for ionospheric and tropospheric delays, resulting in an initial positioning solution for the GNSS signal data. Dead reckoning was used to solve the preprocessed INS inertial measurement data. The solution results of the INS inertial measurement data were corrected by combining the weights of the lidar point cloud data, resulting in an initial positioning solution for the INS inertial measurement data. The spatial transformation relationship between the point cloud in the preprocessed lidar point cloud data and the corresponding features in the preset map was calculated to obtain the initial positioning solution for the lidar point cloud data.

[0046] As an example, for GNSS signal data, the PPP-RTK model is used for solution. The positioning equation is constructed using carrier phase double-difference observations. Atmospheric correction parameters are introduced during the solution process to compensate for ionospheric and tropospheric delays, resulting in position coordinates in the WGS84 coordinate system. Its error model is set as 1 / C / N0, meaning the higher the signal strength, the smaller the error. For INS inertial measurement data, the initial positioning solution is based on dead reckoning, and is calculated according to the corrected velocity and position increments. The formula calculates the current position, and the position correction amount is obtained by matching the LiDAR point cloud. The initial solution results of the INS inertial measurement data were corrected to obtain... .in, This represents the dynamic weights of the LiDAR point cloud data, ensuring the stability of the initial solution results from the INS inertial measurement data over long-term operation. The initial localization solution from the LiDAR point cloud data is obtained through an ICP matching algorithm between the point cloud and a pre-set map. The spatial transformation relationship between the point cloud and corresponding features in the map is calculated, and the absolute position coordinates are output. Its error is negatively correlated with point cloud density, that is 1 / The denser the point cloud, the higher the matching accuracy.

[0047] S5. The initial location and weight of each type of data are fused and calculated to obtain the location result.

[0048] As one possible implementation, S5 includes the following sub-steps: S50. The state prediction equation is constructed using the EKF master filter method, and the current position is predicted based on the EKF master filter state and INS inertial measurement data of the previous moment. S51. Calculate the GNSS deviation between the initial solution of GNSS signal data and the predicted current position, and the point cloud deviation between the initial solution of lidar point cloud data and the predicted current position. S52. The observation residual is obtained by weighted summation of GNSS bias, GNSS signal data weight, point cloud bias, and lidar point cloud data weight; S53. Calculate the Kalman gain based on the covariance of GNSS bias, the covariance of point cloud bias, the observation matrix, and the covariance of observation noise; S54. The positioning result is calculated based on the predicted current position, observation residual, and Kalman gain.

[0049] As an example, the state vector of the EKF master filter is set as follows: ,in, Indicates position coordinates, Indicates speed, This indicates that the accelerometer has zero bias. This indicates zero bias in the gyroscope. The state prediction equation is constructed based on the velocity and position increments from INS inertial measurement data, predicting the current state using the state from the previous moment and the INS inertial measurement data. The observation equation, on the other hand, integrates the initial positioning solutions from GNSS signal data and lidar point cloud data, according to... The observation residuals are constructed using a weighted method, whereby... Indicates the predicted location. Represents the dynamic weights of GNSS signal data. Represents the dynamic weights of lidar point cloud data. Kalman gain. The following method was used to calculate: in, This represents the covariance of the state prediction error. Represents the observation matrix. This represents the observation noise covariance.

[0050] Update the predicted state using Kalman gain: To obtain the optimal estimated position This is the location result.

[0051] As one possible approach, when the scene recognition result is a construction site, the following method can be used to optimize the localization result: S55. Initialize N particles, the position of each particle follows a Gaussian distribution with the mean of the positioning result obtained in step S54, and the particle weights are initialized to equal values; S56. Construct a likelihood function based on point cloud density and visual feature entropy, and calculate the matching degree between each particle and the current environment features; S57. Assign weights to each particle based on its matching degree with the current environment features and normalize the weights. Then, resample the normalized weights using the roulette wheel algorithm. S58. Calculate the weighted average value of the particles after resampling, which is the optimized localization result.

[0052] As an example, initialize 1000 particles, with the position of each particle following a certain rule. The data is distributed using a Gaussian distribution with a mean of 0.5m and a standard deviation of 0.5m. Particle weights are initialized to equal values. Then, based on the motion model of the INS inertial measurement data acquisition device, the particles undergo state transitions to simulate the carrier's trajectory. A likelihood function is then constructed using point cloud density and visual feature entropy to calculate the matching degree between each particle and the current environmental features; the higher the matching degree, the greater the particle weight. Low-weight particles are discarded to ensure the particle swarm always focuses on high-probability regions. Finally, the weighted average position of the particles after resampling is calculated. ,in, For particle position, The particle weights are used as the final positioning result. Experiments have verified that this result can control the positioning error within 0.4m in a construction site environment.

[0053] For the GPU architecture of computer terminals, operators such as convolution and LSTM in the model are optimized to improve the execution efficiency of operators by leveraging the parallel computing capabilities of the GPU. Simultaneously, the computing tasks of the CPU and GPU are rationally allocated, allowing data transmission and computation processes to overlap, further improving the system's operational efficiency. Through these lightweight processing steps, the model can run efficiently on resource-constrained platforms such as automotive embedded devices, forming a complete closed loop with the preceding multi-source sensor data acquisition, preprocessing, deep learning processing, and localization fusion processes. This ensures the real-time performance and stability of the localization function, comprehensively addressing the challenges posed by signal occlusion in occluded spaces.

[0054] Secondly, embodiments of the present invention provide a shading spatial positioning system based on multi-source data fusion, comprising: The multi-source data acquisition module synchronously acquires multi-source data from the obscured space. The multi-source data includes at least: GNSS signal data, environmental image data, INS inertial measurement data, and lidar point cloud data. The preprocessing module preprocesses each type of data in the multi-source data separately, and merges the preprocessed multi-source data to obtain standard fused data; The deep learning module performs feature extraction and scene recognition on standard fusion data, and dynamically adjusts the weight of each type of data based on the scene recognition results; The initial solution localization calculation module performs initial solution calculations on multi-source data to obtain the initial solution localization for each type of data. The system also includes a positioning result acquisition module, which integrates the initial positioning and weights of each type of data to obtain the positioning result.

[0055] As one possible implementation, the occlusion space positioning system also includes: The localization result optimization module initializes N particles that follow a normal distribution, constructs a likelihood function based on point cloud density and visual feature entropy, calculates the matching degree between each particle and the current environment features, assigns weights to each particle based on the matching degree of the current environment features and performs weight normalization, resamples the normalized weights using the roulette wheel algorithm, and calculates the weighted average of the particles after resampling, which is the optimized localization result. The lightweight deployment module performs lightweight processing on the occluded spatial positioning system and optimizes the performance of the occluded spatial positioning system using the knowledge distillation method.

[0056] As an example, lightweight deployment includes: First, the model is compressed using a pruning strategy based on the absolute value of weights. A weight threshold of 1e-4 is set, removing all connections with absolute values ​​less than this threshold and retaining only the larger, valid connections. The pruned model's parameter count is reduced from 1.2M to 360k, only 30% of the original model, ensuring core functionality remains unaffected while reducing model complexity. The pruned model is then further lightweighted by converting 32-bit floating-point weights to 8-bit integers and determining the quantization range through linear mapping. While maintaining an accuracy loss of less than 5%, the model size is compressed to less than 30% of its original size, further reducing storage and computational resource consumption and creating conditions for deployment on resource-constrained devices. Then, knowledge distillation is used to further optimize model performance. The trained original large model is used as the teacher model, and the lightweight model as the student model. During training, the output distribution of the student model is made as close as possible to the teacher model, improving the generalization ability of the lightweight model by minimizing the KL divergence loss. The distilled model achieves 89% accuracy on the test set, meeting practical application requirements and ensuring that the lightweight model does not compromise performance. The lightweight model is accelerated by using the TensorRT inference engine and converted to the ONNX format supported by TensorRT. Through optimization methods such as layer fusion and accuracy calibration of the inference engine, the inference time of the model is reduced. After optimization, the single frame processing time is reduced from 50ms to 18ms, which meets the requirements of real-time positioning for processing latency.

[0057] Thirdly, the present invention provides a terminal including a processor and a communication interface coupled to the processor, the processor being used to run computer programs or instructions to implement the occlusion spatial positioning method based on multi-source data fusion provided in the first aspect.

[0058] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, the disclosure, and the description of the drawings, in carrying out the claimed invention. In this specification, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple components. A single processor or other unit can implement several of the functions listed in the specification. While certain measures are described in different embodiments, this does not mean that these measures cannot be combined to produce good results.

[0059] Although the invention has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made therein without departing from the spirit and scope of the invention. Accordingly, this specification and drawings are merely illustrative of the invention and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if such modifications and modifications fall within the scope of the invention and its equivalents, the invention is also intended to include such modifications and modifications.

Claims

1. A method for occlusion spatial localization based on multi-source data fusion, characterized in that, Includes the following steps: S1. Synchronously collect multi-source data of the obscured space, including: GNSS signal data, environmental image data, INS inertial measurement data, and lidar point cloud data; S2. Preprocess each type of data, merge the preprocessed multi-source data, and obtain standard fused data; S3. Use a trained deep learning model to extract features and identify scenes from the standard fusion data, and dynamically adjust the weights of each data category based on the scene identification results; S4. Perform preliminary solution calculations for each type of multi-source data to obtain the preliminary solution location for each type of data; S5. The initial location and weight of each type of data are fused and calculated to obtain the location result.

2. The occlusion spatial positioning method based on multi-source data fusion according to claim 1, characterized in that, Preprocessing is performed on each type of data, including: Invalid values ​​in the GNSS signal data are removed, and the carrier phase smoothing pseudorange algorithm is used to optimize the GNSS signal data after removing invalid values; zero bias compensation is performed on the INS inertial measurement data based on extended Kalman filtering; voxel filtering downsampling is performed on the lidar point cloud data, and the point cloud density and plane fitting error are calculated; visual feature entropy and edge density features of the environmental image data are extracted.

3. The occlusion spatial positioning method based on multi-source data fusion according to claim 1, characterized in that, The deep learning model includes a feature extraction network and a scene recognition unit. The trained deep learning model is used to extract features and recognize scenes from standard fused data, including the following steps: S30. The feature extraction network extracts local features at each time step of the standard fusion data, obtains the long-term temporal dependencies of the standard fusion data, and obtains the global feature vector through global average pooling; S31. Perform nonlinear mapping on the global feature vector to obtain feature discrimination results, and identify scene categories based on the feature discrimination results. The scene categories include: urban canyons, open areas, tunnels, construction sites, and port work areas. S32. Based on the activation function included in the scene recognition unit, the scene recognition result is converted into a probability distribution of scene classes, and the probability values ​​of the current standard fusion data belonging to each scene class are obtained. The scene corresponding to the maximum probability value is the scene recognition result.

4. The occlusion spatial positioning method based on multi-source data fusion according to claim 3, characterized in that, The weights of each data category are dynamically adjusted based on the scene recognition results, including: Configure a scene probability threshold. When the probability of a tunnel scene is greater than the scene probability threshold, adjust the data weights to: INS inertial measurement data weight > LiDAR point cloud data weight > GNSS signal data weight. When the probability of an open field scene is greater than the scene probability threshold, the data weights are adjusted to: GNSS signal data weight > INS inertial measurement data weight > lidar point cloud data weight. When the probability of a construction site scenario is greater than the scenario probability threshold, the data weights are adjusted to: LiDAR point cloud data weight > GNSS signal data weight > INS inertial measurement data weight. When the probability of an urban canyon scene is greater than the scene probability threshold, the data weights are adjusted as follows: LiDAR point cloud data weight = INS inertial measurement data weight > GNSS signal data weight. When the probability of a port work area scenario is greater than the scenario probability threshold, the data weights are adjusted to: LiDAR point cloud data weight > INS inertial measurement data weight > GNSS signal data weight.

5. The occlusion spatial positioning method based on multi-source data fusion according to claim 2, characterized in that, Preliminary calculations were performed on the multi-source data, including: The PPP-RTK model was used to solve the preprocessed GNSS signal data. A positioning equation was constructed based on carrier phase double-difference observations. Atmospheric correction parameters were introduced during the solution process to compensate for ionospheric and tropospheric delays, resulting in an initial positioning solution for the GNSS signal data. Dead reckoning was used to solve the preprocessed INS inertial measurement data. The solution results of the INS inertial measurement data were corrected by combining the weights of the lidar point cloud data, resulting in an initial positioning solution for the INS inertial measurement data. The spatial transformation relationship between the point cloud in the preprocessed lidar point cloud data and the corresponding features in the preset map was calculated to obtain the initial positioning solution for the lidar point cloud data.

6. The occlusion spatial positioning method based on multi-source data fusion according to claim 5, characterized in that, S5 includes the following sub-steps: S50. The state prediction equation is constructed using the EKF master filter method, and the current position is predicted based on the EKF master filter state and INS inertial measurement data of the previous moment. S51. Calculate the GNSS deviation between the initial solution of GNSS signal data and the predicted current position, and the point cloud deviation between the initial solution of lidar point cloud data and the predicted current position. S52. The observation residual is obtained by weighted summation of GNSS bias, GNSS signal data weight, point cloud bias, and lidar point cloud data weight; S53. Calculate the Kalman gain based on the covariance of GNSS bias, the covariance of point cloud bias, the observation matrix, and the covariance of observation noise; S54. The positioning result is calculated based on the predicted current position, observation residual, and Kalman gain.

7. The occlusion spatial positioning method based on multi-source data fusion according to claim 6, characterized in that, When the scene recognition result is a construction site, the following method is used to optimize the localization result: S55. Initialize N particles, the position of each particle follows a Gaussian distribution with the mean of the positioning result obtained in step S54, and the particle weights are initialized to equal values; S56. Construct a likelihood function based on point cloud density and visual feature entropy, and calculate the matching degree between each particle and the current environment features; S57. Assign weights to each particle based on its matching degree with the current environment features and normalize the weights. Then, resample the normalized weights using the roulette wheel algorithm. S58. Calculate the weighted average value of the particles after resampling, which is the optimized localization result.

8. A spatial positioning system for occlusion based on multi-source data fusion, characterized in that, include: The multi-source data acquisition module synchronously acquires multi-source data of the obscured space, including at least: GNSS signal data, environmental image data, INS inertial measurement data, and lidar point cloud data; The preprocessing module preprocesses each type of data in the multi-source data separately, and merges the preprocessed multi-source data to obtain standard fused data; The deep learning module performs feature extraction and scene recognition on standard fusion data, and dynamically adjusts the weight of each type of data based on the scene recognition results; The initial solution localization calculation module performs initial solution calculations on multi-source data to obtain the initial solution localization for each type of data. The system also includes a positioning result acquisition module, which integrates the initial positioning and weights of each type of data to obtain the positioning result.

9. The occlusion spatial positioning system based on multi-source data fusion according to claim 8, characterized in that, The concealed spatial positioning system also includes: The localization result optimization module initializes N particles that follow a normal distribution, constructs a likelihood function based on point cloud density and visual feature entropy, calculates the matching degree between each particle and the current environment features, assigns weights to each particle based on the matching degree of the current environment features and performs weight normalization, resamples the normalized weights using the roulette wheel algorithm, and calculates the weighted average of the particles after resampling, which is the optimized localization result. The lightweight deployment module performs lightweight processing on the occluded spatial positioning system and optimizes the performance of the occluded spatial positioning system using the knowledge distillation method.

10. A terminal, comprising a processor and a communication interface coupled to the processor, the processor being configured to run a computer program or instructions to implement the occlusion spatial positioning method based on multi-source data fusion as described in any one of claims 1 to 7.

Images