An end-to-end alignment and localization method and system for multi-source heterogeneous sensing data

Abstract

This invention discloses an end-to-end alignment and localization method and system for multi-source heterogeneous sensing data, belonging to the field of wireless communication and positioning technology. The method addresses the problems of inconsistent sampling rates among heterogeneous sensing devices and data loss caused by communication interruptions during acquisition. It includes: initializing the acquisition system and unifying the global absolute clock; synchronously acquiring low-frequency communication signals and high-frequency inertial measurement features; backfilling and compensating for data gaps during disconnection periods based on local offline logs and timestamp indexes; using a time-series sliding window to splice and fuse low-frequency and high-frequency data within the same period according to sampling multiples to generate a multimodal composite feature sequence; inputting the multimodal composite feature sequence into a deep network regression model, and suppressing non-line-of-sight errors through an attention mechanism to output the target's two-dimensional coordinates. This method can be used for wireless sensing and localization in complex environments.

Description

Technical Field

[0001] This invention relates to the field of wireless communication and indoor positioning technology, and more specifically, to an end-to-end alignment and positioning method and system for multi-source heterogeneous sensing data. Background Technology

[0002] In the current field of positioning in complex environments (such as indoors, underground parking lots, or densely populated urban areas), positioning models based on the fusion of multi-source heterogeneous data (such as 5G communication signals and inertial sensor data) have become mainstream. However, in real-world industrial deployment and data acquisition, the following serious challenges remain: First, the sampling frequencies of heterogeneous devices are mismatched, and the fusion method is distorted. Traditional feature fusion often relies on linear interpolation algorithms to upsample low-frequency signals (such as base station signals) to align with high-frequency sensors (such as IMUs). This approach introduces a large amount of spurious "artifact data," destroying the true fading and distortion characteristics of radio signals in physical space, causing the model to learn incorrect channel prior knowledge.

[0003] Secondly, the distributed collaborative acquisition network lacks resilience. In complex environments with physical obstructions, the internal LAN backhaul link between the main control terminal and the acquisition chassis is prone to unpredictable fluctuations and disconnections, leading to the direct loss of key features and trajectory breaks.

[0004] Finally, conventional models suffer from limited accuracy under non-line-of-sight (NLOS) interference. Indoor environments, wall refraction and human occlusion can cause severe NLOS distortion. Traditional neural networks map each sampling point as an independent feature, lacking the ability to extract the global temporal dependencies of heterogeneous multimodal sequences, making it difficult to adaptively identify and suppress sudden noise. Summary of the Invention

[0005] The purpose of this invention is to provide an end-to-end alignment and positioning method and system for multi-source heterogeneous sensing data, so as to solve the problems of heterogeneous data interpolation distortion, fragile acquisition links and low non-line-of-sight positioning accuracy in the prior art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: An end-to-end alignment and localization method for multi-source heterogeneous sensing data includes the following steps: Step S1: Place the main control terminal and the data acquisition robot on the same local area network to establish an internal communication link, and call the network time protocol to synchronize the global clock, using standard Coordinated Universal Time as the unified absolute timestamp; Step S2: Perform multi-source heterogeneous data dual-track synchronous acquisition in the target scene: The main control terminal receives low-frequency communication signal parameters sent by the external base station in real time and simultaneously acquires high-frequency inertial navigation features; the acquisition robot obtains the real physical position coordinates in real time; all acquired data are stamped with the corresponding absolute timestamp. Step S3: The main control terminal persistently saves the collected low-frequency communication signal parameters and high-frequency inertial navigation features in the local offline log and reports them to the server synchronously; after the data acquisition task is completed, disconnection compensation is performed: using the absolute timestamp as the index, the data that the server did not receive due to the internal communication link being disconnected is accurately backfilled using the local offline log to generate a time series reference dataset without missing features; Step S4: The time-series reference dataset is divided into time-axis segments using a fixed-duration time-series sliding window. The low-frequency communication signal parameters and the high-frequency inertial navigation features within the same time window are extracted and spliced ​​according to a preset ratio to construct a multimodal composite feature sequence. Step S5: Input the multimodal composite feature sequence into the pre-constructed deep network localization regression model for feature extraction, and directly output the two-dimensional localization coordinates of the target robot through the output layer of the deep network localization regression model.

[0007] Preferably, the disconnection compensation in step S3 specifically includes: When the internal communication link is detected to be interrupted, the server continuously records the physical position coordinates of the acquisition robot and marks the corresponding signal parameters and feature fields as missing. During the offline reconstruction process, the absolute timestamp sequence of the server-side coordinate dataset is used as the global retrieval benchmark to traverse and detect data rows marked as missing. Using the absolute timestamp corresponding to the missing data row as an index, feature records of the same time are matched in the local offline log file for fixed-point insertion and numerical backfilling.

[0008] Preferably, the process of constructing and generating a multimodal composite feature sequence in step S4 specifically includes: The fixed duration of the timing sliding window is set to 200 milliseconds; For a sampling frequency of 5Hz for the low-frequency communication signal parameters and 10Hz for the high-frequency inertial navigation features, within any one-way sliding time window, the low-frequency communication signal parameters corresponding to one physical sampling point and the high-frequency inertial navigation features corresponding to two physical sampling points within the corresponding time period are extracted. The extracted low-frequency communication signal parameters and the high-frequency inertial navigation features are merged and flattened in the data dimension to generate the multimodal composite feature sequence with a composite feature token structure.

[0009] Preferably, the deep network localization and regression model in step S5 is a sequence model that includes a multi-head self-attention mechanism, and the feature extraction process specifically includes: The multi-head self-attention mechanism is used to calculate the dot product correlation between the multimodal composite feature sequences over multiple consecutive time steps; When the low-frequency communication signal parameters are distorted due to non-line-of-sight fading caused by the physical environment, the multi-head self-attention mechanism adaptively reduces the attention weight of the distorted feature frames, while increasing the weight of the high-frequency inertial navigation features in a stable state, thereby suppressing non-line-of-sight and multipath interference through multi-source collaborative compensation.

[0010] Preferably, the time-series benchmark dataset includes data repeatedly collected and fused within multiple time periods of the target scene with different pedestrian density or obstacle distribution states, so that the deep network localization regression model can learn spatial absolute location features that are not affected by dynamic time-varying environmental factors.

[0011] Preferably, an end-to-end alignment and localization system for multi-source heterogeneous sensing data is characterized by comprising: Initialization and synchronization module: used to establish internal communication links and perform global clock normalization based on standard Coordinated Universal Time; Dual-track synchronous acquisition module: used to synchronously capture low-frequency communication signal parameters, high-frequency inertial navigation characteristics, and real physical position coordinates based on a unified clock; Data reporting and compensation module: used to trigger local offline log storage and perform precise data backfilling based on absolute timestamp gaps after the link anomaly ends; The temporal feature alignment module is used to construct a multimodal composite feature sequence by splicing the low-frequency communication signal parameters and the high-frequency inertial navigation features within the same time window according to a preset ratio using a time-series sliding window of fixed duration. Deep model regression module: used to receive the multimodal composite feature sequence and output the target positioning coordinates through a pre-built deep network localization regression model.

[0012] Preferably, a computer device includes a memory and a processor, characterized in that the memory stores a computer program, and the processor executes the computer program to implement the steps of the positioning method as described in any one of claims 1 to 5.

[0013] Preferably, a computer-readable storage medium stores a computer program thereon, characterized in that, when the computer program is executed by a processor, it implements the steps of the positioning method as described in any one of claims 1 to 5.

[0014] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure.

[0015] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0017] Figure 1 This is a schematic diagram of the system hardware acquisition architecture provided in an embodiment of the present invention.

[0018] Figure 2 The flowchart illustrates the method for aligning and locating multi-source heterogeneous data provided in this embodiment of the invention.

[0019] Figure 3 This is a schematic diagram illustrating the combination logic of a timing window and a multimodal composite feature token provided in an embodiment of the present invention.

[0020] Figure 4 This is a comparison chart of the cumulative distribution of model localization error under different training stages provided in this embodiment of the invention. Detailed Implementation

[0021] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0022] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0023] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0024] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods and means well-known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention. Example 1

[0026] Reference Figure 1 and Figure 2 This embodiment provides a method for data acquisition and preprocessing.

[0027] In step S1, the main control terminal (such as a 5G smartphone) and the data acquisition robot (a chassis equipped with LiDAR) are initialized. The NTP protocol is used to connect to the time server to ensure that the system clock offset is eliminated, and an absolute timestamp is generated using Coordinated Universal Time (UTC). Before formal data acquisition, the device is left idle to acquire IMU zero-bias data for underlying error calibration.

[0028] In step S2, dual-track synchronous acquisition is performed. The main control terminal acquires low-frequency communication signal parameters (such as reference signal received power RSRP, sampling rate 5Hz) and high-frequency inertial navigation characteristics (such as three-axis acceleration and angular velocity, sampling rate 10Hz). The acquisition robot synchronously calculates the true two-dimensional physical coordinates (Ground Truth).

[0029] In step S3, high-resilience disconnection compensation is triggered. When the master terminal reports data via the HTTP link, if a network interruption is detected, the cloud server generates a missing marker (NaN) at the corresponding timestamp position. After the data collection is completed, the complete logs (such as .rtss files) stored locally on the master terminal are parsed offline, and precise fixed-point insertion and numerical backfilling are performed using the UTC timestamp as the primary key to reconstruct a time-series continuous benchmark dataset. Example 2

[0030] Reference Figure 3To address the issue of inconsistent data frequencies in step S4, this embodiment employs a "time-series sliding window" technique instead of the traditional interpolation algorithm to achieve early fusion of multimodal data. The sampling frequency for low-frequency communication signal parameters is set to 5Hz (i.e., one sampling point R is generated every 200ms), and the sampling frequency for high-frequency inertial navigation features is set to 10Hz (i.e., one sampling point I is generated every 100ms). The two exhibit a strict 1:2 ratio.

[0031] The specific logic for constructing the multimodal composite feature sequence (Token) is as follows: The fixed duration of the time-series sliding window is set to 200ms. Within the i-th time window (i.e., within the time interval [200 * (i-1), 200 * i) milliseconds), the system extracts exactly one low-frequency communication signal parameter Ri, and the corresponding two high-frequency inertial navigation features I. 2i-1 with I 2i Subsequently, the aforementioned heterogeneous features are merged and flattened along the feature dimension. If R i The dimension is D R The dimension D of a single I I Then the mathematical expression for the i multimodal composite feature tokens generated by the combination is:

[0032] Its total feature dimension is D R + 2D I Similarly, by sliding along the time axis, a multimodal composite feature sequence with a continuous time step is generated and fed into the downstream model. This scheme completely avoids the spurious data introduced by upsampling, ensuring the absolute authenticity of the physical features. Example 3

[0033] In step S5, this embodiment preferably uses a Transformer sequence model that includes a multi-head self-attention mechanism as the localization regression model. The tokens generated in step S4 for multiple consecutive time steps are packaged into a two-dimensional spatiotemporal sequence matrix, and then positional encoding is superimposed before being input into the encoder network.

[0034] To address non-line-of-sight (NLOS) interference, the model operates as follows: Within a sliding time window, a multi-head self-attention mechanism calculates the global dot product correlation between token sequences. When a communication signal experiences severe fading and distortion due to human occlusion at a given moment, the similarity between the distorted frame and historical stable frames drops sharply. At this point, the self-attention mechanism adaptively assigns extremely low attention weights to the distorted signal while transferring high weights to stable and continuous IMU features within the same window. Through this multimodal temporal compensation mechanism, the model implicitly filters out multipath interference and NLOS noise, ultimately outputting high-precision two-dimensional physical coordinates (x, y) directly through a fully connected regression layer (MLP Head).

[0035] To verify the effectiveness of the end-to-end alignment and positioning method described in this invention, model training and evaluation were performed in a real lobby scenario. For example... Figure 4 As shown, in the initial small sample training phase, the model's 80th percentile localization error is approximately 3.5 meters. With the continuous input of the multimodal composite feature tokens described in this invention and the global learning of temporal features by the multi-head self-attention mechanism, the model converges rapidly. In the full-data scenario, the average error of the method described in this invention is reduced to 1.41 meters, and the 80th percentile error is significantly compressed to 2.3 meters. The experimental results fully demonstrate that the lossless temporal feature alignment mechanism described in this invention completely eliminates the underlying asynchronous artifacts. Combined with joint loss constraints, it can achieve highly robust and high-precision two-dimensional physical coordinate prediction in medium to large-scale complex indoor scenes, possessing strong industrial application value.

[0036] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for end-to-end alignment and localization of multi-source heterogeneous perception data, comprising: Includes the following steps: Step S1: Place the main control terminal and the data acquisition robot on the same local area network to establish an internal communication link, and call the network time protocol to synchronize the global clock, using standard Coordinated Universal Time as the unified absolute timestamp; Step S2: Perform multi-source heterogeneous data dual-track synchronous acquisition in the target scene: The main control terminal receives low-frequency communication signal parameters sent by the external base station in real time and simultaneously acquires high-frequency inertial navigation features; the acquisition robot obtains the real physical position coordinates in real time; all acquired data are stamped with the corresponding absolute timestamp. Step S3: The main control terminal persistently saves the collected low-frequency communication signal parameters and high-frequency inertial navigation features in the local offline log and reports them to the server synchronously; after the data acquisition task is completed, disconnection compensation is performed: using the absolute timestamp as the index, the data that the server did not receive due to the internal communication link being disconnected is accurately backfilled using the local offline log to generate a time series reference dataset without missing features. Step S4: The time-series reference dataset is divided into time-axis segments using a fixed-duration time-series sliding window. The low-frequency communication signal parameters and the high-frequency inertial navigation features within the same time window are extracted and spliced ​​according to a preset ratio to construct a multimodal composite feature sequence. Step S5: Input the multimodal composite feature sequence into the pre-constructed deep network localization regression model for feature extraction, and directly output the two-dimensional localization coordinates of the target robot through the output layer of the deep network localization regression model.

2. The positioning method according to claim 1, characterized in that, The disconnection compensation in step S3 specifically includes: When the internal communication link is detected to be interrupted, the server continuously records the physical position coordinates of the acquisition robot and marks the corresponding signal parameters and feature fields as missing. During the offline reconstruction process, the absolute timestamp sequence of the server-side coordinate dataset is used as the global retrieval benchmark to traverse and detect data rows marked as missing. Using the absolute timestamp corresponding to the missing data row as an index, feature records at the same time are matched in the local offline log file for targeted insertion and numerical backfilling.

3. The positioning method of claim 1, wherein, The process of constructing and generating the multimodal composite feature sequence in step S4 specifically includes: The fixed duration of the timing sliding window is set to 200 milliseconds; For a sampling frequency of 5Hz for the low-frequency communication signal parameters and 10Hz for the high-frequency inertial navigation features, within any one-way sliding time window, the low-frequency communication signal parameters corresponding to one physical sampling point and the high-frequency inertial navigation features corresponding to two physical sampling points within the corresponding time period are extracted. The extracted low-frequency communication signal parameters and the high-frequency inertial navigation features are merged and flattened in the data dimension to generate the multimodal composite feature sequence with a composite feature token structure.

4. The positioning method according to claim 1 or 3, characterized in that, The deep network localization and regression model in step S5 is a sequence model that includes a multi-head self-attention mechanism. The feature extraction process specifically includes: The multi-head self-attention mechanism is used to calculate the dot product correlation between the multimodal composite feature sequences over multiple consecutive time steps; When the low-frequency communication signal parameters are distorted due to non-line-of-sight fading caused by the physical environment, the multi-head self-attention mechanism adaptively reduces the attention weight of the distorted feature frames, while increasing the weight of the high-frequency inertial navigation features in a stable state, thereby suppressing non-line-of-sight and multipath interference through multi-source collaborative compensation.

5. The positioning method of claim 1, wherein, The time-series benchmark dataset contains data that is repeatedly collected and fused within multiple time periods of the target scene with different pedestrian density or obstacle distribution states, so that the deep network localization regression model can learn the spatial absolute position features that are not affected by dynamic time-varying environmental factors.

6. A multi-source heterogeneous perception data end-to-end alignment and localization system, comprising: include: Initialization and synchronization module: used to establish internal communication links and perform global clock normalization based on standard Coordinated Universal Time; Dual-track synchronous acquisition module: used to synchronously capture low-frequency communication signal parameters, high-frequency inertial navigation characteristics, and real physical position coordinates based on a unified clock; Data reporting and compensation module: used to trigger local offline log storage and perform precise data backfilling based on absolute timestamp gaps after the link anomaly ends; The temporal feature alignment module is used to construct a multimodal composite feature sequence by splicing the low-frequency communication signal parameters and the high-frequency inertial navigation features within the same time window according to a preset ratio using a time-series sliding window of fixed duration. Deep model regression module: used to receive the multimodal composite feature sequence and output the target positioning coordinates through a pre-built deep network localization regression model. 7.A computer device, comprising a memory and a processor, and characterized in that, The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the positioning method as described in any one of claims 1 to 5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the positioning method as described in any one of claims 1 to 5.

Images