An inter-device ranging positioning system based on multi-modal fusion

Abstract

The application discloses a multi-modal fusion-based inter-device ranging and positioning system, which comprises a ranging module and a joint positioning module. The ranging module comprises a single-modal ranging unit and a real-time distance fusion unit, and is used for obtaining a predicted distance through a first Kalman filter based on low-frequency absolute distance and high-frequency relative displacement and speed. The joint positioning module is used for taking the predicted distance as a side constraint and reconstructing the relative position topology of nodes. The implementation of the application can realize corresponding ranging processing based on existing commercial devices and corresponding sensing and communication capabilities, and is easy to land in a real multi-device ecosystem. In addition, the improved double-Kalman filter multi-modal signal fusion scheme solves the synchronization problem between heterogeneous sensor signals. In addition, the application can be extended to multi-device joint positioning and topology reconstruction, so that more stable overall spatial relationship inference results can be obtained.

Description

Technical Field

[0001] This invention relates to the field of wireless communication technology, and in particular to a device ranging and positioning system based on multimodal fusion. Background Technology

[0002] As the intelligence level of personal terminals increases and their numbers grow, collaboration and interaction between multiple devices are receiving increasing attention. For example, leading companies like Apple, Samsung, and Huawei have been continuously iterating on related features for years, including iWatch-assisted Mac unlocking, multi-screen collaboration, screen rotation, and iPhone mirroring. Device interaction often relies on their relative positions. For instance, when a watch approaches a laptop, the laptop should automatically unlock; as the distance between devices increases, the screen should automatically lock to protect privacy. Similarly, when a user moves a smartphone towards a tablet, the phone screen content automatically switches to the tablet. Therefore, accurately estimating the relative positions of devices is crucial.

[0003] Current methods for estimating the relative positions of devices include WiFi-based, voice-based, and Bluetooth Low Energy (BLE)-based methods. These methods typically rely on a fixed anchor device to calculate the relative positions of other devices. However, in real-world device interaction scenarios, the positions of all devices (e.g., laptops, smartwatches, smartphones) are often not fixed, making these methods insufficient for estimating the relative positions of devices in certain scenarios.

[0004] Therefore, existing technologies have begun to explore the possibility of self-localization for non-stationary devices. For example, CAT has built a distributed FMCW (Frequency Modulated Continuous Wave) system that can achieve a tracking accuracy of 5 mm, but it requires a complex calibration process to eliminate unknown clock offsets; SCALAR has proposed a self-calibrated acoustic ranging system that can achieve sub-millimeter accuracy on distributed asynchronous devices, but its ranging distance is limited to 3 meters; Shiraki et al. have achieved BLE-based pedestrian localization, but its error is as high as 0.74 m, which is far from meeting the basic requirement of estimating the relative positional relationship between devices.

[0005] There is also a corresponding technical solution that uses the UWB (Ultra-Wideband) capabilities of two Samsung Galaxy S21+ 5G smartphones for ranging and positioning, achieving high accuracy over long distances; however, due to the high cost of UWB modules, not all smartphones are equipped with UWB, which limits the universality of this solution.

[0006] In view of this, the present invention is hereby proposed. Summary of the Invention

[0007] The purpose of this invention is to provide a device ranging and positioning system based on multimodal fusion, so as to realize ranging and positioning operations in a real multi-device environment and solve the problems existing in the prior art.

[0008] The objective of this invention is achieved through the following technical solution:

[0009] A device ranging and positioning system based on multimodal fusion includes a ranging module and a joint positioning module. The ranging module includes a single-modal ranging unit and a real-time distance fusion unit, wherein:

[0010] The single-mode ranging unit includes: a low-power Bluetooth BLE ranging subunit and an ultrasonic ranging subunit for low-frequency absolute distance measurement, and an inertial measurement unit (IMU) subunit for high-frequency relative displacement and velocity measurement.

[0011] A real-time distance fusion unit is used to obtain a predicted distance by processing the low-frequency absolute distance and the high-frequency relative displacement and velocity through a first Kalman filter.

[0012] The joint localization module is used to reconstruct the relative position topology of nodes by using the predicted distance as a constraint on the edges.

[0013] The distance estimation process performed by the BLE ranging subunit includes:

[0014] The Received Signal Strength Indicator (RSSI) value is obtained and mapped to distance according to the classical logarithmic distance path loss model. Then, Gaussian statistical filtering is applied to the RSSI sequence, and the mean µ and standard deviation σ are calculated within a sliding time window. Samples located in the interval [µ-3σ, µ+3σ] are retained as valid data for distance estimation. During the ranging process, the distance range is divided into short-range segments and long-range segments according to predetermined rules, and least squares (LS) is used for curve fitting or model training to achieve segmented distance estimation.

[0015] The distance estimation process performed by the ultrasonic ranging subunit is implemented using a Time-of-Flight (ToF) ranging mechanism and includes:

[0016] distance The calculation formula is: ,in , The time when the sound wave is first transmitted at point A and received at point B. , The time when the second sound wave is transmitted at point B and received at point A. The speed of sound; and the speed of sound is either a direct value or a value adjusted for temperature; the temperature-adjusted speed of sound value is... , The ambient temperature.

[0017] A dual-threshold processing mechanism is adopted: the first threshold is used to filter cross-correlation noise peaks; the second threshold is used to constrain the received energy, and if the energy is insufficient, it is determined to be an invalid arrival.

[0018] The process by which the IMU subunit determines relative displacement and velocity includes:

[0019] Get the pose quaternion ,in: For angle scalar parameters, All parameters are orthogonal rotation axis vectors; attitude quaternions are used to transform the acceleration and angular velocity obtained from accelerometers and gyroscopes to the geographic coordinate system;

[0020] Based on the geographic coordinate system, within the distance update interval of adjacent absolute distances, the acceleration is integrated to obtain the velocity increment, and then integrated again to obtain the displacement increment, thereby realizing the calculation of the corresponding high-frequency relative displacement and velocity.

[0021] The absolute distance measurement strategies implemented by the BLE ranging subunit and the ultrasonic ranging subunit include:

[0022] When only BLE ranging results are available, the distance measured by BLE is used as the absolute distance;

[0023] When both BLE ranging and ultrasonic ranging are available, the distance measured by ultrasonic ranging shall be used as the absolute distance.

[0024] In the intended short-range scenario, if ultrasonic ranging temporarily fails, BLE will be used as a fallback.

[0025] Before each ultrasonic ranging operation, a ranging notification is sent to the other end via BLE, and a BLE segmented fitting model is selected based on the current distance stage.

[0026] The real-time distance fusion unit further includes performing differential processing on the predicted distance to obtain the corresponding theoretical velocity value, and performing an error correction operation on the original velocity of the IMU via a second Kalman filter for the theoretical velocity value, and the corrected velocity of the IMU is used as the input of the first Kalman filter.

[0027] The processing performed by the first Kalman filter included in the real-time distance fusion unit includes:

[0028] The execution state prediction process is as follows: , ;

[0029] Perform the parameter calibration process, namely: , , ;

[0030] in, Let be the absolute distance at time k. This is the IMU relative displacement input at time k. For process noise covariance, To observe the noise covariance, For Kalman gain, For prior prediction of distance, For the current distance estimate, To estimate the covariance posteriorly, for Prior estimation of covariance

[0031] The predicted distance is obtained through the above process. ;

[0032] Furthermore, during the dynamic state estimation process of the first Kalman filter, an adaptive adjustment strategy for the noise covariance is adopted. This adjustment strategy includes: using innovation quantity... Update the process noise covariance at time k. Through residuals Update the observation noise covariance at time k. And introduce a forgetting factor. Control the smooth update of Qi.

[0033] The processes performed by the joint positioning module include:

[0034] Randomly generated The initial positions of the group nodes are obtained by running the mass-spring energy minimization model. N candidate topology proposals are then used; the consistency between the N candidate topologies and the ranging constraints is evaluated using the Global Error Rate (GER), and the topology proposal with the smallest GER is selected as the output of the joint localization.

[0035] In the processing performed by the joint positioning module:

[0036] The processing steps of the running mass-spring energy minimization model include:

[0037] In the mass-spring energy minimization model, connected nodes are considered to be connected by springs, and the entire system tends to the minimum energy state minE; assuming nodes... Its connected nodes The distance measured between them is The actual distance under the current layout is The spring constant is Then the node energy is :

[0038] ;

[0039] Total system energy Where, for cases where no effective distance is measured between two nodes or the distance exceeds the system's range threshold, a penalty term for edge potential energy is introduced, namely:

[0040] If the distance between two nodes in the predicted layout is less than a threshold but the original graph has no edges, then the predicted distance is added as a penalty, resulting in:

[0041] ;

[0042] The topology between nodes is obtained through the above process;

[0043] In the process of evaluating the consistency between the N candidate topologies and the ranging constraints determined by the ranging module using GER, the definition of GER includes:

[0044] ,in It is used to calculate the relative error between the theoretical distance and the measured distance between devices i and j. The total number of nodes to be located. This represents the measured distance between nodes i and j. This is to calculate the distance between nodes i and j in the optimized topology.

[0045] The joint positioning module also includes anchor-assisted absolute positioning processing, which includes:

[0046] When there are three or more devices with known locations that can be used as anchor points, their coordinates are fixed to known values, and the absolute coordinates of the remaining nodes are obtained based on the known values.

[0047] Compared with existing technologies, the device ranging and positioning system based on multimodal fusion provided by this invention can achieve the corresponding ranging processing entirely based on existing commercial equipment and corresponding sensing and communication capabilities, making it easy to implement in real multi-device ecosystems. Furthermore, it proposes a processing method that fuses BLE, audio, and IMU signals to measure the relative distance between mobile devices, and effectively solves the synchronization problem between heterogeneous sensor signals through an improved dual Kalman filter multimodal signal fusion scheme. In addition, embodiments of this invention can be extended to multi-device joint positioning and topology reconstruction, thereby obtaining more stable overall spatial relationship inference results. Attached Figure Description

[0048] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 A flowchart illustrating the overall architecture of the system provided in this embodiment of the invention;

[0050] Figure 2 This is a schematic diagram illustrating the relationship between BLE RSSI and distance in three different environments provided in this embodiment of the invention;

[0051] Figure 3 A block diagram illustrating the implementation principle of single-mode ranging and real-time distance fusion provided in an embodiment of the present invention;

[0052] Figure 4 A schematic diagram illustrating the ranging success rate and error of ultrasonic and Bluetooth Low Energy technologies at different distances, provided in an embodiment of the present invention.

[0053] Figure 5 A schematic diagram of the real-time distance fusion algorithm provided in an embodiment of the present invention;

[0054] Figure 6 A schematic diagram of the joint positioning process provided in an embodiment of the present invention.

[0055] Figure 7 This is a schematic diagram of the experimental scene equipment provided in an embodiment of the present invention;

[0056] Figure 8 This is a schematic diagram of the fusion ranging experiment results provided in the embodiments of the present invention. Figure 1 ;

[0057] Figure 9 This is a schematic diagram of the fusion ranging experiment results provided in the embodiments of the present invention. Figure 2 ;

[0058] Figure 10 Schematic diagram of joint positioning experiment results provided in embodiments of the present invention Figure 1 ;

[0059] Figure 11 Schematic diagram of joint positioning experiment results provided in embodiments of the present invention Figure 2 . Detailed Implementation

[0060] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the specific content of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments, which do not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0061] First, the following explanations are provided for the terms that may be used in this article:

[0062] The term "and / or" means that either or both can be achieved simultaneously. For example, X and / or Y means that it includes both "X" or "Y" as well as the three cases of "X and Y".

[0063] The terms "comprising," "including," "containing," "having," or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.) should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0064] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0065] Unless otherwise explicitly specified or limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this document according to the specific circumstances.

[0066] When concentration, temperature, pressure, size, or other parameters are expressed as numerical ranges, such ranges should be understood to specifically disclose all ranges formed by any pairing of upper limits, lower limits, or preferred values ​​within that range, regardless of whether the range is explicitly stated; for example, if the numerical range "2 to 8" is stated, then that range should be interpreted to include ranges such as "2 to 7", "2 to 6", "5 to 7", "3 to 4 and 6 to 7", "3 to 5 and 7", "2 and 5 to 7", etc. Unless otherwise stated, the numerical ranges described herein include both their endpoints and all integers and fractions within that range.

[0067] The terms “center,” “longitudinal,” “lateral,” “length,” “width,” “thickness,” “upper,” “lower,” “front,” “back,” “left,” “right,” “vertical,” “horizontal,” “top,” “bottom,” “inner,” “outer,” “clockwise,” and “counterclockwise” indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience and simplification of description and do not imply that the device or component referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this document.

[0068] In the implementation process of this invention, it was found that an ideal device-to-device relative ranging and positioning solution needs to meet the following characteristics:

[0069] Zero additional deployment means it can be easily deployed on commercial equipment without modifying the hardware or installing dedicated anchor devices; this feature significantly reduces deployment complexity and cost, making the technology easier to implement for a wider range of users and environments.

[0070] For long-distance measurement, the sensing distance should be at least 10 meters to support diverse scenarios, so as to meet the needs of most indoor IoT applications (such as multi-device interaction and indoor navigation) and provide greater flexibility between short- and medium-range usage scenarios.

[0071] Low latency response provides real-time ranging results with a response time of no more than 0.1 seconds, matching human perception of motion (persistence of vision) to ensure a natural and seamless user experience.

[0072] To achieve the desired ranging and positioning goals by meeting the above characteristics, this invention proposes BAI: a multimodal relative ranging and joint positioning system that can be practically deployed on commercial smartphones, namely, a device ranging and positioning system based on multimodal fusion. BAI can fully utilize Bluetooth signals, ultrasonic signals, and inertial sensor signals on commercial smartphones to achieve real-time, long-distance, and fine-grained relative distance measurement between devices without additional deployment. Furthermore, when multiple devices are in free space, BAI can also generate a relative position map, thereby improving the understanding of global spatial information.

[0073] In the process of implementing this invention, it was found that designing the above-mentioned smartphone-based ranging and positioning system capable of achieving ideal ranging and positioning targets requires solving the following core technical challenges:

[0074] (1) How to align multimodal sensor signals with different measurement objects and different ranging frequencies

[0075] Different sensors measure different objects and have different sampling rates, so consistent data representation is needed for multimodal fusion. At the same time, due to differences in equipment, the information density of each mode also varies greatly. For example, sound-based methods usually operate at a frequency of 1 Hz to measure the absolute distance between devices, while IMUs often output at a higher frequency of 50 Hz to reflect relative distance changes.

[0076] (2) How to achieve real-time response under the condition of limited computing resources?

[0077] To meet the needs of real-world scenarios, the ranging results need to be output within 0.1 seconds; however, the computing power of mobile phones and other terminals is relatively limited. For example, the computing power of the Qualcomm 865 chip is about 15 TOPS, but to achieve real-time response, the corresponding computing power needs to be shared by dozens or even hundreds of system services and applications at the same time.

[0078] (3) How to reconstruct the topology based solely on the distance between devices when there is accumulated error?

[0079] Unlike traditional sensor networks, smartphone-based ranging is more susceptible to errors; it also lacks additional information such as angles; the accumulation of ranging errors between devices weakens the reconstruction effect; to adapt to practical applications, no limit can be placed on the degree of nodes, but this may lead to a "non-rigid" graph structure and further introduce structural ambiguity errors.

[0080] To address the aforementioned three technical challenges and achieve device ranging with zero additional deployment, meeting the requirements for long-distance and low-latency interaction, and supporting joint positioning and topology reconstruction of multiple devices, this invention provides a corresponding device ranging and positioning system based on multimodal fusion. This system may include a ranging module and a joint positioning module. The ranging module estimates the relative distance between two mobile terminals (e.g., smartphones, tablets, smartwatches, etc.) in real time. The joint positioning module reconstructs the relative position topology in a network composed of multiple terminals based on pairwise distance constraints and outputs absolute coordinates when anchor devices are present.

[0081] Specifically, refer to Figure 1 As shown, the system includes a ranging module BAI-R and a joint positioning module BAI-L. After processing by these two modules, an output location map and final positioning can be obtained, thus enabling the corresponding ranging and positioning. The specific implementation methods of the ranging module BAI-R and the joint positioning module BAI-L can include:

[0082] (a) Distance measuring module

[0083] The ranging module may include a single modal-based ranging unit and a real-time distance fusion unit, wherein...

[0084] (1) The corresponding single-mode ranging unit may include a BLE ranging subunit, an ultrasonic ranging subunit, and an IMU (Inertial Measurement Unit) calculation unit, wherein:

[0085] (11) The BLE ranging subunit can output BLE distance estimates based on BLE RSSI acquisition, filtering, outlier removal and piecewise fitting;

[0086] Specifically, in the ranging process based on BLE signals, the mobile terminal can obtain the RSSI (Received Signal Strength Indicator) value through BLE broadcasting / scanning, and map the RSSI to distance according to the classical logarithmic distance path loss model. In order to reduce RSSI fluctuations caused by indoor multipath and transient interference, Gaussian statistical filtering can be applied to the RSSI sequence, and the mean µ and standard deviation σ can be calculated within the sliding time window. Samples located in the interval [µ-3σ, µ+3σ] are retained as valid data to eliminate abnormal jumps.

[0087] like Figure 2As shown, it illustrates the relationship between BLE RSSI and distance in three different environments. It can be seen that due to the frequency hopping mechanism of BLE, there is a "turning point" in the RSSI-distance relationship in actual use. That is, the short distance segment is more stable and the long distance segment is more prone to fluctuation. Therefore, the distance range can be divided into a short distance segment (e.g., 0-4m) and a long distance segment (e.g., 4-10m), and the least squares (LS) method can be used for curve fitting or model training to achieve segmented distance estimation.

[0088] (12) The ultrasonic ranging subunit can estimate the acoustic distance based on the two-way time of arrival (TOA) local time difference method, and use cross-correlation and dual threshold decision to locate the arrival time;

[0089] In the ultrasonic signal-based ranging process, a time-of-flight (ToF) ranging mechanism is employed. To address synchronization errors caused by inconsistencies in the local clocks of different devices, this embodiment of the invention uses a "two-way time of arrival (TOA) local time difference" method to calculate the corresponding distance. Specifically, acoustic signals are transmitted twice between two devices A and B, and their respective transmission / reception times are recorded. The clock deviation is eliminated by combining the local "time intervals," thereby calculating the distance.

[0090] For example, during implementation, distance It can be calculated using the following formula: ,in , The time when the sound wave is first transmitted at point A and received at point B. , The time when the second sound wave is transmitted at point B and received at point A. The speed of sound. Furthermore, to reduce the influence of temperature on the speed of sound, the system can utilize ambient temperature. (Celsius) Dynamic correction of the speed of sound, for example, using .

[0091] Regarding arrival time detection, the system calculates the cross-correlation between the received signal and the reference transmitted signal, which may result in noise peaks and multipath peaks. To enhance robustness, a dual-threshold processing mechanism is adopted: the first threshold is used to filter cross-correlation noise peaks; the second threshold is used to constrain the received energy, and if the energy is insufficient, it is determined to be an invalid arrival.

[0092] (13) IMU calculation unit, which can perform coordinate transformation and short-time inertial calculation based on accelerometer / gyroscope data and attitude quaternions, and provide high-frequency relative displacement / velocity;

[0093] In the short-time displacement estimation process based on IMU, the IMU estimation unit can acquire accelerometer and gyroscope data (50Hz) and complete the following processing steps:

[0094] (a) Attitude and coordinate transformation

[0095] IMU data is located in the device coordinate system, and the system obtains the attitude quaternion. ,in: For angle scalar parameters, All parameters are orthogonal rotation axis vectors; attitude quaternions are used to convert the acceleration / angular velocity obtained from accelerometers and gyroscopes to the geographic coordinate system, avoiding Euler angle gimbal lock issues.

[0096] (b) Short-time inertial integral

[0097] Based on the geographic coordinate system obtained by acceleration / angular velocity transformation, the velocity increment is obtained by integrating the acceleration within the ranging update interval of adjacent absolute distances. Then integrate to obtain the displacement increment. This enables the calculation of high-frequency relative displacement and velocity.

[0098] Because inertial integrals can drift, IMU calculations are only used as relative constraints for a short period of time (which can be limited to a predetermined time). The cumulative displacement and velocity are reset after each fusion correction to avoid long-term error accumulation.

[0099] (2) Real-time distance fusion unit

[0100] A real-time distance fusion unit is used to obtain a predicted distance by processing the low-frequency absolute distance and the high-frequency relative displacement and velocity through a first Kalman filter.

[0101] In the real-time distance fusion process, this real-time distance fusion unit is used to achieve real-time distance output by adopting the "BLE-ultrasound priority fusion strategy (BA-fusion) + dual Kalman filter structure (Dual-KF)"; wherein, BLE and ultrasound provide low-frequency absolute distance observation, and IMU provides high-frequency relative displacement / velocity input; through this unit, the alignment and fusion of absolute distance and relative displacement can be achieved, and the noise covariance can be adaptively updated;

[0102] Reference Figure 3 As shown, in the multi-source single-mode ranging stage of the real-time range fusion process, basic ranging and motion state acquisition are completed through BLE, acoustic sensors, and IMU, providing multi-source input for subsequent fusion. Thus, in the subsequent real-time range fusion stage, closed-loop optimization of range prediction and velocity correction can be achieved through two-stage Kalman filtering (KF). The two core sub-processes of range prediction and velocity correction can include:

[0103] (21) Distance prediction

[0104] Kalman filter prediction of distance involves taking absolute and relative distances as observed values ​​and inputting them into a first-stage Kalman filter (KF). The Kalman filter uses a prediction-update mechanism to suppress noise and perform optimal estimation of multi-source ranging data, predicting the optimal distance estimate after fusion (i.e., predicted distance). Simultaneously, the distance estimate is differentiated to derive the corresponding theoretical velocity value, providing a reference benchmark for subsequent velocity correction.

[0105] (22) Speed ​​correction

[0106] The Kalman filter correction (C. KF) for velocity involves inputting the raw velocity observations acquired by the IMU and the theoretical velocity value obtained by distance differentiation into a second-stage Kalman filter (KF). The Kalman filter uses the theoretical velocity as a reference to perform error correction on the raw velocity of the IMU, compensating for the drift error of the inertial sensor, and outputting a corrected high-precision velocity. The corrected velocity is simultaneously fed back to the first-stage Kalman filter, forming a closed-loop optimization, which further improves the accuracy and stability of distance estimation.

[0107] exist Figure 3 This system combines the absolute ranging accuracy of BLE / acoustics with the high-frequency relative motion tracking capability of IMU, and uses Kalman filtering to achieve noise suppression and error compensation for multi-source data. It solves the problems of poor robustness of single-mode ranging and accumulation of inertial sensor drift, and can realize high-precision and high-real-time distance estimation between distributed devices, providing reliable distance constraints for subsequent multi-device collaborative positioning.

[0108] Specifically, refer to Figure 3 As shown, the processing steps include BA fusion ranging, distance KF prediction, and velocity KF correction. The specific implementation methods of each process will be described in detail below.

[0109] (A) BA Fusion

[0110] In the implementation of the BLE-ultrasound complementary fusion strategy (BA-fusion), refer to Figure 4 As shown, it illustrates the success rate and error characteristics of absolute ranging modes of ultrasonic and Bluetooth Low Energy technologies at different distances. It can be seen that ultrasonic technology has a near 100% success rate and high accuracy at short distances (within 4m), but the success rate decreases significantly with increasing distance; BLE technology has a stable success rate within the 0-10m range, but the error is relatively large. Therefore, based on... Figure 4 In such cases, the following absolute distance measurement strategy can be adopted:

[0111] When only BLE distance measurement results are available, the distance measured by BLE is used as the observation result of the absolute distance;

[0112] When both BLE ranging and ultrasonic ranging are available, the distance measured by ultrasonic ranging shall be used as the observation result of absolute distance.

[0113] When ultrasound temporarily fails due to noise / obstruction in short-range scenarios, BLE is used as a fallback to avoid system failure or long-term IMU drift; for example, if no ultrasound ranging result is received within 3 seconds, the ultrasound is considered to have failed.

[0114] Before each ultrasonic ranging measurement, a ranging prompt / synchronization notification is sent to the other end via BLE to reduce false detections caused by ambient noise. At the same time, a BLE segmented fitting model is selected according to the current distance stage, that is, the distance range of BLE ranging is divided into several different intervals, and the true distance of each interval is calculated using an independent curve.

[0115] (B) Distance Kalman Filter (KF) prediction

[0116] In obtaining absolute distance observation (From BA-fusion) and IMU relative displacement input After obtaining the distance (from short-time integration), a one-dimensional Kalman filter (i.e., the first Kalman filter) can be used to perform state prediction and parameter correction on the distance, so as to output the current distance estimate. .

[0117] State prediction: , .

[0118] Parameter calibration: , , .

[0119] in, Let be the absolute distance at time k. This is the IMU relative displacement input at time k. For process noise covariance, To observe the noise covariance, This is the Kalman gain (weighting coefficient). For prior prediction of distance, The current distance estimate (i.e., the predicted distance obtained by the real-time distance fusion unit), that is , To estimate the covariance (corrected uncertainty) posteriorly, Prior estimate of covariance (uncertainty after prediction).

[0120] The performance of a Kalman filter largely depends on the appropriate selection of the process noise covariance Q and the measurement noise covariance R. However, due to the varying sensor sensitivities under different conditions and smartphone models, using fixed Q and R values ​​is often insufficient. To address this issue, an adaptive adjustment strategy for the noise covariance can be employed in Kalman filters used for dynamic state estimation, including: through innovative methods... Update the process noise covariance at time k. Through residuals Update the observation noise covariance at time k. And introduce a forgetting factor. (It can be, but is not limited to, 0.3) Let and Achieve smooth updates.

[0121] (C) KF correction for velocity

[0122] The second Kalman filter, KF, or velocity correction, uses the differential velocity estimated from the distance. As an observation, the velocity obtained by IMU integration As input, the velocity error is corrected by the second Kalman filter, and the corrected velocity is used as the initial velocity for the next short-time inertial integration, that is, the starting velocity used for the next IMU integration to calculate the position.

[0123] Figure 3 The dual Kalman filter in the IMU is only updated when BLE or ultrasonic absolute ranging results are received; within the update interval, the IMU can perform calculations at 50Hz to ensure a real-time response of <0.1s.

[0124] Reference Figure 5 As shown, the corresponding real-time distance fusion algorithm achieves high-precision and robust real-time distance output by using IMU inertial integration to calculate the relative distance and BLE / ultrasonic absolute distance measurement through closed-loop Kalman filtering (KF) correction. The corresponding implementation process is as follows:

[0125] The input and output of this algorithm are respectively

[0126] Input consists of absolute distance measurements from BLE / ultrasound signals, i.e., the physical distance between the two devices, and velocity and acceleration data from the IMU;

[0127] The output is the current distance predicted after fusion, and the final output is a high-precision real-time distance estimate.

[0128] Based on the above inputs and outputs, the core process of this algorithm includes two branches:

[0129] Branch 1, IMU inertial integration branch (relative distance calculation, lines 2-7), includes;

[0130] When IMU velocity and acceleration data are available, perform inertial dead reckoning to generate relative distances:

[0131] Velocity integral (line 3): The trapezoidal integral method is used to calculate the velocity increment, the acceleration at the previous and next time moments, and the sampling time interval to realize the conversion from acceleration to velocity;

[0132] Distance integral (line 4): Based on the current velocity $v$ and the velocity increment, calculate the displacement increment and complete the integration from velocity to relative displacement;

[0133] Status update (lines 5-6): Update the current velocity and accumulate the relative distance to obtain the relative distance calculated by the IMU, providing high-frequency inertial observations for subsequent fusion.

[0134] Branch 2, Absolute Distance Fusion and Closed-Loop Correction Branch (lines 8-13), includes:

[0135] When absolute distance measurements from BLE / ultrasound are available, perform Kalman filter fusion and error correction:

[0136] Distance prediction (line 9: Input the absolute distance measured by BLE / ultrasound and the relative distance calculated by IMU into the Kalman filter (KF), and output the best current distance estimate after fusion to suppress noise and error in single-mode ranging;

[0137] Velocity Differentiation (line 10): Performs a difference operation on the predicted distance to derive the theoretical velocity, where the predicted distance and time interval of the previous moment provide a reference for IMU velocity correction;

[0138] Error correction (line 11): Input the raw speed and theoretical speed of the IMU into the Kalman filter to complete the error correction of the IMU speed and compensate for the drift error of the inertial sensing.

[0139] Reference Reset (line 12): Reset the relative distance to 0, using the current fused absolute distance as the new benchmark to avoid drift accumulation of the inertial integral and form a closed-loop optimization.

[0140] In the aforementioned algorithm processing, the IMU provides high-frequency, continuous relative motion tracking, addressing the low refresh rate issue of BLE / ultrasonic ranging. BLE / ultrasonic provides an absolute distance reference, and Kalman filtering suppresses IMU drift errors. Furthermore, velocity differential correction further optimizes the inertial state, forming a complete closed-loop optimization chain to achieve continuous error compensation, avoid cumulative errors, and improve long-term ranging accuracy. Moreover, the high-frequency operation of the IMU inertial integral ensures real-time distance estimation, meeting the needs of dynamic scenarios. Correspondingly, multi-source data fusion and Kalman filtering effectively suppress environmental interference, multipath effects, and inertial drift, adapting to complex scenarios.

[0141] (ii) Joint Positioning Module

[0142] The joint localization module is used to reconstruct the relative position topology of nodes by using the predicted distance as a constraint on the edges.

[0143] This joint localization module reconstructs the topology based on the improved mass-spring energy minimization model, and selects the optimal output structure by combining multiple initial solution proposals and global error rate (GER).

[0144] In multi-terminal scenarios, each terminal is abstracted as a node in a graph, and the pairwise distances output by the ranging module serve as edge length constraints. The goal of the joint localization module is to reconstruct the relative position topology of the nodes (i.e.,...) even in the absence of angle information. Figure 1 (The location map in the image), and obtains the absolute coordinates when anchor points are available.

[0145] As shown in the figure, the corresponding joint positioning process can be illustrated, which may include:

[0146] (1) The original topology is input into the ranging module Range;

[0147] (2) The output of the ranging module is sent to the mass-spring optimization (also known as the mass-spring model) for processing. Specifically, the node positions can be iteratively adjusted to minimize the total energy of the system, make the distance between nodes fit the measured value, and punish unreasonable close-range folding. Finally, a stable and correct topology will naturally converge.

[0148] Specifically, the corresponding improved mass-spring energy minimization model includes the following steps:

[0149] The classic mass-spring model treats connected nodes as being linked by springs, thus the entire system tends towards a minimum energy state (minE); assuming nodes... Its connected nodes The measured distance between is , the actual distance under the current layout is , the spring coefficient is , then the node energy:

[0150] ;

[0151] The total energy of the system ; For the folding problem of "non-edge constraints" (that is, no effective distance is measured between two nodes or the distance is greater than the system range threshold, such as 10m), an edge potential energy penalty term penalty can be introduced, that is:

[0152] If the distance between two nodes in the predicted layout is less than the threshold but there is no edge in the original graph, the predicted distance will be added to the penalty to form: , this penalty term enhances the anti-folding ability and structural consistency of the topology.

[0153] (3) Perform corresponding multi-initial solution proposal and GER selection (Proposals and GER Selection) processing;

[0154] Since the gradient descent optimization is affected by the initial position and is prone to falling into local optima; to balance performance and complexity, groups of node initial positions can be randomly generated, and the particle-spring optimization is run respectively to obtain candidate topology proposals (that is, candidate topology solutions); then the global error rate (GER) is used to evaluate the consistency of the N candidate topologies with the ranging constraints (that is, the actual ranging values) determined by the ranging module, and the topology proposal with the smallest GER is selected as the output of joint positioning; for example, the corresponding GER can be defined as:

[0155] ;

[0156] Among them, is the total number of nodes to be located, is the measured distance between nodes i and j, is the calculated distance between nodes i and j in the optimized topology, , which is used to calculate the relative error between the theoretical distance and the measured distance between devices i and j, eliminates the influence of different distance scales, and realizes the normalization of errors.

[0157] In the above formula, all device pairs with i < j are traversed, the relative error of each device pair is calculated, the squares of all errors are summed, and then the square root of the sum is taken to obtain the root mean square of the relative errors of all device pairs, quantifying the global error accumulation level of the candidate topology, so as to amplify the influence of large errors and avoid small errors covering large errors; further, it also includes the total logarithm normalization term It can divide the global root mean square error by the total number of constraints to obtain the average relative error level, eliminating the influence of the number of devices n on the GER value, enabling GER to be compared horizontally in scenarios with different numbers of nodes, and ensuring the consistency of the evaluation standard.

[0158] (4) Perform the corresponding positioning process.

[0159] Since the smaller the GER, the smaller the total deviation between the pairwise distances of all nodes and the actual distance measurement, the positioning process can be carried out based on the selected topology proposal with the smallest GER.

[0160] Furthermore, in this embodiment of the invention, anchor-assisted absolute positioning processing can also be performed, namely:

[0161] When there are three or more devices with known locations (anchor points), their coordinates can be fixed to known values, and only the positions of the remaining nodes need to be optimized. Since three points determine the plane, the anchor points can eliminate the degrees of freedom such as overall translation / rotation / mirror, thus obtaining the absolute coordinates of the remaining nodes.

[0162] Through the implementation of the above embodiments of the present invention, the corresponding ranging processing can be achieved entirely based on existing commercial equipment and corresponding sensing and communication capabilities (such as BLE, ultrasound, IMU, etc.). The entire process requires no additional anchor points or hardware modifications and is easily implemented in real multi-device ecosystems. The embodiments of the present invention also propose a processing method that fuses BLE, sound, and IMU signals to measure the relative distance between mobile devices; and through an improved dual Kalman filter multimodal signal fusion scheme, it can effectively solve the synchronization problem between heterogeneous sensor signals, exhibiting superior performance, robustness, and generalization compared to single-modal systems. Furthermore, the embodiments of the present invention can be extended to multi-device joint localization and topology reconstruction, not only outputting pairwise ranging but also generating relative position maps and performing joint localization in scenarios where multiple devices move freely. By combining geometric constraints / optimization strategies, it becomes more robust to ranging error accumulation and structural ambiguity, thereby obtaining more stable overall spatial relationship inference results.

[0163] For the embodiments of the present invention, corresponding practical experiments were also conducted. Specifically, the technical solutions provided by the embodiments of the present invention were deployed on corresponding test mobile phones, and the experimental scenarios were set up as follows: Figure 7 As shown, experiments were conducted under different conditions.

[0164] based on Figure 7In this scenario, to verify the fusion ranging performance under stationary conditions, two test phones were placed in the same planar environment. A set of test points with a distance range of 0.8m to 9.6m and an interval of 0.8m were selected. 100 ranging results were collected for each test point, and this process was repeated three times to form a total of 300 measurement samples for each distance point. (Refer to...) Figure 8 As shown, it displays the average of 300 measurements taken at each test point in a static state.

[0165] The experimental results above show that:

[0166] (1) The ultrasonic ranging method can work stably within 6m with an average absolute error of about 1.9cm. When the distance exceeds 6m, although the ranging is occasionally successful, the failure probability is high. Therefore, this embodiment does not consider it as the stable working range.

[0167] (2) The distance measurement method based on BLE RSSI can obtain distance output within a range of 10m, but the distance measurement accuracy is low. The overall MAE (mean absolute error) is about 99cm, and there is an error fluctuation caused by the RSSI characteristic transition around about 4m.

[0168] (3) After adopting the fusion ranging algorithm provided in the embodiment of the present invention, the overall MAE is about 11cm within a range of 10m, and the MAE is about 2cm within a range of 4m and about 18cm within a range of 4–10m in the two typical application scenarios of short distance and long distance.

[0169] Therefore, the fusion strategy of this invention can achieve both "short-range high precision" and "long-range availability" without adding extra hardware, and effectively expand the usable range of ultrasound, improve the accuracy of BLE ranging, and meet the real-time and stability requirements of device interaction.

[0170] In addition, to verify the fusion ranging performance under terminal motion conditions, at least one test phone was made to move during the ranging process and continuously output the fusion ranging results. The corresponding statistical analysis results of the errors are as follows: Figure 9 As shown in the experimental results based on the corresponding motion states, it can be seen that:

[0171] (1) At the two marker points where CDF (cumulative distribution function) is 0.5 and CDF is 0.8, the results shown by the embodiments of the present invention are better than the other four configurations. The corresponding results strongly prove the effectiveness of the multimodal fusion scheme in the embodiments of the present invention.

[0172] (2) Although the configuration of ultrasonic + Bluetooth Low Energy + Inertial Measurement Unit (Ultrasound + BLE + IMU) is slightly inferior to that of ultrasonic + Inertial Measurement Unit (Ultrasound + IMU) at the CDF0.95 point, it compensates for a considerable distance in ranging, that is:

[0173] The combination of ultrasound, Bluetooth Low Energy, and an inertial measurement unit (IMU) remains effective within a 10-meter range, while the combination of ultrasound and IMU begins to fail beyond 5 meters. This highlights the trade-off between accuracy and range. The experimental results emphasize the superior comprehensive capabilities of the multimodal fusion scheme proposed in this invention, achieving high accuracy over short distances and demonstrating feasibility over long distances as well.

[0174] The above results demonstrate that the multimodal fusion ranging scheme provided by the embodiments of the present invention can still maintain a decimeter-level error level even when there are factors such as terminal movement and signal instability, and can meet the needs of dynamic interactive scenarios for continuous ranging and real-time response.

[0175] Finally, to verify the applicability and scalability of the joint positioning scheme in multi-terminal scenarios, considering that common indoor environments (such as conference rooms and offices) are typically smaller than 20m × 20m, four environmental ranges were set: 5m × 5m, 10m × 10m, 15m × 15m, and 20m × 20m; and the number of nodes was set to 5, 10, 15, and 20 respectively. Since the implementation cost of deploying real terminals and conducting synchronous testing increases significantly with the number of nodes, this experimental embodiment uses a simulation method for batch evaluation: the experiment was repeated 100 times for each "environmental range × number of nodes" configuration, and the results were averaged.

[0176] The corresponding joint positioning performance is evaluated using the following two metrics:

[0177] (1) Mean absolute error (MAE) of the existing edge (i.e., the edge with a distance measurement);

[0178] (2) Global Edge Recovery Rate (GER, used to measure the correctness of the overall graph structure recovery).

[0179] The corresponding experimental results are referenced. Figure 10 and Figure 11As shown in the figure, the joint positioning experiment results indicate that in a typical 6m×6m scenario with joint positioning using 5 mobile phones, the MAE is approximately 12.50cm and the GER is 0.00. Furthermore, when the number of nodes is less than 10, the GER can be less than 0.01 and the MAE can be less than 15cm. Therefore, the joint positioning scheme provided by this embodiment of the invention can construct a relative position map and achieve stable topology reconstruction under conditions of free distribution of multiple terminals, exhibiting good positioning accuracy and structural restoration reliability.

[0180] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0181] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. A device ranging and positioning system based on multimodal fusion, characterized in that, It includes a ranging module and a joint positioning module. The ranging module includes a single-mode ranging unit and a real-time distance fusion unit, wherein: The single-mode ranging unit includes: a low-power Bluetooth BLE ranging subunit and an ultrasonic ranging subunit for low-frequency absolute distance measurement, and an inertial measurement unit (IMU) subunit for high-frequency relative displacement and velocity measurement. A real-time distance fusion unit is used to obtain a predicted distance by processing the low-frequency absolute distance and the high-frequency relative displacement and velocity through a first Kalman filter. The joint localization module is used to reconstruct the relative position topology of nodes by using the predicted distance as a constraint on the edges.

2. The system according to claim 1, characterized in that, The distance estimation process performed by the BLE ranging subunit includes: The Received Signal Strength Indicator (RSSI) value is obtained and mapped to distance according to the classical logarithmic distance path loss model. Then, Gaussian statistical filtering is applied to the RSSI sequence, and the mean µ and standard deviation σ are calculated within a sliding time window. Samples located in the interval [µ-3σ, µ+3σ] are retained as valid data for distance estimation. During the ranging process, the distance range is divided into short-range segments and long-range segments according to predetermined rules, and least squares (LS) is used for curve fitting or model training to achieve segmented distance estimation.

3. The system according to claim 1, characterized in that, The distance estimation process performed by the ultrasonic ranging subunit is implemented using a Time-of-Flight (ToF) ranging mechanism and includes: distance The calculation formula is: ,in , The time when the sound wave is first transmitted at point A and received at point B. , The time when the second sound wave is transmitted at point B and received at point A. The speed of sound; and the speed of sound is either a direct value or a value adjusted for temperature; the temperature-adjusted speed of sound value is... , Ambient temperature; A dual-threshold processing mechanism is adopted: the first threshold is used to filter cross-correlation noise peaks; the second threshold is used to constrain the received energy, and if the energy is insufficient, it is determined to be an invalid arrival.

4. The system according to claim 1, characterized in that, The process by which the IMU subunit determines relative displacement and velocity includes: Get the pose quaternion ,in: For angle scalar parameters, All parameters are orthogonal rotation axis vectors; attitude quaternions are used to convert the acceleration and angular velocity obtained from accelerometers and gyroscopes to geographic coordinates. Based on the geographic coordinate system, within the distance update interval of adjacent absolute distances, the acceleration is integrated to obtain the velocity increment, and then integrated again to obtain the displacement increment, thereby realizing the calculation of the corresponding high-frequency relative displacement and velocity.

5. The system according to any one of claims 1 to 4, characterized in that, The absolute distance ranging strategies implemented by the BLE ranging subunit and the ultrasonic ranging subunit include: When only BLE ranging results are available, the distance measured by BLE is used as the absolute distance; When both BLE ranging and ultrasonic ranging are available, the distance measured by ultrasonic ranging shall be used as the absolute distance. In the intended short-range scenario, if ultrasonic ranging temporarily fails, BLE will be used as a fallback. Before each ultrasonic ranging operation, a ranging notification is sent to the other end via BLE, and a BLE segmented fitting model is selected based on the current distance stage.

6. The system according to claim 5, characterized in that, The real-time distance fusion unit further includes performing differential processing on the predicted distance to obtain the corresponding theoretical velocity value, and performing an error correction operation on the original velocity of the IMU via a second Kalman filter for the theoretical velocity value, and the corrected velocity of the IMU is used as the input of the first Kalman filter.

7. The system according to claim 5, characterized in that, The processing performed by the first Kalman filter included in the real-time distance fusion unit includes: The execution state prediction process is as follows: , ; Perform the parameter calibration process, namely: , , ; in, Let be the absolute distance at time k. This is the IMU relative displacement input at time k. For process noise covariance, To observe the noise covariance, For Kalman gain, For prior prediction of distance, For the current distance estimate, To estimate the covariance posteriorly, for Prior estimation of covariance The predicted distance is obtained through the above process. ; Furthermore, during the dynamic state estimation process of the first Kalman filter, an adaptive adjustment strategy for the noise covariance is adopted. This adjustment strategy includes: using innovation quantity... Update the process noise covariance at time k. Through residuals Update the observation noise covariance at time k. And introduce a forgetting factor. Control the smooth update of Qi.

8. The system according to claim 5, characterized in that, The processes performed by the joint positioning module include: Randomly generated The initial positions of the group nodes are obtained by running the mass-spring energy minimization model. N candidate topology proposals are then used; the consistency between the N candidate topologies and the ranging constraints is evaluated using the Global Error Rate (GER), and the topology proposal with the smallest GER is selected as the output of the joint localization.

9. The system according to claim 8, characterized in that, In the processing performed by the joint positioning module: The processing steps of the running mass-spring energy minimization model include: In the mass-spring energy minimization model, connected nodes are considered to be connected by springs, and the entire system tends to the minimum energy state minE; assuming nodes... Its connected nodes The distance measured between them is The actual distance under the current layout is The spring constant is Then the node energy is : ； Total system energy Where, for cases where no effective distance is measured between two nodes or the distance exceeds the system's range threshold, a penalty term for edge potential energy is introduced, namely: If the distance between two nodes in the predicted layout is less than a threshold but the original graph has no edges, then the predicted distance is added as a penalty, resulting in: ； The topology between nodes is obtained through the above process; In the process of evaluating the consistency between the N candidate topologies and the ranging constraints determined by the ranging module using GER, the definition of GER includes: ,in It is used to calculate the relative error between the theoretical distance and the measured distance between devices i and j. The total number of nodes to be located. This represents the measured distance between nodes i and j. This is to calculate the distance between nodes i and j in the optimized topology.

10. The system according to claim 8, characterized in that, The joint positioning module also includes anchor-assisted absolute positioning processing, which includes: When there are three or more devices with known locations that can be used as anchor points, their coordinates are fixed to known values, and the absolute coordinates of the remaining nodes are obtained based on the known values.

Images