Multimodal fusion indoor positioning method

Abstract

The present application relates to the field of indoor positioning, and specifically discloses a multi-modal fusion indoor positioning method. First, the positioning base station system sends a positioning audio signal, and the positioned communication device receives and records the audio signals from different base stations and their corresponding timestamps, and obtains TDOA position detection data through the TDOA algorithm. Next, the positioned communication device sends a predefined Bluetooth signal to the positioning base station system. After receiving the signal, each base station in the positioning base station system calculates the AOA position detection data of the communication device through the Bluetooth AOA technology. Then, by fusing the acoustic signal TDOA and the Bluetooth AOA position detection data, more accurate location information of the communication device is output. This method effectively overcomes the problem of limited accuracy of a single positioning method by fusing TDOA and AOA positioning observation information, and can better cope with challenges such as multipath effects and non-line-of-sight propagation, providing more reliable positioning services in two-dimensional dynamic scenes.

Description

Technical Field

[0001] The present application relates to the field of indoor positioning technology, and more specifically, to a multi-modal fusion indoor positioning method. Background Art

[0002] Integrated indoor and outdoor high-precision positioning is one of the important applications of current smartphones. At present, outdoor positioning has widely relied on the Global Navigation Satellite System (GNSS) to achieve high-precision positioning and meet various location-based service needs. However, since GNSS signals are difficult to penetrate indoor spaces, non-line-of-sight (NLOS) transmission and multipath effects occur, which dramatically reduces the accuracy of indoor positioning.

[0003] In recent years, a variety of indoor positioning technologies and solutions have emerged, including Wi-Fi, Bluetooth, ultra-wideband (UWB), acoustic signals, ultrasound, 5G positioning, inertial navigation systems, optical positioning, etc. At the same time, various source positioning methods based on different measurement methods have been proposed, such as received signal strength (RSS), time of arrival (TOA), time difference of arrival (TDOA), etc. However, these single positioning technologies still face defects such as poor compatibility, high energy consumption, high deployment cost, limited coverage, susceptibility to multipath effects and signal interference, and inconsistent positioning accuracy when applied on mobile phones. It is difficult to achieve high-precision, low-cost and widely applicable positioning services in all scenarios, and further technological innovation and integration solutions are needed to overcome these limitations.

[0004] Therefore, a multimodal fusion indoor positioning method is expected. Summary of the invention

[0005] In order to solve the above technical problems, the present application is proposed. An embodiment of the present application provides a multi-modal fusion indoor positioning method. First, the positioning base station system sends a positioning audio signal, and the positioned communication device receives and records the audio signals from different base stations and their corresponding timestamps, and obtains TDOA position detection data through the TDOA algorithm. Then, the positioned communication device sends a predefined Bluetooth signal to the positioning base station system. After receiving the signal, each base station in the positioning base station system calculates the AOA position detection data of the communication device through the Bluetooth AOA technology. Then, by fusing the acoustic signal TDOA and the Bluetooth AOA position detection data, more accurate location information of the communication device is output. This method effectively overcomes the problem of limited accuracy of a single positioning method by fusing TDOA and AOA positioning observation information, and can better cope with challenges such as multipath effects and non-line-of-sight propagation, providing more reliable positioning services in two-dimensional dynamic scenes.

[0006] Accordingly, according to one aspect of the present application, a multi-modal fusion indoor positioning method is provided, which includes:

[0007] The positioning base station system sends a positioning audio signal to the located communication device, and the positioning base station system includes a main base station and three slave base stations;

[0008] The located communication device receives and records the arrival timestamp of the positioning audio signal from the positioning base station system to obtain a set of signal arrival timestamps;

[0009] Based on the set of signal arrival timestamps and the location data of each base station in the positioning base station system, the TDOA location detection data of the positioning communication device is calculated by a TDOA algorithm;

[0010] Sending a Bluetooth signal to the positioning base station system through the located communication device;

[0011] Each base station in the positioning base station system captures the Bluetooth signal to obtain a set of Bluetooth IQ signals;

[0012] Based on the set of Bluetooth IQ signals, the AOA position detection data of the positioning communication device is calculated by the AOA algorithm;

[0013] The AOA position detection data of the positioning communication device and the TDOA position detection data of the positioning communication device are integrated to obtain the positioning data of the positioned communication device.

[0014] In the above-mentioned multimodal fusion indoor positioning method, based on the set of Bluetooth IQ signals, the AOA position detection data of the positioning communication device is calculated through the AOA algorithm, including: using the AOA algorithm to process each Bluetooth IQ signal in the set of Bluetooth IQ signals to obtain a set of angles of arrival at the base station; sending the set of angles of arrival at the base station to the positioning server for positioning data solution to obtain the AOA position detection data of the positioning communication device.

[0015] In the above-mentioned multimodal fusion indoor positioning method, the AOA position detection data of the positioning communication device and the TDOA position detection data of the positioning communication device are fused to obtain the positioning data of the positioned communication device, including: inputting the AOA position detection data of the positioning communication device and the TDOA position detection data of the positioning communication device into an improved unscented Kalman filtering algorithm to obtain the positioning data of the positioned communication device.

[0016] In the above-mentioned multimodal fusion indoor positioning method, the AOA position detection data of the positioning communication device and the TDOA position detection data of the positioning communication device are input into the improved unscented Kalman filtering algorithm to obtain the positioning data of the positioned communication device, including: setting an initial state estimation vector and an initial covariance matrix; performing an unscented transformation on the initial state estimation vector and the initial covariance matrix to generate Sigma points; setting a motion state transfer function under a two-dimensional dynamic scene; performing prediction based on the Sigma points and in combination with the motion state transfer function to obtain predicted observation values; determining AOA error observation information and TDOA error observation information based on the predicted observation values ​​and the AOA position detection data of the positioning communication device and the TDOA position detection data of the positioning communication device; and determining the positioning data of the positioned communication device based on the AOA error observation information and the TDOA error observation information.

[0017] In the above-mentioned multimodal fusion indoor positioning method, the AOA position detection data of the positioning communication device and the TDOA position detection data of the positioning communication device are fused to obtain the positioning data of the positioned communication device, including: embedding and encoding the AOA position detection data of the positioning communication device and the TDOA position detection data of the positioning communication device to obtain an AOA position detection embedded coding vector and a TDOA position detection embedded coding vector; guiding the interaction of the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector based on implicit joint features to obtain a position detection significant joint interaction coding vector; and inputting the position detection significant joint interaction coding vector into a decoder-based position parsing module to obtain the positioning data of the positioned communication device.

[0018] In the above-mentioned multimodal fusion indoor positioning method, the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector are subjected to guided interaction based on implicit joint features to obtain a position detection significant joint interaction coding vector, including: extracting the implicit joint features between the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector as a conditional feature vector; based on the content contribution of the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector relative to the conditional feature vector, feature modulating the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector to obtain a modulated AOA position detection embedded coding vector and a modulated TDOA position detection embedded coding vector; cross-domain interactive fusion based on the attention mechanism is performed on the modulated AOA position detection embedded coding vector, the modulated TDOA position detection embedded coding vector and the conditional feature vector to obtain the position detection significant joint interaction coding vector.

[0019] In the above-mentioned multimodal fusion indoor positioning method, the implicit joint features between the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector are extracted as conditional feature vectors, including: inputting the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector into an implicit joint feature capture network to obtain an AOA-TDOA position detection implicit joint feature vector; performing feature activation based on a Sigmoid function on the AOA-TDOA position detection implicit joint feature vector to obtain the conditional feature vector.

[0020] In the above-mentioned multimodal fusion indoor positioning method, based on the content contribution of the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector relative to the conditional feature vector, the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector are feature modulated to obtain the modulated AOA position detection embedded coding vector and the modulated TDOA position detection embedded coding vector, including: calculating the first content contribution of the AOA position detection embedded coding vector relative to the conditional feature vector; calculating the second content contribution of the TDOA position detection embedded coding vector to the conditional feature vector; normalizing the first content contribution and the second content contribution, and using the normalized first content contribution and the second content contribution to weighted modulate the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector to obtain the modulated AOA position detection embedded coding vector and the modulated TDOA position detection embedded coding vector.

[0021] In the above-mentioned multimodal fusion indoor positioning method, the modulated AOA position detection embedded coding vector, the modulated TDOA position detection embedded coding vector and the conditional feature vector are subjected to cross-domain interactive fusion based on the attention mechanism to obtain the position detection significant joint interaction coding vector, including: using the modulated AOA position detection embedded coding vector as the query vector, the modulated TDOA position detection embedded coding vector as the key vector and the conditional feature vector as the value vector, and inputting the modulated AOA position detection embedded coding vector, the modulated TDOA position detection embedded coding vector and the conditional feature vector into a feature-to-feature significant guided interaction module based on a converter structure to obtain the position detection significant joint interaction coding vector.

[0022] Compared with the prior art, the multimodal fusion indoor positioning method provided by the present application is as follows: first, the positioning base station system sends a positioning audio signal, and the positioned communication device receives and records the audio signals from different base stations and their corresponding timestamps, and obtains TDOA position detection data through the TDOA algorithm. Next, the positioned communication device sends a predefined Bluetooth signal to the positioning base station system, and each base station in the positioning base station system calculates the AOA position detection data of the communication device through the Bluetooth AOA technology after receiving the signal. Then, by fusing the acoustic signal TDOA and the Bluetooth AOA position detection data, more accurate location information of the communication device can be output. This method effectively overcomes the problem of limited accuracy of a single positioning method by fusing TDOA and AOA positioning observation information, and can better cope with challenges such as multipath effects and non-line-of-sight propagation, providing more reliable positioning services in two-dimensional dynamic scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0024] Figure 1 The figure is a flowchart of a multi-modal fusion indoor positioning method according to an embodiment of the present application.

[0025] Figure 2 Schematic diagram of the architecture of a multi-modal fusion indoor positioning method according to an embodiment of the present application.

[0026] Figure 3 Schematic diagram of TDOA positioning of acoustic signals in a multi-modal fusion indoor positioning method according to an embodiment of the present application.

[0027] Figure 4 Schematic diagram of Bluetooth AOA positioning in the multi-modal fusion indoor positioning method according to an embodiment of the present application.

[0028] Figure 5 The present invention is a flowchart of fusing the AOA position detection data of the positioning communication device and the TDOA position detection data of the positioning communication device in the multimodal fusion indoor positioning method according to an embodiment of the present application to obtain the positioning data of the positioned communication device.

[0029] Figure 6 The present invention is a flowchart of performing guided interaction based on implicit joint features on the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector in the multimodal fusion indoor positioning method according to an embodiment of the present application to obtain a position detection significant joint interaction coding vector. DETAILED DESCRIPTION

[0030] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0031] Figure 1 The figure is a flowchart of a multi-modal fusion indoor positioning method according to an embodiment of the present application. Figure 2 FIG. 1 is a schematic diagram of the architecture of a multi-modal fusion indoor positioning method according to an embodiment of the present application. Figure 1 and Figure 2 As shown, according to the multimodal fusion indoor positioning method of the embodiment of the present application, the method includes the following steps: S1, a positioning base station system sends a positioning audio signal to a positioned communication device, the positioning base station system includes a main base station and three slave base stations; S2, the positioned communication device receives and records the arrival timestamp of the positioning audio signal from the positioning base station system to obtain a set of signal arrival timestamps; S3, based on the set of signal arrival timestamps and the position data of each base station in the positioning base station system, the TDOA position detection data of the positioning communication device is calculated by the TDOA algorithm; S4, a Bluetooth signal is sent to the positioning base station system by the positioned communication device; S5, each base station in the positioning base station system captures the Bluetooth signal to obtain a set of Bluetooth IQ signals; S6, based on the set of Bluetooth IQ signals, the AOA position detection data of the positioning communication device is calculated by the AOA algorithm; S7, the AOA position detection data of the positioning communication device and the TDOA position detection data of the positioning communication device are fused to obtain the positioning data of the positioned communication device.

[0032] In the above-mentioned multimodal fusion indoor positioning method, in step S1, the positioning base station system sends a positioning audio signal to the located communication device, and the positioning base station system includes a main base station and three slave base stations. In the technical solution of the present application, the positioning base station system consists of a main station and three slave base stations, each base station is equipped with a speaker and a Bluetooth antenna array, wherein the speaker is used to send an audio signal of a specific frequency. Through the synchronous delay design, the base station can coordinate the sending of signals within a predetermined time interval to ensure the synchronous transmission of the audio signal, thereby achieving an accurate positioning effect.

[0033] In the above-mentioned multimodal fusion indoor positioning method, in step S2, the located communication device receives and records the arrival timestamp of the positioning audio signal from the positioning base station system to obtain a set of signal arrival timestamps. It should be understood that the TDOA algorithm is a technology that determines the location of the signal source based on measuring the time difference between the arrival of the signal at different receiving points. In the present application, when multiple base stations simultaneously send positioning audio signals to the located communication device, the time when the signal arrives at the located device will be different due to the different distances between each base station and the located device. By recording the arrival timestamps of each signal, the located device can use the time difference information of the signal arrival to calculate the relative position of the located device through the TDOA algorithm.

[0034] In the above multi-modal fusion indoor positioning method, the step S3 is based on the set of signal arrival timestamps and the position data of each base station in the positioning base station system, and the TDOA algorithm is used to calculate the TDOA position detection data of the positioning communication device. Specifically, after the positioned communication device receives the positioning audio signal, it calculates at least any three valid TDOA ranging value sets in combination with the predefined synchronization delay and the calibrated anchor point position information, so as to calculate the TDOA position detection data of the positioning communication device in the absence of clock synchronization.

[0035] by Figure 3 Taking the acoustic signal TDOA positioning model shown in the figure as an example, the coordinates of the four base station positions are , ,in The first base station is the master base station and the others are slave base stations.

[0036] Then we can get,

[0037] (1)

[0038] in, is the position of the target to be located, To the goal The distance between base stations, To the goal The difference between the distance from the base station and the distance from the target to the main base station, is the speed of sound, is the delay of receiving signals from two base stations; and Substitute into Then combine similar terms and simplify to get:

[0039] (2)

[0040] in, The target to be located is The difference between the distance to the first base station and the distance from the target to the main base station.

[0041] (3)

[0042] Formula (2) can be expressed in matrix form:

[0043] (4)

[0044] in,

[0045] (5)

[0046] (6)

[0047] (7)

[0048] The above-mentioned TDOA position detection data based on the time difference of arrival of acoustic signals can be solved by numerical optimization methods.

[0049] In the above-mentioned multimodal fusion indoor positioning method, the step S4 sends a Bluetooth signal to the positioning base station system through the located communication device. It should be understood that, considering the actual use process, due to the influence of multipath effect, the difference between the distance from the target to the slave base station and the distance from the target to the main base station in the acoustic signal TDOA positioning may be inaccurate, resulting in an increase in positioning error. At the same time, the maximum likelihood estimation commonly used in the traditional full set TDOA algorithm is prone to the problem of too long calculation time in practice and cannot meet the real-time positioning. Therefore, in order to reduce the amount of calculation brought by the use of full set TDOA and improve the real-time position calculation, the present application is further combined with the Bluetooth signal AOA positioning method, by using the built-in Bluetooth module of the located communication device to actively broadcast and send predefined Bluetooth signals to provide additional location information and improve positioning accuracy and real-time performance.

[0050] In the above-mentioned multimodal fusion indoor positioning method, in step S5, each base station in the positioning base station system captures the Bluetooth signal to obtain a set of Bluetooth IQ signals. It should be understood that Bluetooth signals usually carry specific IQ identifiers to ensure that they can be accurately identified and processed. Therefore, Bluetooth IQ signals (i.e., I / Q signals, also called complex signals) can be used for positioning. The I / Q signal consists of two parts: an in-phase component (In-phase, I) and an orthogonal component (Quadrature, Q), which can accurately describe the phase and amplitude information of the signal. The located communication device can communicate with the positioning base station in real time by sending Bluetooth signals, transmitting location information or receiving instructions.

[0051] In the above-mentioned multimodal fusion indoor positioning method, the step S6 is based on the set of Bluetooth IQ signals and uses the AOA algorithm to calculate the AOA position detection data of the positioning communication device. In a specific example of the present application, the step S6 includes: using the AOA algorithm to process each Bluetooth IQ signal in the set of Bluetooth IQ signals to obtain a set of angles of arrival at the base station; sending the set of angles of arrival at the base station to the positioning server for positioning data solution to obtain the AOA position detection data of the positioning communication device.

[0052] It should be understood that each base station is configured with a Bluetooth antenna array for receiving Bluetooth signals sent from the positioning communication device. After receiving the Bluetooth signal, the base station first uses the AOA algorithm to perform AOA-based angle measurement on each Bluetooth IQ signal in the set of Bluetooth IQ signals, such as Figure 4 Taking the Bluetooth AOA positioning model as an example, the deployment of the Bluetooth AOA base station is set to be consistent with the deployment position of the signal base station, so the coordinates of the four AOA base station positions are also ,in is the position of the target to be located.

[0053] Based on this, the angle of arrival The estimation formula is:

[0054] (8)

[0055] in, To reach the The angle of the base station, is the number of antennas contained in each antenna array, is the distance between antennas, is the wavelength of the signal, To measure the phase difference of each antenna, the partial derivative of equation (8) is obtained:

[0056] (9)

[0057] Let equation (9) equal to 0 to minimize the angle estimation error, and the angle can be solved by numerical optimization method. , thereby obtaining the set of angles of arrival at the base station and obtaining a rough position estimated by Bluetooth AOA.

[0058] Based on the set of arrival angles at the base station, a closed-form full-set TDOA algorithm is designed. Since it is a closed-form solution, the solution can be obtained in a very short time. At the same time, because there is a rough initial position, there is no local optimal value problem. Theoretical analysis proves that the performance of the algorithm is the best in the sense of CRLB when the noise is small.

[0059] The base station sends the resolved set of arrival angles to the positioning server for positioning data resolution, and then the server transmits the obtained AOA position detection data back to the located communication device for further fusion processing. The calculation process of AOA position detection data is as follows:

[0060] , (10)

[0061] Solving the above four equations together can get the position of the signal source .

[0062] In the above-mentioned multimodal fusion indoor positioning method, the step S7 fuses the AOA position detection data of the positioning communication device and the TDOA position detection data of the positioning communication device to obtain the positioning data of the positioned communication device. It should be understood that under the demand for high-precision two-dimensional positioning, it is often necessary to add multiple base stations to collaborate to optimize angle estimation, but this will greatly increase the cost and deployment difficulty. To this end, the present application further fuses TDOA and AOA observation information based on an improved unscented Kalman filter (UKF) algorithm to achieve more accurate position estimation. The improved unscented Kalman filter algorithm can effectively handle nonlinear problems in TDOA and AOA data fusion by considering nonlinear system dynamics and measurement models.

[0063] In a specific example of the present application, the AOA position detection data of the positioning communication device and the TDOA position detection data of the positioning communication device are input into the improved unscented Kalman filter (UKF) algorithm, and the fusion positioning observation is further optimized through the weighted average strategy to achieve information fusion, prediction and update, and finally output the positioning data of the positioned communication device. The details are as follows:

[0064] First, set the initial values ​​of UKF, namely the initial state estimation vector and the initial covariance matrix:

[0065] (11)

[0066] (12)

[0067] in, is the initial state The estimated vector contains the two-dimensional coordinates and angle information of the initial node to be located. is the initial covariance matrix of the node to be located.

[0068] Next, an unscented transform (UT) is performed on the initial state estimation vector and the initial covariance matrix to generate Sigma points:

[0069] (13)

[0070] The corresponding weight used to calculate the mean of the predicted state at the initial moment , used to calculate the weight of the predicted state covariance at the initial moment , used to calculate the The weight of the mean of the predicted state vector of Sigma points And for calculating the The weight of the predicted state covariance of the Sigma points The formula is:

[0071] ,

[0072] (14)

[0073] (15)

[0074] in, is the dimension of the state vector, and the parameter , and Used to adjust the distribution and weight of Sigma points, set here , , .

[0075] Secondly, set the motion state transfer function in the two-dimensional dynamic scene Make predictions:

[0076] (16)

[0077] (17)

[0078] (18)

[0079] (19)

[0080] in, is the control input, and the Sigma point passes through the state transfer function Propagate, get at time step Based on a given time step Information generated Sigma Points The predicted state mean and the predicted state covariance ; is the state covariance.

[0081] Then, the AOA error observation information and the TDOA error observation information are determined based on the predicted observation value and the AOA position detection data of the positioning communication device and the TDOA position detection data of the positioning communication device:

[0082]

[0083] (20)

[0084] in, , , are the ground truth coordinates, the position estimation coordinates based on the TDOA method, and the position estimation coordinates based on the AOA method, respectively. and It represents the error observation information of position estimation under the two methods.

[0085] Furthermore, based on the AOA error observation information and the TDOA error observation information, the positioning data of the located communication device is determined by weighted average calculation, wherein the weights corresponding to the error observation information of the AOA and TDOA methods are respectively:

[0086] , (twenty one)

[0087] Based on this, the weighted average position after fusion for:

[0088] (twenty two)

[0089] The method with the smallest error will be selected To update:

[0090] (twenty three)

[0091] (twenty four)

[0092] (25)

[0093] in, refers to the time step Given a time step In the case of information, The observation vector obtained by predicting Sigma points, is the observation function containing the positioning coordinates;

[0094] Calculate the mean of the predicted observations The covariance of and observation noise is:

[0095] (26)

[0096] (27)

[0097] in, is the observed noise covariance; then the cross covariance matrix between the state vector and the observation vector for:

[0098] (28)

[0099] Its Kalman gain Then:

[0100] (29)

[0101] Finally, the updated state vector is obtained and covariance :

[0102] (30)

[0103] (31)

[0104] According to the iterative process described above, the proposed algorithm can effectively handle the positioning method based on TDOA or AOA positioning observation information with strong nonlinear characteristics, thereby achieving accurate positioning in two-dimensional dynamic scenes.

[0105] In particular, considering that the performance of UKF is affected by the initial state estimation and largely depends on the selection of Sigma points and the corresponding weight distribution, inappropriate parameter selection may lead to estimation bias or filter divergence, and poor adaptability to unknown or changing environments. In this regard, in another embodiment of the present application, an optimized data fusion method is proposed, which uses a deep learning-based data analysis technology to encode the AOA position detection data and TDOA position detection data of the positioning communication device, and by mining the implicit association between the AOA position detection data and the TDOA position detection data, the two are significantly guided to fuse, so as to obtain the position detection features of the fused multimodal information, thereby realizing the accurate position analysis of the located communication device. In this way, the data fusion effect in a complex environment can be effectively improved, thereby improving the positioning accuracy and the robustness of the system.

[0106] Figure 5 The present invention is a flowchart of fusing the AOA position detection data of the positioning communication device and the TDOA position detection data of the positioning communication device in the multi-modal fusion indoor positioning method according to an embodiment of the present application to obtain the positioning data of the positioned communication device. Figure 5 As shown, the step S7 includes: S71, embedding and encoding the AOA position detection data of the positioning communication device and the TDOA position detection data of the positioning communication device to obtain an AOA position detection embedded coding vector and a TDOA position detection embedded coding vector; S72, guiding the interaction of the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector based on implicit joint features to obtain a position detection significant joint interaction coding vector; S73, inputting the position detection significant joint interaction coding vector into a decoder-based position parsing module to obtain the positioning data of the positioned communication device.

[0107] Specifically, the step S71 embeds and codes the AOA position detection data of the positioning communication device and the TDOA position detection data of the positioning communication device to obtain an AOA position detection embedded coding vector and a TDOA position detection embedded coding vector. It should be understood that, considering that AOA and TDOA are two different wireless positioning technologies, there are differences between the two in positioning principles and measurement methods. In order to enable data from different sources and different dimensions to be compared and fused in the same dimension, the present application uses embedded coding technology to process the AOA position detection data of the positioning communication device and the TDOA position detection data of the positioning communication device, so as to convert the two into the same vector representation and retain the key features of the original data, so as to facilitate the use of deep learning algorithms for effective information interaction between the two in the subsequent data fusion processing process. In an embodiment of the present application, the BERT model is used as a specific implementation method of the embedded coding technology to obtain an AOA position detection embedded coding vector and a TDOA position detection embedded coding vector.

[0108] Specifically, in step S72, the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector are guided to interact based on implicit joint features to obtain a position detection significant joint interaction coding vector. That is, in order to make full use of the complementarity between the AOA position detection data and the TDOA position detection data to improve the positioning accuracy, it is necessary to further fuse the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector. It is worth mentioning that in order to achieve deep interaction between data, the present application proposes a guided interaction method based on implicit joint features, which guides the explicit fusion of features between the AOA position detection data and the TDOA position detection data by mining the implicit association between the two, thereby achieving a deeper level of feature interaction.

[0109] Figure 6 The present invention is a flowchart of performing guided interaction based on implicit joint features on the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector in the multimodal fusion indoor positioning method according to an embodiment of the present application to obtain a position detection significant joint interaction coding vector. Figure 6 As shown, the step S72 includes: S721, extracting the implicit joint features between the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector as a conditional feature vector; S722, based on the content contribution of the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector relative to the conditional feature vector, feature modulating the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector to obtain a modulated AOA position detection embedded coding vector and a modulated TDOA position detection embedded coding vector; S723, performing cross-domain interactive fusion based on the attention mechanism on the modulated AOA position detection embedded coding vector, the modulated TDOA position detection embedded coding vector and the conditional feature vector to obtain the position detection significant joint interactive coding vector.

[0110] Specifically, the step S721 includes: inputting the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector into an implicit joint feature capture network to obtain an AOA-TDOA position detection implicit joint feature vector; and performing feature activation based on a Sigmoid function on the AOA-TDOA position detection implicit joint feature vector to obtain the conditional feature vector. Among them, inputting the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector into an implicit joint feature capture network to obtain an AOA-TDOA position detection implicit joint feature vector includes: fusing the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector according to position points and passing them through a neural network layer based on a tanh function to obtain the AOA-TDOA position detection implicit joint feature vector.

[0111] Specifically, the step S722 includes: calculating the first content contribution of the AOA position detection embedded coding vector relative to the conditional feature vector; calculating the second content contribution of the TDOA position detection embedded coding vector to the conditional feature vector; normalizing the first content contribution and the second content contribution, and using the normalized first content contribution and the second content contribution to weighted modulate the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector to obtain the modulated AOA position detection embedded coding vector and the modulated TDOA position detection embedded coding vector. Among them, calculating the first content contribution of the AOA position detection embedded coding vector relative to the conditional feature vector includes: calculating the dot division vector between the AOA position detection embedded coding vector and the conditional feature vector, and calculating the base-two logarithmic value of the absolute value of each eigenvalue in the dot division vector to obtain the AOA position detection content contribution representation vector; calculating the dot product vector between the AOA position detection content contribution representation vector and the AOA position detection embedded coding vector, and calculating an exponential function value with base e and the sum of the eigenvalues ​​of the dot product vector as the exponent to obtain the first content contribution.

[0112] Specifically, the step S723 includes: using the modulated AOA position detection embedded coding vector as a query vector, the modulated TDOA position detection embedded coding vector as a key vector and the conditional feature vector as a value vector, inputting the modulated AOA position detection embedded coding vector, the modulated TDOA position detection embedded coding vector and the conditional feature vector into a significant guided interaction module between features based on a converter structure to obtain the position detection significant joint interaction coding vector. More specifically, the step S723 includes: multiplying the modulated AOA position detection embedded coding vector by the transposed vector of the modulated TDOA position detection embedded coding vector and then dividing it by the square root of the length of the TDOA position detection embedded coding vector to obtain the AOA-TDOA position detection query attention score matrix; multiplying the AOA-TDOA position detection query attention score matrix by the conditional feature vector after passing it through a softmax function to obtain the position detection significant joint interaction coding vector.

[0113] Accordingly, the step S72 includes: processing the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector using the following feature interaction fusion formula to obtain the position detection significant joint interaction coding vector, wherein the feature interaction fusion formula is:

[0114]

[0115]

[0116]

[0117]

[0118]

[0119]

[0120]

[0121]

[0122]

[0123] in, represents the AOA position detection embedding coding vector, represents the TDOA position detection embedded coding vector, Indicates adding by position point. represents the implicit joint feature vector of AOA-TDOA position detection, and Represent the weight matrix and bias parameters of the neural network layer, respectively. represents the hyperbolic tangent function, represents the sigmoid activation function, represents the conditional eigenvector, , and Respectively represent the first eigenvalue, the TDOA position detection is embedded in the encoding vector eigenvalue and the conditional eigenvector eigenvalues, is the absolute value symbol, represents the logarithmic function with base 2, represents the exponential function with base e, represents the length of the AOA position detection embedded coding vector, and the lengths of the AOA position detection embedded coding vector, the TDOA position detection embedded coding vector and the conditional feature vector are the same, and Represent the first content contribution and the second content contribution respectively, and They represent the normalized first content contribution and second content contribution, respectively. and They represent the AOA position detection embedded coding vector after modulation and the TDOA position detection embedded coding vector after modulation, represents the transpose of a vector, represents the matrix multiplication operation, is the normalized exponential function, Represents the position detection significant joint interaction encoding vector.

[0124] Specifically, it first extracts the implicit correlation features shared between the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector through an implicit joint feature capture network, and uses the Sigmoid function to perform nonlinear transformation on the extracted implicit correlation features to emphasize key information and suppress irrelevant noise, thereby constructing a conditional feature vector to guide the subsequent feature interaction fusion process. Then, the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector are respectively evaluated for correlation with the conditional feature vector to determine the importance of the two for the joint feature representation, and the original AOA position detection embedded coding vector and the TDOA position detection embedded coding vector are weighted modulated to effectively emphasize the key information in the two and suppress irrelevant information. Finally, the modulated AOA position detection embedded coding vector is used as the query vector, the modulated TDOA position detection embedded coding vector is used as the key vector, and the conditional feature vector is used as the value vector, and the Transformer architecture is used to guide the interaction between features, thereby realizing the deep interactive fusion of cross-modal features to generate a position detection significant joint interaction coding vector containing the association information of multi-source position detection data.

[0125] Specifically, in step S73, the position detection significant joint interaction coding vector is input into a decoder-based position analysis module to obtain the positioning data of the located communication device. That is, by utilizing the powerful fitting ability of the decoder model, the multi-source position detection information contained in the position detection significant joint interaction coding vector is feature analyzed to restore the actual position information of the located communication device and output the corresponding positioning data.

[0126] In the technical solution of the present application, the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector represent low-dimensional embedded coding features of the AOA position detection data and the TDOA position detection data of the positioning communication device, respectively. In this way, when performing guided interaction based on implicit joint features, considering that there are modal differences and internal noises in the data source domain between the AOA position detection data and the TDOA position detection data of the positioning communication device, the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector have expression heterogeneity in the high-dimensional feature space, which makes the extraction of the joint implicit feature insufficient and inaccurate, resulting in the feature manifold of the position detection significant joint interaction coding vector obtained by the significant guided interaction between features with the joint implicit feature as conditional information having expression sparsity and uneven distribution in the high-dimensional feature space, thereby affecting the accuracy of the positioning data of the located communication device obtained by the decoder-based position resolution module.

[0127] Based on this, in a preferred embodiment, the position detection significant joint interaction coding vector is input into a decoder-based position analysis module to obtain the positioning data of the located communication device, including: performing L random sampling on the position detection significant joint interaction coding vector to obtain L positioning random perturbation eigenvalues; arranging the L positioning random perturbation eigenvalues ​​in order from large to small to obtain a positioning random perturbation ordered vector; calculating the self-inner product of the position detection significant joint interaction coding vector to obtain a positioning energy modulation eigenvalue; performing semantic intensity scaling on the positioning random perturbation ordered vector based on the positioning energy modulation eigenvalue to obtain a semantically enhanced positioning random perturbation ordered vector; calculating the product between the semantically enhanced positioning random perturbation ordered vector and the transposed vector of the position detection significant joint interaction coding vector to obtain a positioning best sample hint orthogonal modulation matrix; calculating the matrix product between the positioning best sample hint orthogonal modulation matrix and the position detection significant joint interaction coding vector to obtain an optimized position detection significant joint interaction coding vector; and inputting the optimized position detection significant joint interaction coding vector into the decoder-based position analysis module to obtain the positioning data of the located communication device.

[0128] Accordingly, the optimization process of the position detection significant joint interaction coding vector can be expressed by the following optimization formula:

[0129]

[0130]

[0131]

[0132]

[0133]

[0134]

[0135]

[0136]

[0137] in, is the position detection significant joint interaction encoding vector, represents a normal distribution, represents a random variable that follows a normal distribution, represents the number of random samplings, and denote the feature mean and feature variance of the position detection significant joint interaction encoding vector, respectively, represents each positioning random perturbation eigenvalue obtained by performing L random sampling on the position detection significant joint interaction encoding vector, is the ordered vector of the positioning random disturbance, represents the calculation of the inner product of the eigenvector itself, is the positioning energy modulation characteristic value, is the positioning best sample hint orthogonal modulation matrix, It means point multiplication by position. represents the matrix multiplication operation, is the optimized position detection significant joint interaction encoding vector.

[0138] Accordingly, by performing L random sampling on the position detection significant joint interaction encoding vector to obtain a set of new feature values ​​based on the original features but containing slight changes, randomness is introduced to explore different aspects of the feature space, and by simulating the noise or variation in the actual data, it helps the model to better generalize to unseen data. Then, these positioning random perturbation feature values ​​are arranged in order from large to small to form a positioning random perturbation ordered vector to identify which perturbation directions may be the most important for the current problem. Such an ordering not only helps the algorithm focus on the feature differences that can best distinguish different categories, but also helps to reveal hidden data structures or patterns. Subsequently, the inner product of the position detection significant joint interaction coding vector is calculated to obtain the positioning energy modulation eigenvalue, which reflects the overall strength or "energy" of the original feature vector and provides a benchmark for measuring the importance of the feature. Using this positioning energy modulation eigenvalue as a reference point, the previously formed positioning random perturbation ordered vector is semantically scaled to construct a semantically enhanced positioning random perturbation ordered vector. In this way, those disturbances that appear to be more important in the original feature space are amplified, thereby enhancing the model's sensitivity to changes in key features, while also retaining relatively small but potentially useful disturbance information.

[0139] Then, the product of the ordered vector of random perturbation of semantic reinforcement localization and the transposition of the position detection significant joint interaction encoding vector is calculated to obtain the best localization sample prompt orthogonal modulation matrix, the purpose of which is to find the interaction between features under two different perspectives. The formed matrix can be regarded as a conversion tool for capturing and representing the complex linear relationship between features. Finally, the final feature structure modulation is completed by applying the best localization sample prompt orthogonal modulation matrix constructed above to the original position detection significant joint interaction encoding vector, so that the features are more in line with the requirements of the objective function, reducing redundant information, while maximizing the retention of useful information, so as to improve learning efficiency and prediction performance.

[0140] In summary, the feature manifold optimization process of the position detection significant joint interaction coding vector is clarified, which extracts the best prompt samples based on fine-grained target retrieval with random perturbations, and then guides adversarial robust learning based on the best prompt samples to complete the structure of the high-dimensional manifold of the feature vector in the feature space, so as to improve the detail and structure of the feature expression, thereby improving the accuracy of the positioning data of the located communication device obtained by the position resolution module based on the decoder input of the position detection significant joint interaction coding vector.

[0141] In summary, according to the multimodal fusion indoor positioning method of the embodiment of the present application, first, the positioning base station system sends a positioning audio signal, and the positioned communication device receives and records the audio signals from different base stations and their corresponding timestamps, and obtains the TDOA position detection data through the TDOA algorithm. Next, the positioned communication device sends a predefined Bluetooth signal to the positioning base station system. After receiving the signal, each base station in the positioning base station system calculates the AOA position detection data of the communication device through the Bluetooth AOA technology. Then, by fusing the acoustic signal TDOA and the Bluetooth AOA position detection data, more accurate location information of the communication device is output. This method effectively overcomes the problem of limited accuracy of a single positioning method by fusing TDOA and AOA positioning observation information, and can better cope with challenges such as multipath effects and non-line-of-sight propagation, providing more reliable positioning services in two-dimensional dynamic scenes.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A multi-modal fusion indoor positioning method, characterized in that: include: The positioning base station system sends a positioning audio signal to the located communication device, and the positioning base station system includes a main base station and three slave base stations; The located communication device receives and records the arrival timestamp of the positioning audio signal from the positioning base station system to obtain a set of signal arrival timestamps; Based on the set of signal arrival timestamps and the location data of each base station in the positioning base station system, the TDOA location detection data of the positioning communication device is calculated by a TDOA algorithm; Sending a Bluetooth signal to the positioning base station system through the located communication device; Each base station in the positioning base station system captures the Bluetooth signal to obtain a set of Bluetooth IQ signals; Based on the set of Bluetooth IQ signals, the AOA position detection data of the positioning communication device is calculated by the AOA algorithm; Fusion of the AOA position detection data of the positioning communication device and the TDOA position detection data of the positioning communication device to obtain the positioning data of the positioned communication device; Wherein, fusing the AOA position detection data of the positioning communication device and the TDOA position detection data of the positioning communication device to obtain the positioning data of the positioned communication device includes: Embedding and coding the AOA position detection data of the positioning communication device and the TDOA position detection data of the positioning communication device to obtain an AOA position detection embedded coding vector and a TDOA position detection embedded coding vector; Performing guided interaction based on implicit joint features on the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector to obtain a position detection significant joint interaction coding vector; The position detection significant joint interaction coding vector is input into a decoder-based position resolution module to obtain the positioning data of the located communication device.

2. The multi-modal fusion indoor positioning method according to claim 1, characterized in that: Based on the set of the Bluetooth IQ signals, the AOA position detection data of the positioning communication device is calculated by the AOA algorithm, including: Using the AOA algorithm to process each Bluetooth IQ signal in the set of Bluetooth IQ signals to obtain a set of arrival angles at a base station; The set of arrival angles at the base station is sent to a positioning server for positioning data solution to obtain the AOA position detection data of the positioning communication device.

3. The multi-modal fusion indoor positioning method according to claim 1, characterized in that: The AOA position detection embedded coding vector and the TDOA position detection embedded coding vector are subjected to guided interaction based on implicit joint features to obtain a position detection significant joint interaction coding vector, including: Extracting an implicit joint feature between the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector as a conditional feature vector; Based on the content contribution of the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector relative to the conditional feature vector, feature modulate the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector to obtain a modulated AOA position detection embedded coding vector and a modulated TDOA position detection embedded coding vector; The modulated AOA position detection embedded coding vector, the modulated TDOA position detection embedded coding vector and the conditional feature vector are cross-domain interactively fused based on the attention mechanism to obtain the position detection significant joint interactive coding vector.

4. The multi-modal fusion indoor positioning method according to claim 3, characterized in that: Extracting an implicit joint feature between the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector as a conditional feature vector, comprising: Inputting the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector into an implicit joint feature capture network to obtain an AOA-TDOA position detection implicit joint feature vector; The AOA-TDOA position detection implicit joint feature vector is subjected to feature activation based on a Sigmoid function to obtain the conditional feature vector.

5. The multi-modal fusion indoor positioning method according to claim 4, characterized in that: Based on the content contribution of the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector relative to the conditional feature vector, feature modulating the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector to obtain a modulated AOA position detection embedded coding vector and a modulated TDOA position detection embedded coding vector, including: Calculating a first content contribution of the AOA position detection embedded coding vector relative to the conditional feature vector; Calculate a second content contribution of the TDOA position detection embedded coding vector to the conditional feature vector; The first content contribution and the second content contribution are normalized, and the normalized first content contribution and the second content contribution are used to weighted modulate the AOA position detection embedded coding vector and the TDOA position detection embedded coding vector to obtain the modulated AOA position detection embedded coding vector and the modulated TDOA position detection embedded coding vector.

6. The multi-modal fusion indoor positioning method according to claim 5, characterized in that: The modulated AOA position detection embedded coding vector, the modulated TDOA position detection embedded coding vector and the conditional feature vector are subjected to cross-domain interactive fusion based on an attention mechanism to obtain the position detection significant joint interactive coding vector, including: Taking the post-modulation AOA position detection embedded coding vector as the query vector, the post-modulation TDOA position detection embedded coding vector as the key vector and the conditional feature vector as the value vector, the post-modulation AOA position detection embedded coding vector, the post-modulation TDOA position detection embedded coding vector and the conditional feature vector are input into a feature-to-feature significant guided interaction module based on a converter structure to obtain the position detection significant joint interaction coding vector.

Images