Indoor navigation method based on multi-modal data fusion and large model enhancement

Abstract

The invention provides an indoor navigation method based on multi-modal data fusion and large model enhancement, which comprises the following steps: acquiring multi-modal data including Bluetooth signal intensity data, IMU (Inertial Measurement Unit) motion data and image data through a user mobile terminal, and constructing multi-source sensor input; iMM-EKF is adopted to carry out deep fusion on multi-modal data, different motion models are established, different motion states of a user can be adapted, and accurate navigation can be output; the accuracy and the anti-interference performance of the system are improved by performing optimization enhancement through a large model, compensating IMU (Inertial Measurement Unit) through MLP, enhancing visual observation through Transformer and dynamically estimating IMM-EKF noise parameters through LSTM (Long Short Term Memory); and regulating and controlling output weights of the IMM-EKF and the large model through an A2C reinforcement learning framework strategy, and outputting final positioning.

Description

Technical Field

[0001] The present invention relates to the technical field of indoor navigation, and particularly to an indoor navigation method based on multi-modal data fusion and large model enhancement. Background Art

[0002] In indoor navigation scenarios, the Global Positioning System (GPS) signal often fails due to building blockages. Traditional methods usually rely on single-modal data for navigation, such as relying only on Bluetooth for navigation.

[0003] However, the above single-modal methods have significant drawbacks: Bluetooth positioning has insufficient accuracy due to limited signal coverage and device compatibility issues; Inertial Measurement Unit (IMU) positioning can provide continuous motion data, but it will cause positioning drift due to cumulative errors; Image positioning may also fail in scenarios with changing light or single texture. These limitations make it difficult for traditional methods to meet the actual requirements of navigation accuracy in complex indoor environments.

[0004] Therefore, it is necessary to propose an indoor navigation method that can comprehensively solve the problems of using the above single-modal data. Summary of the Invention

[0005] The present invention provides an indoor navigation method based on multi-modal data fusion and large model enhancement to solve the defects existing in the prior art of using single-modal data for navigation.

[0006] In a first aspect, the present invention provides an indoor navigation method based on multi-modal data fusion and large model enhancement, including:

[0007] Obtain multi-modal data through the user's mobile terminal;

[0008] Obtain navigation prediction results of different modalities, and perform deep fusion on the navigation prediction results of different modalities using IMM-EKF to obtain an IMM-EKF fusion positioning result;

[0009] Optimize and enhance the multi-modal data using a comprehensive large model to obtain a comprehensive large model corrected positioning result;

[0010] Use the A2C reinforcement learning framework strategy to regulate the output weights of the IMM-EKF fusion positioning result and the comprehensive large model corrected positioning result to determine the final navigation positioning result.

[0011] In a second aspect, the present invention further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the indoor navigation method based on multi-modal data fusion and large model enhancement as described in any one of the above.

[0012] The indoor navigation method based on multi-modal data fusion and large model enhancement provided by the present invention adapts to different motion states of users by introducing IMM-EKF to fuse multi-modal data, and dynamically adjusts parameters through a large model and an A2C learning framework, which can provide users with more accurate and reliable indoor navigation services. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0014] Figure 1 is one of the flowcharts of the indoor navigation method based on multi-modal data fusion and large model enhancement provided by the present invention;

[0015] Figure 2 is another flowchart of the indoor navigation method based on multi-modal data fusion and large model enhancement provided by the present invention;

[0016] Figure 3 is the method flowchart of the IMM-EKF provided by the present invention;

[0017] Figure 4 is the method flowchart of the A2C learning framework provided by the present invention;

[0018] Figure 5 is the structural diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0019] To make the objectives, technical solutions, and advantages of the present invention clearer, the following will clearly and completely describe the technical solutions in the present invention in conjunction with the drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments in the present invention belong to the scope of protection of the present invention.

[0020] Figure 1 is one of the flowcharts of the indoor navigation method based on multi-modal data fusion and large model enhancement provided by the embodiments of the present invention, as Figure 1As shown in the figure, it includes:

[0021] Step 1: Obtain multimodal data through the user's mobile device;

[0022] Step 2: Obtain different modal navigation prediction results, and use IMM-EKF to deeply fuse the different modal navigation prediction results to obtain an IMM-EKF fusion positioning result;

[0023] Step 3: Use a comprehensive large model to optimize and enhance the multimodal data to obtain a comprehensive large model corrected positioning result;

[0024] Step 4: Use the A2C reinforcement learning framework strategy to regulate the output weights of the IMM-EKF fusion positioning result and the comprehensive large model corrected positioning result to determine the final navigation positioning result.

[0025] In the embodiment of the present invention, multimodal data is obtained through the user's mobile device. The multimodal data includes Bluetooth signal strength data, IMU motion data, and image data, and a multi-source sensor input is constructed. The Interacting Multiple Model Extended Kalman Filter (IMM-EKF) is used to deeply fuse the multimodal data, and different motion models are established, which can adapt to different motion states of the user and output accurate navigation. Through the large model for optimization and enhancement, MLP compensates for the IMU, Transformer enhances visual observation, and LSTM dynamically estimates the IMM-EKF noise parameters, improving the accuracy and anti-interference ability of the system. Through the A2C reinforcement learning framework strategy to regulate the output weights of IMM-EKF and the large model, the final positioning is output.

[0026] Compared with traditional indoor navigation methods, the present invention introduces multimodal data, and innovatively introduces IMM-EKF to deeply fuse the multimodal data, uses a variety of large models to correct the positioning result, and finally heuristically uses the A2C learning framework to fuse and dynamically adjust the weights of the positioning results of IMM-EKF and the large model to obtain the final high-precision positioning. The present invention introduces IMM-EKF to fuse multimodal data to adapt to different motion states of the user, and dynamically adjusts parameters through the large model and the A2C learning framework, which can provide more accurate and reliable indoor navigation services for users.

[0027] Specifically, as Figure 2 shown, in Step 1, multimodal data is obtained through the mobile phone. Bluetooth data is obtained by scanning the existing low-power Bluetooth beacons in the mall through the mobile phone, IMU data is collected by the mobile phone, and the system is allowed to automatically take pictures during navigation, and image data is obtained through the mobile phone camera;

[0028] In step 2, IMM-EKF is used to deeply fuse multimodal data, five motion models are established, and a Markov transition matrix for a shopping mall with a large flow of people is preset, that is, the transition probabilities of various motion models of users in the shopping mall are preset. For example, in the corridor area, the preset model is transferred to a uniform motion model, and in the corner area, the preset model is transferred to a turning model;

[0029] Step 3: Optimize and enhance through a large model. Input the IMU data of the past T = 5 frames, and compensate the IMU through MLP. Input five consecutive image blocks, enhance visual observation through Transformer, and specifically detect the areas where feature point mis-matching occurs due to reasons such as glass reflection. Dynamically estimate the IMM-EKF noise parameters through LSTM, and specifically conduct tests for violent motions such as sudden speed increase or sharp turning. In addition, tests are also carried out in areas with large Bluetooth signal fluctuations;

[0030] Step 4: Regulate the output weights of IMM-EKF and the large model through the A2C reinforcement learning framework strategy. There are two output methods when iterating to the last layer:

[0031] 1. Directly output after calculation through the finally updated ξ

[0032] 2. Substitute the updated Q j,k , R j,k , into the calculation again to obtain and Finally, obtain through calculation with the finally updated ξ

[0033] The embodiment of the present invention adopts method 1 because method 1 is computationally efficient and suitable for scenarios with high real-time requirements. In method 2, the iteratively updated Q j , R j , will affect the update of the next model, thereby indirectly affecting the finally generated . Therefore, method 2 is applicable to situations where parameter mutations occur or scenarios where high navigation accuracy is required.

[0034] In one embodiment, as Figure 3 shown, IMM-EKF is used to deeply fuse multimodal data, and the specific process is as follows:

[0035] Step 201: Define the state vector x k at time k. The state vector x k includes the position variable p k , the velocity variable v k and the attitude variable θ k , and their relationship is represented by the following formula:

[0036]

[0037] Step 202: Predict the navigation by driving with IMU motion data. The specific process is as follows:

[0038] Step 2021: Obtain IMU data including acceleration a k and angular velocity ω k for predicting the state;

[0039] Step 2022: Predict x k|k-1 through the following non - linear motion model:

[0040] x k|k-1 = f(x k-1 , u k )

[0041] where u k represents the input of the IMU:

[0042]

[0043] f(x k-1 , u k ) is specifically:

[0044]

[0045] Step 2023: Since the motion model is non - linear, calculate the Jacobian matrix using the following formula:

[0046]

[0047] Step 2024: Predict the covariance using the following formula:

[0048]

[0049] where Q k is the IMU noise covariance matrix:

[0050]

[0051] where, Q IMU is the inherent noise covariance matrix of the IMU sensor;

[0052] Step 203: Predict the navigation by driving with Bluetooth data. The specific process is as follows:

[0053] Step 2031: Obtain the position observation z through the ranging principle of Received Signal Strength Indication (RSSI)蓝牙 :

[0054] Step 20311: The Bluetooth beacon continuously broadcasts a signal, and the user terminal receives the received signal strength indicator (RSSI), which is converted into a distance through a path loss model:

[0055] RSSI = A - 10n log 10 (d) + X σ

[0056] where A is the reference signal strength at 1 meter, which needs to be calibrated on-site. n is the path loss exponent. d is the actual distance between the Bluetooth beacon and the user terminal. X σ is random noise.

[0057] Then the formula for converting RSSI to distance d is as follows:

[0058]

[0059] Define z through distance d 蓝牙 :

[0060]

[0061] Step 2032: Define the Bluetooth observation function h 蓝牙 (x k ):

[0062] z 蓝牙 = h 蓝牙 (x k ) + v 蓝牙

[0063] where v 蓝牙 is the observation noise.

[0064] Assume there are 3 Bluetooth beacons in the system, and p i is the coordinate of the i-th Bluetooth beacon:

[0065]

[0066] Input the position component p k in the state vector x k , and calculate the theoretical Euclidean distance from the user terminal to each Bluetooth beacon through the Bluetooth observation function h 蓝牙 (x k ):

[0067]

[0068] Step 2033: Calculate the Jacobian matrix using the following formula:

[0069]

[0070] Step 2034: Define the noise covariance of Bluetooth:

[0071]

[0072] Step 204: Predict navigation through image data driving. The specific process is as follows:

[0073] Step 2041: Visual SLAM outputs the position p k , and the specific steps are as follows:

[0074] Step 20411: Extract feature points through ORB, solve the camera pose through the PnP algorithm, and convert it to p in the global coordinate system k ;

[0075] Step 20412: Optimize and calculate the covariance ∑SLAM through Bundle Adjustment (BA):

[0076] ∑SLAM = (J T J) -1

[0077] where J is the Jacobian matrix of BA;

[0078] Step 2042: Define the observation equation of z 图像 as follows:

[0079] z 图像 = p k + v 图像

[0080] where v 图像 is the observation noise.

[0081] Define the observation function h 图像 (x k ):

[0082] h 图像 (x k ) = C·p k

[0083] where C is the coordinate transformation matrix for rotation and translation from the SLAM coordinate system to the EKF global coordinate system;

[0084] Step 2043: Calculate the Jacobian matrix according to the following formula:

[0085]

[0086] Step 2044: Calculate R 图像 , R 图像Only contains the position uncertainty of visual SLAM, and its diagonal elements reflect the observation noise variances of each coordinate axis (x / y / z):

[0087]

[0088] Step 205: Use the method of interacting multiple model extended Kalman filter (IMM-EKF) to fuse multi-modal data:

[0089] Step 2051: Define the interaction model and perform calculations:

[0090] Step 20511: Typical motion models include uniform linear motion, uniformly accelerated linear motion, left-turn motion, right-turn motion, and stationary model. Define r as the total number of motion models, then r = 5. Define i and j as the indices of the motion models;

[0091] Step 20512: Calculate the mixing probability from model i to model j, which represents the contribution weight of model i at the previous moment to model j at the current moment

[0092]

[0093] where p ij is the Markov transition probability from model i to model j, which is a constant matrix preset according to the typical motion model, and μ i (k - 1) is the probability of model i at the previous moment;

[0094] Step 20513: Weight the state estimates of each model at the previous moment according to the mixing probability and use it as the initial input of model j:

[0095]

[0096] Step 20514: Calculate the covariance

[0097]

[0098] Step 2052: State prediction:

[0099]

[0100] where F j is the state transition matrix of model j, B j is the control input matrix, u j (k) is the IMU input of model j, and A j (k) is the correction term:

[0101]

[0102] Aj (k) = B j Δu j (k)

[0103] Step 2053, Covariance Prediction:

[0104]

[0105] where Q j is the process noise covariance of model j, calculated in Step 2024;

[0106] Step 2054, Assuming the noises of Bluetooth and image data are independent, update the joint observation of Bluetooth and image:

[0107] Step 20541, Combine the Bluetooth observation and the image observation, and define z k as the joint vector of Bluetooth and image data:

[0108]

[0109] The following formula can be obtained:

[0110]

[0111] Step 20542, Calculate the Kalman gain using the following formula:

[0112]

[0113] where S j (k) is the innovation covariance matrix, representing the uncertainty between the observation prediction value and the actual observation value:

[0114]

[0115] where H j is the augmented observation matrix of model j, and R j is the observation noise covariance of model j:

[0116]

[0117] Step 20543, State Update:

[0118] x j (k|k) = x j (k|k - 1) + K j (k)(z k - H j x j (k|k - 1))

[0119] Step 20544, Covariance Update:

[0120] P j (k|k)=(I-K j (k)H j )P j (k|k-1)

[0121] Step 2055: Perform model probability update:

[0122] Step 20551: Calculate the likelihood function:

[0123]

[0124] Calculate the matching degree between the current Bluetooth observation data and image observation data and the prediction of model j through the likelihood function for dynamic update;

[0125] Step 20552: Calculate the probability of model j:

[0126]

[0127] Step 2056: Finally output the positioning result after fusing multiple models:

[0128]

[0129] The present invention creatively integrates Bluetooth signal strength data, IMU motion data, and visual image data to construct a triple redundant perception system. When any data source fails, the system can automatically switch to other valid data sources for continuous navigation, which has advantages compared with traditional single data-driven navigation methods; in addition, based on the IMM-EKF framework, the present invention establishes a five-dimensional motion model library including uniform motion, accelerated motion, left and right turning, and stationary. By calculating the probabilities of the five motion models through IMU data and introducing Bluetooth data and image data for likelihood calculation, the present invention can intelligently identify the current dominant motion mode and dynamically adjust the fusion weights of each motion model to obtain accurate positioning results.

[0130] In one embodiment, in step 3, optimization and enhancement are performed through a large model, MLP compensates for IMU, Transformer enhances visual observation, and LSTM dynamically estimates the IMM-EKF noise parameters to improve the accuracy and anti-interference ability of the system. The specific process is as follows:

[0131] Step 301: Define the multi-layer perceptron MLP:

[0132] Input layer: h0 = x

[0133] where x is the input vector;

[0134] Step 3012: Assume there are a total of L layers, and the l-th hidden layer is as follows:

[0135] h l = σ l (W l h l-1 + b l )

[0136] where W l is the weight matrix, b l is the bias, and σ l is the activation function;

[0137] Step 3013, Output layer:

[0138] y = σ out (W L h L-1 + b L )

[0139] where σ out is the output activation function;

[0140] Step 302, Use the MLP to perform drift compensation on the IMU of each model j:

[0141] Step 3021, Define x k during the process of using the MLP to perform drift compensation on the IMU as:

[0142]

[0143] where T is the time step of the sliding window, used to capture temporal information and needs to be set in advance;

[0144] Step 3022, Predict the acceleration Δa k and the angular velocity deviation Δω k :

[0145]

[0146] Step 3023, Assume that an L-layer MLP is used, the activation function is ReLU, and the first layer h0 = x k , then the expansion of the formula in Step 3021 is as follows:

[0147] The l-th layer:

[0148] h l = ReLU(W l h l-1 + b l )

[0149] Output layer:

[0150]

[0151] Step 3024. Correct the initial a k and ω k as follows:

[0152]

[0153] Determine what type of motion model the model j is through μ j (k), so as to determine whether to preferentially compensate for the acceleration deviation or the angular velocity deviation;

[0154] Step 303. Use Transformer to correct the visual observations of each model j:

[0155] Step 3031. Input the time-series visual observation window {z 图像，k-T , ……, z 图像，k} and the original image patches {I k-T , ……, I k}, where T is a preset period;

[0156] Step 3032. The formula of the multi-head self-attention mechanism (MSA) of Transformer is as follows:

[0157]

[0158] Among them, Q, K, and V are obtained by linear projection of the input, and d k is the attention dimension;

[0159] Step 3033. Calculate the corrected value of the positioning obtained from the image data:

[0160] Δz k = MLP(MSA(z 图像，k-T:k , I k-T:k ))

[0161] Step 3034. Correct z 图像 as follows:

[0162]

[0163] Step 304. Dynamically estimate the noise parameters of the IMM-EKF for each model j through LSTM:

[0164] Step 3041. Define the input vector x k , x k of LSTM to carry the information of the current sensor and the probability μ j (k) of model j:

[0165]

[0166] Step 3042: The LSTM updates the cell state c and the hidden state h through the gating mechanism: k and the hidden state h k :

[0167] Step 30421: The forget gate determines how much information to discard from the previous cell state c: k-1 f = σ(W[h, x] + b) where σ is the sigmoid function with an output range of [0,1], controlling the proportion of information retained. W is the forget gate weight matrix and b is the bias term.

[0168] f k = σ(W f [h k-1 , x k + b f )

[0169] where σ is the sigmoid function with an output range of [0,1], controlling the proportion of information retained. W f is the forget gate weight matrix and b f is the bias term;

[0170] Step 30422: The input gate defines the activation value i of the input gate, controlling the proportion of new information to be written, and defines the candidate cell state c, containing the new information generated by the current input: k i = σW[h, x] + b ～ c k = tanh(σW[h, x] + b

[0171] i k = σW i [h k-1 , x k + b i

[0172] ～ c k = tanh(σW c [h k-1 , x k + b c )

[0173] Step 30423: Cell state update, the forget gate f selectively discards the old state c, and the input gate i selectively adds the new state k c: k-1 c = f ⊙ c + i ⊙ k c ～ c k :

[0174] c k = f k ⊙ c k-1 + i k ⊙ ～ c k

[0175] Step 30424: Define o to control the cell state c k c kHow much information is input into the hidden state h k :

[0176] o k = σ(W o [h k-1 , x k + b o )

[0177] h k = o k ⊙ tanh(c k )

[0178] Step 3043: Calculate the noise parameters Q k and R k :

[0179]

[0180] The regulation of the noise parameters by the LSTM is specifically manifested as follows: If a k suddenly increases, indicating intense movement and unreliable IMU prediction, then increase Q j,k ; If μ j (k) fluctuates violently, then evenly adjust the Q of each model j,k ; If z 蓝牙 fluctuates more than the threshold, then increase R 蓝牙 ; If the solution of z 图像 fluctuates greatly, then increase r 图像 ; If all inputs are stable, then restore the default Q j,k and R j,k ;

[0181] Step 305: After compensating the IMU of each model j through the MLP and correcting with the Transformer, the LSTM predicts the Q k and R k of each model j in real time. After substituting these parameters adjusted by the large model back into Step 2 for calculation, the localization corrected by the large model is finally obtained

[0182] The present invention innovatively introduces three types of deep learning models to construct an error compensation closed-loop: The MLP network compensates for the IMU zero-bias error through temporal modeling, the Transformer architecture corrects visual observation outliers using the attention mechanism, and the LSTM network dynamically predicts the optimal noise parameters. The parameters optimized by the large model are re-injected into the IMM-EKF iterative calculation to obtain the localization corrected by the large model.

[0183] In one embodiment, as Figure 4As shown in the figure, in step 4, the output weights of the IMM-EKF and the large model are regulated through the A2C reinforcement learning framework policy to output the final positioning. The specific process is as follows:

[0184] Step 401: At time k, input the state s into the Actor network k , and the Actor network outputs the action a k :

[0185] Step 4011: Input the state s at time k k :

[0186]

[0187] Step 4012: The Actor network outputs the action a k :

[0188]

[0189] Step 4013: α j Dynamically adjust Q j , β j Dynamically adjust R j , γ j Supplement and adjust the model fusion weights, and dynamically adjust the output weights of the IMM-EKF and the large model:

[0190]

[0191] When α j > 1, increase the process noise of the IMU to adapt to high maneuverability, such as sharp turns. When α j < 1, reduce the process noise of the IMU and increase the weight of the IMU data to adapt to motions with small IMU drifts such as uniform motion. β j Adjust the trust levels of the Bluetooth data and the image data. When β j decreases, rely more on the Bluetooth observations and the image observations. It is necessary to re-normalize to ensure the sum of is 1.

[0192] In areas with stable Bluetooth signals and rich visual features, trust the output results of the IMM-EKF, increase ξ. When the Bluetooth signal is unstable or when visual SLAM fails due to reasons such as glass reflection, use the state predicted by the large model based on the IMU and LSTM, reduce ξ, and the A2C framework adjusts ξ in real time to achieve soft switching;

[0193] Step 402: The Critic network outputs the state value function and the TD error δ k ;

[0194] Step 4021: The feature extraction layer of the Actor network takes s t as the input to the Critic network, and the Critic network outputs the state value function

[0195] Step 4022: Calculate the TD error δ k using the following formula:

[0196] δ k = r k + γV(S k+1 ) - V(S k )

[0197] r k is the current reward function, and γ is the discount factor:

[0198]

[0199] where is the true position, and tr(p(k|k) represents the penalty for positioning uncertainty;

[0200] When the true position is not available, the multi-sensor consistency is used as the reward:

[0201] r k = -λ1||z 蓝牙 - h 蓝牙 (x k )|| - λ2||z 图像 - h 图像 (x k )||

[0202] Step 403: Update the parameters of the Critic network to reduce the TD error δ k ;

[0203] Step 404: Use the TD error δ k to calculate the advantage function A π (s k , a k ):

[0204] A π (s k , a k ) = Q π (s k , a k ) - B(s k )

[0205] where Q π (s k , a k ) is the action value function, and B(s k) is the introduced benchmark function, then A π (s k ,a k ) is the advantage function relative to the benchmark function;

[0206] Step 405: Use the policy gradient formula of the REINFORCE algorithm to calculate the gradient of the Actor network to improve the policy performance:

[0207]

[0208] Step 406: Use the updated gradient to update the parameters of the Actor network;

[0209] Step 407: Update the state to the next state. Define the maximum number of updates N max , define the early stopping condition as the TD error δ k is less than the preset threshold of the positioning result change rate is less than the preset threshold, and repeat the above steps to perform A2C update when the number of updates has not reached N max and the early stopping condition is not met, and stop the update after reaching the above conditions;

[0210] Step 408: Output the final positioning based on the formula in step 4013

[0211] The present invention performs policy adjustment on the weights of the positioning obtained by IMM-EKF and the positioning obtained after correction by the large model through the A2C learning framework to obtain the final optimal positioning.

[0212] Figure 5 Illustrates a schematic physical structure diagram of an electronic device, as Figure 5 shown. The electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540. Among them, the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call the logical instructions in the memory 530 to execute an indoor navigation method based on multimodal data fusion and large model enhancement. The method includes: obtaining multimodal data through the user's mobile terminal; obtaining different modal navigation prediction results, and performing deep fusion on the different modal navigation prediction results by using IMM-EKF to obtain an IMM-EKF fusion positioning result; using an integrated large model to optimize and enhance the multimodal data to obtain an integrated large model corrected positioning result; using the A2C reinforcement learning framework strategy to regulate the output weights of the IMM-EKF fusion positioning result and the integrated large model corrected positioning result to determine the final navigation positioning result.

[0213] In addition, when the logical instructions in the aforementioned memory 530 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs, Read-Only Memories), random access memories (RAMs, Random Access Memories), magnetic disks, or optical discs that can store program codes.

[0214] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative labor.

[0215] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the technical solution, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium such as ROM / RAM, magnetic disk, optical disc, etc., and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments.

[0216] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention, and are not intended to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An indoor navigation method based on multi-modal data fusion and large model enhancement, characterized in that It includes: Obtain multi-modal data through the user's mobile device; Obtain different modal navigation prediction results, and use the interacting multiple model extended Kalman filter (IMM-EKF) to deeply fuse the different modal navigation prediction results to obtain the IMM-EKF fusion positioning result; Use an integrated large model to optimize and enhance the multi-modal data to obtain the integrated large model corrected positioning result; Utilize the A2C reinforcement learning framework strategy to regulate the output weights of the IMM-EKF fusion positioning result and the integrated large model corrected positioning result to determine the final navigation positioning result.

2. The indoor navigation method based on multi-modal data fusion and large model enhancement according to claim 1, characterized in that, Obtain multi-modal data through the user's mobile device, including: Periodically scan the received signal strength indication (RSSI) data of a preset Bluetooth beacon through the mobile device; Collect IMU data through the inertial measurement unit (IMU) built into the mobile device; Obtain environmental image data through the camera built into the mobile device, and the camera is authorized by the user to enable the navigation permission; The multi-modal data is composed of Bluetooth RSSI data, IMU data, and environmental image data.

3. The indoor navigation method based on multi-modal data fusion and large model enhancement according to claim 1, characterized in that, Obtain different modal navigation prediction results, and use the interacting multiple model extended Kalman filter (IMM-EKF) to deeply fuse the different modal navigation prediction results to obtain the IMM-EKF fusion positioning result, including: Define the state vector x at time k k , the state vector x k includes the position variable p k , the velocity variable v k and the attitude variable θ k : Predict the navigation through the driving of IMU data to obtain the IMU navigation prediction result; Predict the navigation through the driving of Bluetooth RSSI data to obtain the Bluetooth navigation prediction result; Predict the navigation through the driving of environmental image data to obtain the image navigation prediction result; Use IMM-EKF to deeply fuse the different modal navigation prediction results to obtain the IMM-EKF fusion positioning result.

4. The indoor navigation method based on multi-modal data fusion and large model enhancement according to claim 3, wherein, Predict the navigation through the driving of IMU data to obtain the IMU navigation prediction result, including: Obtain the acceleration a in the IMU data k and the angular velocity ω k ; Determine the non-linear model for x k|k-1 Make a prediction: x k|k-1 = f(x k-1 , u k ) where, u k represents the input of the IMU: f(x k-1 ,u k )Specifically: Calculate the Jacobian matrix F k : Obtain the predicted covariance: where Q k is the IMU noise covariance matrix: where Q IMU is the inherent noise covariance matrix of the IMU sensor.

5. The indoor navigation method based on multi-modal data fusion and large model enhancement according to claim 4, characterized in that, Predict the navigation through the driving of Bluetooth RSSI data to obtain the Bluetooth navigation prediction result, including: Obtain the position observation z from the RSSI ranging principle 蓝牙 ; Obtain the path loss model from the broadcast signal of the Bluetooth beacon and the RSSI received by the user device: RSSI = A - 10nlog 10 (d) + X σ Among them, A is the reference signal strength at 1 meter, which needs to be calibrated on-site, n is the path loss exponent, d is the actual distance between the Bluetooth beacon and the user terminal, and X σ is random noise; Convert the RSSI to the distance d: Obtain the position observation z through the distance d 蓝牙 : Define the Bluetooth observation function h 蓝牙 (x k ): z 蓝牙 = h 蓝牙 (x k ) + v 蓝牙 where v 蓝牙 is the observation noise; Select any 3 Bluetooth beacons in the system, p i is the coordinate of the i-th Bluetooth beacon: Input state vector x k The position component p k in it, through the Bluetooth observation function h 蓝牙 (x k ) calculates the theoretical Euclidean distance h from the user to each Bluetooth beacon 蓝牙 (x k ): Calculate the Jacobian matrix H 蓝牙 : Define the noise covariance R of Bluetooth 蓝牙 :

6. The indoor navigation method based on multimodal data fusion and large model enhancement according to claim 5, wherein Predict the navigation through the driving of environmental image data to obtain the image navigation prediction result, including: Extract feature points through ORB, solve the camera pose through the PnP algorithm, and convert it into the position p in the global coordinate system k ; Optimize and calculate the covariance ∑SLAM through the bundle adjustment (BA) method: ∑SLAM=(J T J) -1 Where J is the Jacobian matrix of BA; Define the environmental image data z 图像 The observation equation is as follows: z 图像 = p k + v 图像 where v 图像 is the observation noise; Define the observation function h 图像 (x k ): h 图像 (x k ) = C·p k Where C is the coordinate transformation matrix for rotation and translation from the SLAM coordinate system to the EKF global coordinate system; Calculate the Jacobian matrix H 图像 : Calculate R 图像 : where R 图像 The diagonal elements reflect the observation noise variances of the x / y / z axes respectively, and R 图像 only contains the position uncertainty of visual SLAM.

7. The indoor navigation method based on multi-modal data fusion and large model enhancement according to claim 6, wherein Use IMM-EKF to deeply fuse the different modal navigation prediction results to obtain the IMM-EKF fusion positioning result, including: Determine the motion models including uniform linear motion, uniformly accelerated linear motion, left turn motion, right turn motion, and stationary model, determine the total number of motion models as r, and i and j are any two indices of the motion models; Calculate the mixing probability from model i to model j, representing the contribution weight of model i at the previous moment to model j at the current moment; where p ij is the Markov transition probability from model i to model j, which is a constant matrix preset according to a typical motion model, and μ i (k - 1) is the probability of model i at the previous moment before time k; Weight the state estimates of each model at the previous moment of time k according to the mixing probability as the initial input of model j; Calculate covariance Perform state prediction: Among them, F j is the state transition matrix of model j, B j is the control input matrix, u j (k) is the IMU input of model j, A j (k) is the correction term: A j (k) = B j Δu j (k) Covariance prediction: where Q j is the process noise covariance of model j; Assume that the noises of Bluetooth and image data are independent, and update the joint observation of Bluetooth and image; Define z by combining Bluetooth observations and image observations k as the combined vector of Bluetooth and image data: Obtain: Calculate the Kalman gain: Among them, S j (k) is the innovation covariance matrix, representing the uncertainty between the observed predicted value and the actual observed value: Among them, H j is the augmented observation matrix of model j, and R j is the observation noise covariance of model j: Perform state update: x j (k|k) = x j (k|k - 1)+K j (k)(z k -H j x j (k|k - 1)) Covariance update: P j (k|k) = (I - K j (k)H j )P j (k|k - 1) Calculate the likelihood function: Calculate the matching degree between the current Bluetooth observation data and image observation data and the prediction of model j through the likelihood function for dynamic update; Calculate the probability of model j: Output the IMM-EKF fusion positioning result:

8. The indoor navigation method based on multi-modal data fusion and large model enhancement according to claim 1, wherein, Optimize and enhance the multi-modal data using a comprehensive large model to obtain the corrected positioning result of the comprehensive large model, including: Determine the multi-layer perceptron (MLP) structure: Input layer h0 = x where x is the input vector; There are a total of L layers, and the l-th hidden layer h l = σ l (W l h l-1 + b l ), where W l is the weight matrix, b l is the bias, and σ l is the activation function; Output layer \(y = \sigma\) out (W L h L-1 + b L ), where \(\sigma\) out is the output activation function; Use the MLP to perform drift compensation on the IMU of each model j. Define x during the process of using the MLP to perform drift compensation on the IMU k as follows: where T is the time step of the sliding window for capturing temporal information and needs to be preset in advance; Predict the acceleration Δa of the IMU k and the angular velocity deviation Δω k : Suppose an L-layer MLP is used with the ReLU activation function, and \(h_0 = x\) for the \(l\)th layer. k Then it unfolds as follows: The l-th layer: h l = ReLU(W l h l-1 + b l ) Output layer: For the initial a k and ω k Make corrections as follows: ω k corrected = ω k + Δω k Through μ j (k) determines the type of model j. If it is determined to be a uniform motion or a uniformly accelerated motion, the acceleration deviation is preferentially compensated. If it is determined to be a left-turning motion or a right-turning motion, the angular velocity deviation is preferentially compensated. If it is a stationary model, the compensation amount is ignored; Use Transformer to correct the visual observations of each model j: Input timing visual observation window {z 图像，k-T , ……, z 图像，k} and original image blocks {I k-T , ……, I k}, where T is a preset period; The formula of the multi-head self-attention mechanism (MSA) of Transformer is as follows: Among them, Q, K, and V are linearly projected from the input, and d k is the attention dimension; The positioning correction value Δz obtained from the computed environmental image data k : Δz k = MLP(MSA(z 图像，k-T:k , I k-T:k )) For z 图像 Make corrections to: Dynamically estimate the IMM-EKF noise parameters of each model j through LSTM: Define the input vector x of the LSTM k , x k includes the information of the current sensor and the probability μ of model j j (k): The LSTM updates the cell state c and the hidden state h through a gating mechanism k and the hidden state h k : Decided by the forget gate f k Determine how much information to discard from the previous cell state c k-1 How much information to discard: f k = σ(W f [h k-1 , x k + b f ) where σ uses the sigmoid function with an output range of [0, 1] to control the proportion of retained control information. W f is the weight matrix of the forgetting gate, and b f is the bias term; Determine the activation value i of the input gate k , control the writing ratio of new information, and define the candidate cell state Contain new information generated by the current input: i k = σW i [h k-1 , x k + b i Cell state update, from forgotten gate f k Selectively discard the old state c k-1 , input gate i k Selectively add new state Define o k Control the cell state c k How much information is input into the hidden state h k : o k = σ(W o [h k-1 , x k + b o ) h k = o k ⊙tanh(c k ) Calculate the noise parameters Q k and R k : Regulate the noise parameters through LSTM. If a k suddenly increases, it indicates intense movement and the IMU prediction is unreliable, then increase Q j,k ; If μ j (k) fluctuates violently, then evenly adjust the Q of each model j,k ; If z 蓝牙 fluctuation is greater than the threshold, then increase R 蓝牙 ; If z 图像 the solution calculation fluctuates greatly, then increase R 图像 ; If all inputs are stable, then restore the default Q j,k and R j,k ; The IMU of each model j is compensated by the MLP, and the Transformer is corrected to obtain The LSTM predicts Q of each model j in real time k and R k After that, these parameters adjusted by the large model are substituted back into the IMM-EKF for calculation, and finally the comprehensive large model corrected positioning result after the large model correction is obtained 9. The indoor navigation method based on multi-modal data fusion and large model enhancement according to claim 1, wherein, Use the A2C reinforcement learning framework policy to regulate the output weights of the IMM-EKF fusion positioning result and the corrected positioning result of the comprehensive large model to determine the final navigation positioning result, including: Determine the state s input to the Actor network at time k k : The Actor network outputs the action a k : α j Dynamically adjust Q j ,β j Dynamically adjust R j ,γ j Supplemental adjustment of the model fusion weights, dynamically adjusting the output weights of IMM-EKF and the large model: α j Increase the process noise of the IMU when α > 1 to adapt to high-speed motion; α j Reduce the process noise of the IMU and increase the weight of the IMU data when α < 1 to adapt to uniform motion; β j Adjust the confidence levels of the Bluetooth data and the environmental image data, β j When β decreases, rely more on Bluetooth observations and image observations, for Renormalize to make the sum equal to 1; Trust the IMM-EKF output result in areas with stable Bluetooth signals and rich visual features, and increase ξ; when the Bluetooth signal is unstable or when visual SLAM fails, use the state predicted by the large model based on IMU and LSTM to reduce ξ, and the A2C framework adjusts ξ in real time to achieve soft switching; The feature extraction layer of the Actor network takes s t as input to the Critic network, and the Critic network outputs the state value function Calculate the TD error δ k : δ k = r k + γV(S k+1 ) - V(S k ) r k is the current reward function, and γ is the discount factor: where is the true position, and tr(p(k|k)) represents the penalty for positioning uncertainty; When the true position is not available, multi-sensor consistency is used as a reward: r k = -λ1||z 蓝牙 -h 蓝牙 (x k )|| -λ2||z 图像 -h 图像 (x k )|| Update the parameters of the Critic network to reduce the TD error δ k ; Use the TD error δ k Calculate the advantage function A π (s k ,a k ): A π (s k ,a k )=Q π (s k ,a k )-B(s k ) Among them, Q π (s k , a k ) is the action-value function, and B(s k ) is the introduced baseline function. Then A π (s k , a k ) is the advantage function relative to the baseline function; Use the policy gradient formula of the REINFORCE algorithm to calculate the gradient of the Actor network: Use the updated gradient to update the parameters of the Actor network; Update the state to the next state and define the maximum number of updates N max , define the early stopping condition as the TD error δ k less than a preset threshold or the change rate of the positioning result less than a preset threshold and when the number of updates has not reached N max and the early stopping condition is not met, repeat the above steps to perform A2C update, and stop the update after reaching the above conditions; The output weight formula of the dynamically adjusted IMM-EKF and the large model is used to output the final navigation and positioning result 10. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, When the processor executes the program, it implements the indoor navigation method based on multi-modal data fusion and large model enhancement as described in any one of claims 1 to 9.

Images