Cross-scene multi-modal heterogeneous data fusion positioning method and system

Abstract

The invention provides a cross-scene multi-mode heterogeneous data fusion positioning method and system, electronic equipment, a storage medium and a program product, and aims to solve the problem of poor data alignment mobility, the method comprises the following steps: receiving sensor data at least comprising a wireless signal and a visual image, carrying out normalization and spatial interpolation processing, and generating a wireless signal image; pre-training the deep fusion network comprising the shared encoder and the CEN by using the annotation data of the source domain scene; the shared encoder is used for performing feature extraction on the two images and mapping the two images to a unified feature space, and the CEN is used for exchanging information between channels; using the pre-training model as priori knowledge, and based on a small number of samples of a target domain, calibrating feature distribution of the samples to obtain a positioning model; and inputting the to-be-processed multi-modal data into the positioning model, and outputting the position coordinates of the target. According to the invention, the space-time dimension can be aligned, the calculation complexity is reduced, and the ideal positioning performance is ensured.

Description

Technical Field

[0001] This disclosure relates to the field of communication technology, and in particular to a cross-scenario multimodal heterogeneous data fusion positioning method and system, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] Multimodal heterogeneous data refers to data with significantly different input structures, such as RGB images and wireless signals, 2D images and 3D images, audio and video, etc. Due to the large differences in structure and dimensionality, most current multimodal fusion algorithms struggle to process and fuse this type of data, resulting in problems such as dimensionality misalignment and loss of modal information. However, compared to multimodal homogeneous data, multimodal heterogeneous data exhibits better complementarity and lower redundancy. Multimodal heterogeneous data can be applied in many scenarios, such as: 1. Fusing visual tracking and map information for pedestrian dead reckoning. 2. Fusing Wi-Fi and magnetic field data and processing these fingerprint data as images. 3. Fusing normalized magnetic field strength, normalized Wi-Fi fingerprints, and images to generate a new image called RGB-WM, and then extracting homogeneous features from the RGB-WM image based on an improved CNN (Convolutional Neural Network) to achieve context awareness, etc.

[0003] Current mainstream cross-scene multimodal fusion localization algorithms mainly employ fingerprint feature matching mechanisms. However, these fingerprint-based localization technologies face challenges in practical deployment, such as long fingerprint database construction cycles and reduced fingerprint feature matching accuracy due to dynamic environmental changes. Therefore, constructing multimodal fusion localization algorithms with cross-scene adaptability has become an important research direction. Many current methods utilize techniques such as transfer learning (TL), enabling the basic model to quickly adapt to new scenarios with only a small number of samples. This optimization strategy significantly enhances the cross-scene adaptability of the localization system, providing a new research path for the evolution of indoor localization technology. The RSS (Received Signal Strength) fingerprint localization framework based on transfer learning can learn distance metrics from source domain data, then obtain a metric suitable for environmental changes by minimizing the data difference between the target domain and the source domain, and finally achieve fingerprint localization through a distance-based clustering algorithm. A CNN-based CSI (Channel State Information) fingerprint localization model was pre-trained using wireless signal data collected under various conditions. After environmental changes, the model was fine-tuned using a small number of real data samples to adapt to the new environment.

[0004] However, existing methods still have the following problems: 1. When processing large-scale data, the computational complexity of heterogeneous feature space construction and weight learning processes remains high, leading to longer localization response times and affecting user experience, especially in application scenarios requiring real-time localization. 2. The alignment and transferability of heterogeneous multimodal data is poor; low quality source domain data or significant differences from the target domain will affect the transfer effect. Summary of the Invention

[0005] To address the difficulties in fusion caused by inconsistent dimensions of multimodal heterogeneous data in existing technologies, as well as the poor alignment and transferability of heterogeneous multimodal data, and the impact of low-quality source domain data or significant differences from the target domain on transfer performance, this disclosure provides a cross-scene multimodal heterogeneous data fusion localization method and system, electronic device, computer-readable storage medium, and computer program product. It can align the spatiotemporal dimensions of different modalities, efficiently extract features using a shared encoder to reduce computational complexity, and use features from a pre-trained localization model (source domain) as prior knowledge to optimize and calibrate the feature distribution of samples (target domain) in the new scene, achieving ideal localization performance with only a small number of samples.

[0006] Firstly, this disclosure provides a cross-scenario multimodal heterogeneous data fusion localization method, the method comprising:

[0007] Data preprocessing steps: Receive sensor data from at least two different modalities, including wireless signals and visual images; normalize and spatially interpolate the wireless signal data to generate a wireless signal image, so that the wireless signal data and the visual image data are aligned in data dimension;

[0008] Model training steps: The deep fusion network is pre-trained using labeled data from the source domain scene to obtain a pre-trained model; the deep fusion network includes a shared encoder and a channel exchange network CEN(), the shared encoder is used to extract features from the wireless signal image and the visual image and map them to a unified feature space, and the CEN is used to exchange information between channels according to the importance of the features of each modality to a specific task;

[0009] Feature transfer calibration steps: In the target domain scene, the pre-trained model is used as prior knowledge, and the feature distribution of the target domain samples is calibrated based on a small number of samples in the target domain to obtain the localization model. The calibration includes distribution calibration based on statistical information and mapping calibration based on metric learning.

[0010] Localization steps: Input the preprocessed multimodal data collected in real time in the target domain scene into the localization model optimized by the feature transfer calibration step, and output the position coordinates of the target.

[0011] Furthermore, the normalization and spatial interpolation processing of the wireless signal data includes:

[0012] Normalize the RSSI (Received Signal Strength Indicator) value to the range of 0-255;

[0013] The normalized wireless signal data is spatially interpolated using the Kriging interpolation method to generate the wireless signal image.

[0014] Furthermore, the exchange of information between channels based on the importance of each modality's characteristics to a specific task includes:

[0015] For each feature channel of each modality, a trainable scaling factor is calculated, which is a factor that affects the multi-scale output of the channel;

[0016] The scaling factor is compared with a preset threshold.

[0017] If the scaling factor is greater than the threshold, the features of the channel are normalized according to standard procedures.

[0018] If the scaling factor is less than or equal to the threshold, the feature of that channel is replaced with the weighted average of the features of the corresponding channels of other modalities, so as to realize information exchange between channels.

[0019] Furthermore, in the model training step, the loss function used for pre-training the deep fusion network is the dual contrast loss function, which is a weighted sum of heterogeneous contrast loss and homogeneous contrast loss;

[0020] The heterogeneous contrast loss is used to narrow the distance between different modalities at the same location in the feature space;

[0021] The isomorphic contrast loss is used to bring the samples of the same type in the source and target domains closer together in the feature space, and to increase the distance between samples of different types.

[0022] Furthermore, the heterogeneous contrast loss function calculates the matching probability in two directions: image to wireless signal and wireless signal to image, and uses the cross-entropy loss function to maximize the similarity between image-wireless signal pairs from the same location.

[0023] Furthermore, the isomorphic contrast loss function adopts the interval loss function, the expression of which is:

[0024] ;

[0025] in, For isomorphic contrast loss, N is the batch size, n represents the nth sample pair in the batch, y is the similarity label of whether the two samples belong to the same category, d is the distance between the samples in the feature space, and margin is a preset interval value.

[0026] Furthermore, the distribution calibration based on statistical information includes:

[0027] Calculate the mean and covariance matrix of the features of each category of samples in the source domain;

[0028] Based on KL divergence, we search for categories in the source domain that are similar to the feature distribution of samples in the target domain;

[0029] The statistical information of the found categories is combined with the features of the target domain samples to obtain the calibrated feature distribution parameters;

[0030] Optimized features are obtained by sampling from the calibrated distribution.

[0031] Furthermore, the metric learning-based mapping calibration includes:

[0032] Select the most similar and least similar features from the source domain samples to the target domain samples;

[0033] Learn a transfer function such that, after mapping the target domain features, the transfer function minimizes the distance to similar source domain features and maximizes the distance to dissimilar source domain features.

[0034] The original target domain features are concatenated with the mapped features to obtain the final optimized features.

[0035] Furthermore, in the shared encoder, the visual image encoder adopts the DINO architecture based on self-supervised knowledge distillation, and the wireless signal image encoder adopts the improved LeNet-5 architecture, in which the first convolutional layer is replaced with a single-channel convolutional layer.

[0036] Secondly, this disclosure provides a cross-scenario multimodal heterogeneous data fusion positioning system, the system comprising:

[0037] The data acquisition and preprocessing module is configured to receive sensor data from at least two different modalities, including wireless signals and visual images, and to perform normalization and spatial interpolation processing on the wireless signal data to generate a wireless signal image so that the wireless signal data and the visual image data are aligned in the data dimension.

[0038] The pre-training module is configured to pre-train the deep fusion network using labeled data from the source domain scene to obtain a pre-trained model. The deep fusion network includes a shared encoder and a channel exchange network (CEN). The shared encoder is used to extract features from the wireless signal image and the visual image and map them to a unified feature space. The CEN is used to exchange information between channels based on the importance of the features of each modality to a specific task.

[0039] The feature transfer calibration module is configured to use the pre-trained model as prior knowledge in the target domain scene and calibrate the feature distribution of the target domain samples based on a small number of samples in the target domain to obtain a localization model. The calibration includes distribution calibration based on statistical information and mapping calibration based on metric learning.

[0040] The localization module is configured to input preprocessed multimodal data collected in real time in the target domain scene into the localization model optimized by the feature transfer calibration step, and output the position coordinates of the target.

[0041] Thirdly, this disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to execute the above-described cross-scene multimodal heterogeneous data fusion positioning method.

[0042] Fourthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the aforementioned cross-scenario multimodal heterogeneous data fusion and positioning method.

[0043] Fifthly, this disclosure provides a computer program product that includes computer-readable code or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described cross-scenario multimodal heterogeneous data fusion positioning method.

[0044] Beneficial effects:

[0045] This disclosure provides a cross-scenario multimodal heterogeneous data fusion positioning method and system, electronic device, computer-readable storage medium, and computer program product. It effectively solves the technical challenges of difficult heterogeneous data fusion, poor cross-scenario adaptability, and high computational complexity. It can align the spatiotemporal dimensions of different modalities, and through heterogeneous data spatiotemporal alignment and channel exchange fusion networks, it fully exploits the complementary characteristics of multimodal data, significantly improving positioning accuracy and robustness in complex environments. Secondly, the cross-domain feature transfer calibration mechanism enables the model to utilize prior knowledge from the source domain, requiring only a small number of samples to quickly adapt to the target domain, greatly reducing sample dependence and manpower costs in new scenarios, and achieving excellent cross-scenario generalization capabilities. Finally, the shared encoder and efficient fusion strategy significantly reduce computational complexity, ensuring the system's real-time response capability on terminal devices, and providing a feasible technical path for high-precision, low-cost, universal indoor positioning services.

[0046] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0047] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which:

[0048] Figure 1 This is a flowchart illustrating a cross-scenario multimodal heterogeneous data fusion positioning method provided in Embodiment 1 of this disclosure;

[0049] Figure 2 This is a schematic diagram of a cross-scenario multimodal heterogeneous data fusion positioning system framework provided in Embodiment 1 of this disclosure;

[0050] Figure 3 This is a schematic diagram of a heterogeneous multimodal deep fusion network architecture provided in Embodiment 1 of this disclosure;

[0051] Figure 4 This is a block diagram of a cross-scenario multimodal heterogeneous data fusion positioning system provided in Embodiment 2 of this disclosure;

[0052] Figure 5 This is a block diagram of an electronic device provided in Embodiment 3 of this disclosure. Detailed Implementation

[0053] To enable those skilled in the art to better understand the technical solutions of this disclosure, exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of this disclosure to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0054] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0055] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0056] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0057] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in common dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning unless expressly so defined herein. Those skilled in the art will understand that the specific order of execution of the steps in the methods described above in the specific embodiments should be determined by their function and possible internal logic.

[0058] Compared to localization based on multimodal data fusion, single-modal methods are more susceptible to disturbances from single-source sensors, leading to decreased positioning accuracy. By leveraging the complementary advantages of multimodal data, multimodal localization models not only improve positioning accuracy due to increased information volume but also remain unaffected by changes in individual modal data, thus enhancing the stability of the localization system. Furthermore, the continuous development of multimodal fusion technology and the expanding application of fusion algorithms have solved many problems that single-modal localization cannot address. Therefore, multimodal fusion-based localization methods have attracted considerable research attention and become one of the most popular localization methods today. Based on data type, this method can be divided into two categories: homogeneous multimodal fusion localization algorithms and heterogeneous multimodal fusion localization algorithms. Based on different scenarios, it can also be divided into single-scenario / specific-scenario multimodal localization algorithms and cross-scenario multimodal localization algorithms. Since single-scenario multimodal localization algorithms broadly refer to all localization models trained only in a single scenario and are difficult to apply across scenarios (most fingerprint-based methods are included in this category), cross-scenario multimodal fusion localization algorithms have gained wider application.

[0059] I. Multimodal isomorphic fusion localization algorithm

[0060] Multimodal isomorphic data refers to input data with the same or similar structures, such as RGB images and RGB-D images, Wi-Fi signals and Bluetooth signals, infrared images and X-ray images, etc. These data generally have similar data structures, and may even have the same data dimensions. Therefore, multimodal fusion algorithms based on this type of data are simpler and easier to implement.

[0061] The following are examples of multimodal isomorphic data fusion:

[0062] 1. By fusing visual information from RGB and RGB-D cameras with depth data, this approach leverages the strengths of different data types to provide a more robust and accurate localization solution, thereby improving localization performance. It is primarily applied to enhance the accuracy of Simultaneous Localization and Mapping (SLAM) for robot navigation and environmental mapping. Robustness in dynamic environments is improved by fusing depth information from individual RGB images with learned outlier masks. Convolutional Neural Networks (CNNs) are used for pose estimation and to generate outlier masks to handle unreliable regions caused by occlusion or dynamic objects.

[0063] 2. By fusing visual data with other modal data such as sonar or lidar, and using image segmentation and Gaussian Mixture Module (GMM), the system creates a comprehensive environmental map, improving the robot's ability to navigate and avoid obstacles in complex dynamic environments.

[0064] 3. Wireless signal fusion typically combines Wi-Fi, IMU, and UWB data, utilizing recurrent neural networks to process multimodal sequence data and improve positioning accuracy. Effective data fusion of Wi-Fi and RFID signals can be achieved by deploying Wi-Fi access points and RFID tags on demand, eliminating positioning blind spots and enabling low-cost, high-precision positioning.

[0065] Indoor positioning methods based on multimodal homogeneous data fusion have achieved good positioning results by fusing data from multiple similar sensors, but they still have shortcomings: the homogeneous data has high redundancy but insufficient complementarity, and the robustness of the positioning system will decrease significantly when a single signal source fails or the environment changes drastically.

[0066] (1) Multimodal heterogeneous fusion localization algorithm

[0067] Multimodal heterogeneous data refers to input data with significantly different structures, such as RGB images and wireless signals, 2D images and 3D images, audio and video, etc. Due to the large differences in structure and dimensionality, most current multimodal fusion algorithms struggle to process and fuse this type of data, resulting in problems such as dimensionality misalignment and loss of modal information. However, compared to multimodal homogeneous data, multimodal heterogeneous data exhibits better complementarity and lower redundancy.

[0068] Examples of multimodal heterogeneous data are as follows:

[0069] 1. Fusing visual tracking and map information for pedestrian dead reckoning. This method uses inertial sensors and cameras, employing visual data to assist smartphone inertial navigation, thus solving the drift accumulation problem in position estimation. Specifically, it fuses visual data with inertial navigation data and uses a CNN to detect pedestrians in recorded images, significantly improving positioning accuracy.

[0070] 2. Integrate Wi-Fi and magnetic field data, and process this fingerprint data as an image. High-resolution fingerprint images are constructed from time-series sensor data. By designing a custom localization model based on a CNN, this model has independent network branches to process Wi-Fi and magnetic field data, as well as a unified network branch. This network structure can effectively learn the relationship between the actual location and the fingerprint image.

[0071] 3. Normalized magnetic field strength, normalized Wi-Fi fingerprint and image are fused to generate a new image called RGB-WM, and then homogeneous features are extracted from the RGB-WM image based on an improved CNN to achieve context awareness.

[0072] (2) Cross-scene multimodal fusion localization algorithm

[0073] Current mainstream positioning algorithms primarily employ fingerprint feature matching mechanisms. However, these fingerprint-based positioning technologies face challenges in practical deployment, such as long fingerprint database construction cycles and reduced fingerprint feature matching accuracy due to dynamic environmental changes. Taking deep neural network-based positioning systems as an example, while single-scene positioning methods can achieve high positioning accuracy in specific scenarios, cross-scene positioning tasks require repeated fingerprint database construction and network parameter retraining, resulting in additional resource consumption. Therefore, constructing multimodal fusion positioning algorithms with cross-scene adaptability has become an important research direction. Many current methods utilize techniques such as transfer learning, enabling the base model to quickly adapt to new scenes with only a small number of samples. This optimization strategy significantly enhances the cross-scene adaptability of positioning systems, providing a novel research path for the evolution of indoor positioning technology.

[0074] The RSS fingerprint localization framework based on transfer learning (TL) learns a distance metric from source domain data, then obtains a metric suitable for environmental changes by minimizing the data difference between the target domain and the source domain, and finally achieves fingerprint localization through a distance-based clustering algorithm. A CNN-based CSI fingerprint localization model is pre-trained using wireless signal data collected under various conditions. After environmental changes, the model is fine-tuned using a small number of real data samples to adapt to the new environment. Specifically, firstly, the source domain is optimized by removing redundant knowledge from the source domain based on the target domain. Then, a cross-domain knowledge transfer framework is established to transfer features from the source domain to the target domain, achieving a unified representation of multimodal features. In the new feature space, a dynamic weight allocation strategy is used to train a classifier to predict the target location.

[0075] However, current multimodal heterogeneous data localization algorithms still have problems such as difficulty in fusion due to inconsistent dimensions of multimodal heterogeneous data, poor alignment and transferability of heterogeneous multimodal data, and low quality of source domain data or large differences from the target domain affecting the transfer effect.

[0076] To address the challenges of fusing multimodal heterogeneous data and the poor generalization of models across different scenarios, this disclosure proposes a localization framework based on multimodal heterogeneous data fusion across single and cross-scenario scenarios. By leveraging the complementarity between heterogeneous data and cross-domain feature transfer calibration, the localization and generalization performance of the model is improved.

[0077] The cross-scenario multimodal heterogeneous data fusion positioning method according to embodiments of this disclosure can be executed by electronic devices such as terminal devices or servers. Terminal devices can be in-vehicle devices, user equipment (UE), mobile devices, user terminals, terminals, cellular phones, cordless phones, personal digital assistants (PDAs), handheld devices, computing devices, in-vehicle devices, wearable devices, etc. The method can be implemented by a processor calling computer-readable program instructions stored in memory. Alternatively, the method can be executed by a server.

[0078] Example 1

[0079] Figure 1 This is a flowchart illustrating a cross-scenario multimodal heterogeneous data fusion and localization method provided in Embodiment 1 of this disclosure, with reference to... Figure 1 The method includes:

[0080] Step S101: Data preprocessing step: Receive sensor data from at least two different modalities, including wireless signals and visual images; normalize and spatially interpolate the wireless signal data to generate a wireless signal image so that the wireless signal data and the visual image data are aligned in the data dimension.

[0081] Step S102: Model training step: The deep fusion network is pre-trained using the labeled data of the source domain scene to obtain a pre-trained model; the deep fusion network includes a shared encoder and a channel exchange network CEN, the shared encoder is used to extract features from the wireless signal image and the visual image and map them to a unified feature space, and the CEN is used to exchange information between channels according to the importance of the features of each modality to a specific task;

[0082] Step S103: Feature transfer calibration step: In the target domain scene, the pre-trained model is used as prior knowledge, and the feature distribution of the target domain samples is calibrated based on a small number of samples in the target domain to obtain the localization model. The calibration includes distribution calibration based on statistical information and mapping calibration based on metric learning.

[0083] Step S104: Localization step: Input the preprocessed multimodal data collected in real time in the target domain scene into the localization model optimized by the feature transfer calibration step, and output the position coordinates of the target.

[0084] The localization algorithm in this disclosure mainly includes a multimodal heterogeneous data fusion algorithm and a cross-domain multimodal heterogeneous feature transfer calibration method. During the offline training phase, the system performs data sampling and model pre-training, using the pre-trained model as prior knowledge. In the online localization phase, the user sends the acquired data to a local server. Since the data is susceptible to spatiotemporal fluctuations, the feature distribution of the target domain samples needs to be calibrated. Based on the constructed fingerprint database and the calibrated data, the server fine-tunes the pre-trained model and predicts the target's location according to the corresponding fusion localization algorithm.

[0085] The framework of a cross-scenario multimodal heterogeneous data fusion positioning system, such as Figure 2 As shown, the core objective of the system is to address the difficulties in fusing multimodal heterogeneous data and the poor generalization of cross-scene models. It proposes a multimodal heterogeneous data fusion localization framework based on single-scene and cross-scene models, and improves localization and generalization performance by utilizing data complementarity and cross-domain feature transfer calibration.

[0086] The system mainly consists of a multimodal heterogeneous data fusion algorithm and a cross-domain multimodal heterogeneous feature transfer calibration method; including:

[0087] The offline training phase consists of two parts: data sampling and model pre-training. First, multimodal data (such as images and RSSI measurements [-86, -91, ...]) is collected using devices like cameras and RSSI sensors and stored in a database. Second, multimodal data is processed by general encoder 1 and general encoder 2 and then aligned to build a pre-trained model. This pre-trained model serves as prior knowledge for subsequent online localization.

[0088] The online positioning process includes:

[0089] Data input: New scene data (including images, RSSI, etc.) acquired by the user is sent to the local server. The data is susceptible to interference from spatiotemporal fluctuations.

[0090] Feature distribution calibration includes: Distribution calibration: calibrating the feature distribution of the target domain samples, optimizing the target domain feature distribution through the "Calibrate Distribution" module (e.g., feature adjustment from Source Domain Data to Target Domain Data to make the feature distribution more consistent). And, Feature value optimization: further optimizing the calibrated features to improve feature quality.

[0091] Model fine-tuning and localization prediction: Based on the constructed fingerprint database (including fingerprint features such as FR1, FR2...FRM) and calibrated data, the server uses a contrastive classifier to fine-tune the pre-trained model.

[0092] Similarity calculation: The target location L is predicted by calculating the similarity matrix (such as the similarity between features F11-FR1, F12-FR2, etc.) and combining it with the fusion localization algorithm.

[0093] The system enables multimodal data processing, employing universal encoder 1 and universal encoder 2 to process different modalities of data respectively, and achieving heterogeneous data fusion through "alignment". Cross-domain transfer calibration: Adapting the feature distribution between the source and target domains is achieved through "distribution calibration" and "feature value optimization," solving the cross-scenario generalization problem. Offline-online collaboration: The pre-trained model stores prior knowledge in the offline stage, while the online stage calibrates features and fine-tunes the model in real time, balancing efficiency and accuracy.

[0094] This positioning algorithm system, through a two-stage architecture of "offline pre-training - online calibration and fine-tuning", combined with multimodal fusion and cross-domain feature calibration, effectively solves the problems of difficulty in multimodal heterogeneous data fusion and poor cross-scene generalization, and achieves high-precision positioning.

[0095] The following describes the specific process of cross-scenario multimodal heterogeneous data fusion and localization, which mainly includes the following steps:

[0096] Data preprocessing step (S101): In the data preprocessing stage, the system collects multimodal sensor data through a mobile terminal. Specifically, this includes wireless signal acquisition: simultaneously acquiring Received Signal Strength Indication (RSSI) data from Bluetooth and Wi-Fi, concatenating the RSSI values ​​of the two data points according to a certain ratio (e.g., [0.3, 0.7]) to form a fusion vector; and performing normalization and spatial interpolation processing on the wireless signal data to generate a wireless signal image.

[0097] In a preferred embodiment, the normalization process can employ a min-max normalization method to linearly map RSSI values ​​from their original range (e.g., [-100dBm, -30dBm]) to the interval [0, 255]. Spatial interpolation can use ordinary kriging interpolation to generate a 32×32 pixel RSSI image within a 2m × 2m area centered on the acquisition point. The variogram uses a spherical model with a search radius of 1.5 meters.

[0098] Image data synchronization is performed: a timestamp alignment mechanism ensures that the time deviation between the visual image and the wireless signal data is less than 50ms. Images are uniformly adjusted to 224×224 pixels and histogram equalization is applied.

[0099] The above preprocessing stage solves the problem of dimensional mismatch in heterogeneous data, enabling RSSI signal and image data to be processed in the same network structure. Kriging interpolation fully utilizes the spatial correlation of RSSI data, improving data integrity and availability, while the time synchronization mechanism ensures the consistency of multimodal data in the time domain.

[0100] Before the data preprocessing step, a data acquisition step is also included: using an application on a mobile terminal to call the device's built-in inertial measurement unit (IMU), Wi-Fi module, and camera to synchronously acquire multimodal data; and using a timestamp alignment mechanism to synchronize the time of each modal data.

[0101] Model training step (S102): Pre-train the deep fusion network (also known as a heterogeneous multimodal deep fusion network) using labeled data from the source domain scene to obtain a pre-trained model; the deep fusion network includes:

[0102] 1. RSSI and image sharing encoder:

[0103] After aligning heterogeneous multimodal data, although RSSI images and visual images have similar data structures, the distribution of deep features among the multimodal data remains different. Furthermore, the large data space of heterogeneous multimodal data complicates feature computation. Additionally, both RSSI and image data exhibit spatial correlation, but traditional feature extraction networks cannot effectively extract spatial information.

[0104] To address these challenges, embodiments of this disclosure design a multimodal feature extraction network with a shared structure. Specifically, the original network is modified and optimized into a shared encoder structure. The two encoders extract RSSI and image features while sharing their parameters, uniformly mapping the multimodal data to a unified feature space. Finally, a unified feature representation of the multimodal data is obtained. Simultaneously, the computational complexity of the algorithm is reduced due to the narrowed feature search range in the information space. However, because the output feature dimension of the encoder is relatively large, a Long Short-Term Memory (LSTM) network is added after the encoder to reduce the dimensionality of the output features.

[0105] 2. Multi-timescale fusion network based on channel switching:

[0106] Since the quality of latent space feature representations affects model performance, and multimodal heterogeneous data possesses a larger latent space, the fusion of multimodal heterogeneous data is necessary and important. Traditional fusion strategies, such as splicing and element-wise summation, do not fully utilize the complementary properties of multimodal data. Specifically, since visual images do not contain NLOS noise, image data can effectively suppress NLOS noise when it exists in the channels. Simultaneously, RSSI data can play a dominant role when image data is affected by illumination, motion, and occlusion. Furthermore, due to different data sampling rates, many multimodal fusion networks cannot effectively match data across multiple time scales. There is no explicit correspondence between modalities, making multimodal fusion relatively complex. Moreover, many methods cannot determine the trade-off between fusing shared modal information and preserving modality-specific features. Most fusion networks may introduce noise and retain redundant information.

[0107] To address these issues, this embodiment incorporates a parallel convolutional kernel structure to extract multi-scale features based on the original network. Furthermore, to fully leverage the complementary information between network channels, a channel-swapping-based fusion network is inserted into the encoder channels. The channel-swapping network (CEN) exchanges information between channels based on the importance of each modality's features to a specific task, effectively utilizing the complementary characteristics of multimodal heterogeneous data and fully fusing the features of each channel. Moreover, the channel-swapping mechanism effectively matches different time scales. Finally, through the information exchange mechanism based on task importance, CEN can integrate the advantages of modality-specific and modality-shared features, suppressing noise.

[0108] The training of deep fusion networks based on multimodal heterogeneous data adopts a staged training strategy. The heterogeneous multimodal deep fusion network architecture is as follows: Figure 3 As shown, to extract spatial features from multimodal heterogeneous data and fully utilize the rich complementary relationships between modalities, the core modules are analyzed as follows:

[0109] 1. Modal Input Layer

[0110] Input data type:

[0111] Image modality: The input on the left is N q Images (such as scene pictures) provide visual spatial information through image data. RSSI modality: The input on the right is RSSI signal data of NAP (number of access points) and Ns (number of samplings), represented as a bar graph, providing the spatial distribution characteristics of wireless signal strength.

[0112] Modal characteristics: Images are high-dimensional visual data, while RSSI is low-dimensional temporal / numerical data. Their heterogeneity is complemented and fused through subsequent modules.

[0113] 2. Pre-trained encoding layer

[0114] Function: Performs independent feature extraction and pre-training on two modalities to provide initial feature representations for cross-modal interaction.

[0115] Structure: Image modalities and RSSI modalities are processed through independent pre-trained modules (green squares in the diagram). The pre-training process may include self-supervised learning within each modality (such as image masking modeling and RSSI temporal prediction). The pre-trained data is input into a shared encoder (yellow trapezoid in the diagram). The encoder may use a parameter-sharing neural network (such as the base layer of a Transformer) to map different modal data to a unified feature space, achieving preliminary modality alignment.

[0116] 3. Cross-modal interaction layer

[0117] Core Mechanism: Channel Swapping (red square in the image)

[0118] Function: Enables cross-modal information interaction between images and RSSI features, breaking down modal barriers.

[0119] Principle: The two modal features (image features and RSSI features) output by the shared encoder interact at the feature channel level through the "channel exchange" module. This may be achieved through cross-attention mechanisms, feature stitching, or channel shuffling, which can promote the flow of complementary information between modalities (such as using the distance attenuation characteristics of RSSI to assist in the spatial positioning of the image, or using the scene structure of the image to correct the noise interference of RSSI).

[0120] Multimodal alignment: The label "multimodal alignment" on the left indicates that this layer ensures the consistency of two modal features in semantic and spatial dimensions through pre-training and channel swapping (such as image features and RSSI features at the same location being close in feature space).

[0121] 4. Feature Fusion Layer

[0122] Fusion Method: Image features after channel swapping are fused element-wise with RSSI features through addition (+ sign). The feature vectors after cross-modal interaction are superimposed to form a fused feature representation. Advantages: Additive fusion can retain the key information of both modalities while avoiding feature dimensionality expansion, making it suitable for lightweight deployment in subsequent localization tasks.

[0123] 5. Locate the output layer

[0124] Multilayer Perceptron (MLP, blue square in the figure): It receives fused feature vectors, performs nonlinear transformation and dimensionality compression through fully connected layers, and finally outputs position coordinates L (such as two-dimensional or three-dimensional spatial coordinates).

[0125] Task objective: To map multimodal fusion features to physical spatial locations, thereby achieving accurate positioning based on multimodal data.

[0126] This heterogeneous multimodal deep fusion network employs heterogeneous modal independent pre-training and shared encoding: Two modalities first learn intra-modal features (such as image edges and textures, RSSI signal intensity distribution) through independent pre-training modules, then unify the feature space through a shared encoder, balancing modal specificity and commonality. Channel swapping enables cross-modal deep interaction: Unlike simple feature concatenation, the channel swapping module promotes intermodal information complementarity through fine-grained channel-level interaction, improving the robustness of feature representation (e.g., using images to correct the multipath effect of RSSI, or using RSSI to enhance image scale invariance). End-to-end multimodal localization is achieved: from raw heterogeneous data input to position coordinate output, an end-to-end process of "input-encoding-interaction-fusion-output" is formed, avoiding intermediate manual feature engineering and improving localization accuracy and generalization ability.

[0127] Dataflow logic, image modality: N q Image → Pre-training → Shared encoder → Image features; RSSI modality: NAP / Ns signal → Pre-training → Shared encoder → RSSI features; Cross-modal interaction: Image features and RSSI features exchange information through channel exchange; Feature fusion: The interacting features are fused into a unified feature vector through addition; Localization output: The fused features are input into a multilayer perceptron to predict the location coordinates L.

[0128] This heterogeneous multimodal deep fusion network architecture uses dual-modal inputs of images and RSSI, achieves modal alignment through independent pre-training and a shared encoder, performs cross-modal feature interaction through a channel switching module, and then inputs the resulting data into a multilayer perceptron to output the position coordinates L. By independently encoding heterogeneous modalities and deeply interacting with them, the complementary nature of visual and wireless signals is effectively utilized to improve positioning accuracy.

[0129] The detailed configuration and parameters of the heterogeneous multimodal deep fusion network are shown in Table 1.

[0130] Table 1: Detailed configuration and parameters of heterogeneous multimodal deep fusion networks

[0131]

[0132] Network architecture configuration:

[0133] The shared encoder is based on an improved GoogLeNet architecture and contains 5 Inception modules; a channel exchange network (CEN) is inserted after each Inception module, and the threshold θ is set to 0.15; feature dimensions: the convolutional layers output 1024-dimensional features, and the LSTM layers reduce the dimensions to 256-dimensional.

[0134] Training parameter settings: The Adam optimizer is used, with an initial learning rate of 0.001, decaying by 0.5 every 50 epochs; batch size is set to 32, and the number of training epochs is 200; the weight α for the two-contrast loss is 0.7, and the margin for the interval loss is 1.0.

[0135] Training process: In the source domain scenario (such as an already deployed office building), 5000 labeled location points are used for training. Each location point contains 10 sets of multimodal data samples, for a total of 50,000 training samples.

[0136] The technical effects of this training phase:

[0137] The shared encoder significantly reduces the number of model parameters through parameter sharing, reducing computation by 40% compared to traditional two-branch networks.

[0138] The CEN network reduces positioning errors by approximately 35% in NLOS environments through dynamic channel switching.

[0139] The dual-contrast loss ensures consistency in the feature space, enabling cross-modal retrieval accuracy to reach 88%.

[0140] Feature transfer calibration step (S103): This involves feature calibration of the cross-scene multimodal localization model. In the target domain scene, the pre-trained model is used as prior knowledge, and the feature distribution of the target domain samples is calibrated based on a small number of samples from the target domain.

[0141] Due to the dimensionality differences between multimodal heterogeneous data, it is difficult to transfer spatial information from multimodal features. Therefore, embodiments of this disclosure further employ a multimodal heterogeneous alignment method to reduce the dimensionality gap and ensure effective transfer of multimodal information.

[0142] After the above processing, aligned multimodal features are obtained. Next, to address the issues of small sample sizes and cross-scene generalization, it is necessary to utilize both source and target domain data simultaneously. Based on metric learning and transfer learning, this disclosure innovatively proposes a cross-domain feature transfer calibration framework. Transfer learning can improve the learning performance for new tasks by transferring information from relevant original tasks. By utilizing prior knowledge from the source domain, the localization model can effectively calibrate the feature distribution of the target domain and perform localization in new scenes, improving the cross-scene generalization performance of the localization model. This calibration framework aims to reduce the overhead of in-situ sampling while maintaining ideal localization accuracy.

[0143] A preferred process for implementing feature transfer calibration in a new scenario (such as a shopping mall) includes:

[0144] Statistical Information Calibration Process: Calculate the feature statistics for each reference point in the source domain. Select the three source domain scenes most similar to the target domain environment. Calculate the mean and covariance matrix of the 256-dimensional feature vector for each reference point. Distribution Similarity Measurement: Use Jensen-Shannon divergence instead of KL divergence to improve stability. Set a similarity threshold of 0.3 and select the five most similar source domain distributions. Distribution Transformation: Employ Box-Cox transformation to make the feature distribution closer to a Gaussian distribution. Set the calibration parameter γ to 0.01 to control the degree of distribution dispersion.

[0145] Metric learning calibration process: Similarity feature selection: Cosine similarity is used to search in the source domain, and the top-10 similar features and bottom-10 dissimilar features are selected. Transfer function design: A two-layer fully connected network is used as the transfer function; Triplet Loss is used for optimization, and the boundary value is set to 0.5.

[0146] This feature transfer calibration only requires 50 labeled samples in the target domain to achieve a localization accuracy of 85%. The intra-class distance of the calibrated feature distribution is greatly reduced, while the inter-class distance is increased. The localization error in the new scene is reduced from 3.5 meters before calibration to 1.2 meters.

[0147] Localization step (S104): Input the preprocessed multimodal data collected in real time in the target domain scene into the localization model optimized by the feature transfer calibration step, and output the position coordinates of the target.

[0148] In the preceding steps, the preprocessed multimodal data first passed through two independent encoders. Then, the outputs of these two encoders were fed into a feature calibration model to optimize the cross-domain feature distribution. Next, a fusion network combined the optimized features to obtain a multimodal feature representation. Finally, a regressor further mapped the fused features to the predicted locations.

[0149] Then, based on the calibration method and loss function, multimodal heterogeneous feature calibration and fusion models are designed respectively. First, the encoder output is input into the calibration model. Prior knowledge of the source domain is used to optimize the feature distribution of the calibration target domain. Next, contrastive learning is used to learn the complementary relationships between the optimized multimodal heterogeneous features. Finally, the fusion model uses nonlinear transformation to calculate the fused features of the output multimodal heterogeneous features.

[0150] During localization, the network ultimately maps the multimodal fusion features to the predicted location using a multilayer perceptron with a regressor. Specifically, a linear projection from the multimodal fusion features to the location space is employed. Represented as a mapping matrix, This is the offset term. The predicted position can be represented as follows:

[0151]

[0152] in It is the predicted location. It is a multimodal fusion feature.

[0153] It is a 3×1 vector representing the coordinates of the target predicted by the model in three-dimensional space.

[0154] Specific form: Usually represented as [x, y, z] T Here, x and y are planar coordinates, and z represents the floor or height information. For purely two-dimensional planar positioning, it can also be [x, y]. T (at this time ).

[0155] (Multimodal fusion feature), meaning: This is a d i The high-dimensional vector. It is the core output of the entire network and is the final result after the following complex processing:

[0156] Heterogeneous data alignment (step S101), shared encoder feature extraction, channel-switching network fusion (step S102), cross-domain feature transfer calibration (step S103). This vector contains all the compressed and abstracted information used to determine the location. It can be understood as a "feature fingerprint" describing the current location.

[0157] W (mapping matrix or weight matrix) is a linear transformation matrix that maps a high-dimensional feature space to a three-dimensional physical space. Its function is to determine the weights of the fused features. It measures the contribution of each dimension to the final x, y, z coordinates. Essentially, it learns how to "interpret" the eigenvectors and "translate" them into coordinates.

[0158] b (offset or bias term), dimension This is a constant vector. Its function is to adjust the reference point of the output. It can be understood as the offset of the origin of the coordinate system of the entire positioning area. It ensures that even if all input features are 0, the output position is not necessarily (0,0,0).

[0159] This disclosure enables the alignment of spatiotemporal dimensions of different modalities. Through heterogeneous data spatiotemporal alignment and channel exchange fusion networks, it fully leverages the complementary characteristics of multimodal data, significantly improving positioning accuracy and robustness in complex environments. Secondly, the cross-domain feature transfer calibration mechanism allows the model to utilize prior knowledge from the source domain, enabling rapid adaptation to the target domain with only a small number of samples. This greatly reduces sample dependence and manpower costs in new scenarios, achieving excellent cross-scenario generalization capabilities. Finally, the shared encoder and efficient fusion strategy significantly reduce computational complexity, ensuring real-time response capabilities on terminal devices and providing a feasible technical path for high-precision, low-cost, and universal indoor positioning services.

[0160] Furthermore, the normalization and spatial interpolation processing of the wireless signal data includes:

[0161] Normalize the Received Signal Strength Indicator (RSSI) value to the range of 0-255;

[0162] The normalized wireless signal data is spatially interpolated using the Kriging interpolation method to generate the wireless signal image.

[0163] Heterogeneous multimodal data alignment, similar to data normalization preprocessing, adjusts the range of acquired RSSI data values ​​to 0-255, the same range as image pixel values, ensuring both types of data have equal task weight for localization. Without this preprocessing, features with higher weights will have a greater impact on the model results, potentially leading to decreased localization accuracy. The formula for this preprocessing is as follows:

[0164]

[0165] in It equals 255. The RSSI value of the sample. and These are the maximum and minimum values ​​of RSSI, respectively. This is the preprocessed RSSI.

[0166] The wireless signal data includes Bluetooth Received Signal Strength Indicator (RSSI) and Wi-Fi Received Signal Strength Indicator (RSSI). Considering the possibility of RSSI being unavailable due to AP malfunctions, the Bluetooth RSSI value and the Wi-Fi RSSI value are concatenated to form a fused wireless signal vector, which is then subjected to normalization and spatial interpolation. A method of concatenating the two data types at a certain ratio is adopted to reduce the impact of this problem. The expression of the concatenated data vector is as follows:

[0167]

[0168] in This represents the RSSI value of Bluetooth, while This indicates the RSSI value of Wi-Fi.

[0169] RSSI is a spatially correlated variable. Due to its spatial correlation, the RSSI value of a location can be approximated by measurements from neighboring locations. Specifically, Kriging interpolation is used to extend the dimensionality of the RSSI data. Kriging interpolation is a stochastic process interpolation regression algorithm based on a covariance function. It is suitable for this localization task because it can handle spatially correlated regional variables. After interpolation, RSSI data in image form is obtained, which will be referred to as RSSI images. The expression for Kriging interpolation is as follows:

[0170]

[0171] in It is a location The approximate RSSI value at that location. These are Kriging weights, which are spatially correlated. It is the variation function value between adjacent positions. It represents the number of sample locations. It is the distance between adjacent positions. It is the introduced Lagrange multiplier.

[0172] Furthermore, the exchange of information between channels based on the importance of each modality's characteristics to a specific task includes:

[0173] For each feature channel of each modality, a trainable scaling factor is calculated, which is a factor that affects the multi-scale output of the channel;

[0174] The scaling factor is compared with a preset threshold.

[0175] If the scaling factor is greater than the threshold, the features of the channel are normalized according to standard procedures.

[0176] If the scaling factor is less than or equal to the threshold, the feature of that channel is replaced with the weighted average of the features of the corresponding channels of other modalities, so as to realize information exchange between channels.

[0177] CEN is added to each encoder channel to fuse multi-timescale features. If a channel of one modality has a small impact on the final position prediction, the network replaces the original features with the mean of features from other modalities. The principle of this network is as follows:

[0178]

[0179] in, It is a characteristic of mode l in channel c. and These are the average and standard deviation of pixel positions across all active layers, respectively. and These are the trainable scaling factor and the offset, respectively. It is an arbitrarily small constant to avoid division by zero. It represents the number of modes. It is a factor that affects the multi-scale output of channel c. If , It will lose its influence on location prediction.

[0180] For each modality l∈{R, I}, where R represents the RSSI modality and I represents the image modality, each feature channel c∈{1,2,…,C} of l, where C is the total number of channels, and m∈{1,2,…,M} where M is the total number of location samples. The feature map of the RSSI mode. Feature maps of image modalities

[0181] Through CEN, it can be achieved

[0182] 1. Adaptive Feature Selection: Important Channel Preservation: When At that time, this channel was considered important for the positioning task and its unique information was preserved;

[0183] Redundant channel replacement: when When the information in this channel is redundant or unimportant, it is replaced with complementary information from other modes.

[0184] 2. Multimodal complementary utilization: When the image is occluded: ⇒ Information is obtained from RSSI modes;

[0185] When the signal is interfered with: ⇒ Obtain information from image modalities;

[0186] 3. Improved computational efficiency:

[0187] By exchanging information at the channel level, the computational complexity of fully connected fusion is avoided;

[0188] Dynamically select important features to reduce redundant calculations.

[0189] Furthermore, in the model training step, the loss function used for pre-training the deep fusion network is the dual contrast loss function, which is a weighted sum of heterogeneous contrast loss and homogeneous contrast loss;

[0190] The heterogeneous contrast loss is used to narrow the distance between different modalities at the same location in the feature space;

[0191] The isomorphic contrast loss is used to bring the samples of the same type in the source and target domains closer together in the feature space, and to increase the distance between samples of different types.

[0192] Pre-training of a cross-scene multimodal localization model. To address the small sample size problem, a pre-trained model is needed to learn a transferable depth distance metric. Therefore, a deep fusion network was trained end-to-end on a fingerprint database of the source domain. Furthermore, a cross-domain loss function was designed based on contrastive learning to leverage the correlations between multimodal heterogeneous data.

[0193] To address the challenge of feature learning in small sample sizes, this embodiment innovatively constructs a dual-contrast loss mechanism based on contrastive learning. Contrastive learning is a self-supervised method that learns useful feature representations by comparing the differences and similarities between samples. The core idea is to make similar samples closer in the feature space, while increasing the distance between different samples. First, a heterogeneous contrastive loss is designed at the modality level to address the multimodal semantic difference problem by forcing the feature spaces of images and Wi-Fi data to align. Then, a homogeneous contrastive loss is introduced at the cross-domain level to establish a cross-domain feature similarity measure. This contrastive loss architecture not only utilizes the complementary characteristics of multimodal heterogeneous data but also lays a transferable representation foundation for subsequent feature transfer calibration methods by establishing cross-domain feature associations.

[0194] To train the localization model, it is necessary to match the feature representations of multimodal heterogeneous data from the same reference point. This disclosure employs a cross-modal similarity matrix. Based on the principle of contrastive learning, the feature similarity between heterogeneous data from the same reference point is maximized. Conversely, for data from different reference points, the feature similarity between the data is minimized.

[0195] Combining heterogeneous and homogeneous contrastive loss functions, the total loss function L is the heterogeneous contrastive loss. Comparison loss with isomorphism The weighted sum. These are the weight hyperparameters. The formula is shown below:

[0196]

[0197] This joint optimization mechanism provides a metric space basis for the design of subsequent feature transfer calibration methods. By forcing inter-modal semantic alignment and cross-domain feature similarity learning, a transferable feature representation is established.

[0198] Furthermore, the heterogeneous contrast loss function calculates the matching probability in two directions: image to wireless signal and wireless signal to image, and uses the cross-entropy loss function to maximize the similarity between image-wireless signal pairs from the same location.

[0199] The expression for heterogeneous contrast loss is as follows:

[0200]

[0201] in, , Indicates the matched tag. , It refers to the similarity between samples from two modalities. CE stands for InfoNCE loss. This means "For all image-wireless signal pairs (I, R) randomly selected from data distribution D, calculate the average of the content in the square brackets following them".

[0202] Image-to-wireless signal matching tags: ∈{0,1} N N is the batch size; wireless signal to image matching tags: ∈{0,1} N ;

[0203] Similarity probability distribution, the probability of similarity between an image and a wireless signal: ∈RN, Probability of similarity between wireless signals and images: ∈RN, .

[0204] InfoNCE loss, .

[0205] This loss function, through bidirectional InfoNCE loss, ensures that: image-wireless signal pairs at the same location have high similarity in the feature space; and image-wireless signal pairs at different locations have low similarity in the feature space; thus achieving cross-modal feature alignment and laying the foundation for subsequent multimodal fusion.

[0206] Furthermore, the isomorphic contrast loss function adopts the interval loss function, the expression of which is:

[0207] ;

[0208] in, For isomorphic contrast loss, N is the batch size, n represents the nth sample pair in the batch, y is the similarity label of whether the two samples belong to the same category, d is the distance between the samples in the feature space, and margin is a preset interval value.

[0209] The purpose of isomorphic contrastive loss is to measure the difference between samples in the source and target domains. This loss enables the network to learn a mapping relationship. Based on prior knowledge of the source domain, it maps the feature vectors of the target domain from the original space to a new space, making the vectors have better topological properties in the new space, i.e., their distances within a class are as compact as possible, and their distances between different classes are as far apart as possible. By optimizing the isomorphic contrastive loss, the bound of the localization generalization error can be effectively reduced, and the localization accuracy of the model can be improved. The expression is as follows:

[0210]

[0211] It is a similarity function. It is Hinge's loss.

[0212] The isomorphic contrast loss in this disclosure directly optimizes the geometry of the feature space, enabling the model to learn the "relative relationships" and "discriminative features" of locations. In the source domain (old scene), the model learns how to distinguish different locations. When migrating to the target domain (new scene), although the absolute values ​​of wireless signal strength and visual features may change completely, the discriminative principle that "features at different locations should be different" remains unchanged. Utilizing this principle, the model can quickly re-establish location discrimination boundaries in the feature distribution of the new scene.

[0213] Furthermore, these samples can be used efficiently for model adaptation (fine-tuning).

[0214] Pre-training phase: Isomorphic contrastive loss and heteromorphic contrastive loss work together to shape an initial feature space that is highly discriminative and modally aligned.

[0215] Calibration Phase: During feature calibration in the target domain, the well-structured feature space, aided by isomorphic contrastive loss, makes "metric learning-based mapping calibration" more effective. This is because samples from both the source and target domains already possess clear intra-class aggregation and inter-class separation characteristics in the feature space, making the transfer function easier to learn.

[0216] This disclosed embodiment improves the model's cross-scene generalization ability and few-shot learning efficiency through isomorphic contrastive loss. Mechanistically, it creates a highly discriminative feature space through the principle of margin maximization. Systematically, it is deeply coupled with modules such as heterogeneous contrastive loss and feature transfer calibration, forming a powerful and robust solution that effectively addresses the core pain points of fingerprint positioning technology, namely, high deployment costs and poor environmental adaptability.

[0217] Furthermore, the distribution calibration based on statistical information includes:

[0218] Calculate the mean and covariance matrix of the features of each category of samples in the source domain;

[0219] Based on KL divergence, we search for categories in the source domain that are similar to the feature distribution of samples in the target domain;

[0220] The statistical information of the found categories is combined with the features of the target domain samples to obtain the calibrated feature distribution parameters;

[0221] Optimized features are obtained by sampling from the calibrated distribution.

[0222] For the unified calibration of feature distribution statistics, since the statistics for each class include the mean and variance, both are related to semantic similarity. That is, adjacent RP data have similar statistical data. If the sample size is large, the statistical information of the source domain data can be estimated more accurately. Based on transfer learning, statistical information can be transferred from the source domain to the target domain. The specific steps are as follows.

[0223] First, calculate the statistical information of the features in the source domain. Assuming the feature distribution in the source domain is Gaussian, then the features originating from the reference point... The mean and covariance matrices of the features can be expressed as:

[0224]

[0225] in Reference point The number of samples. Reference point The j-th sample is a feature. A shrinkage estimation method is used to adjust the covariance matrix to prevent overfitting to small samples.

[0226] Next, Tukey's power series transformation is used to transform the feature distribution, the similarity between samples is calculated based on KL divergence, and features with similar distributions are found in the source domain.

[0227]

[0228] in and These are the probability distributions of samples P and Q. and It is a probability density function. It can be used as a difference between two domains.

[0229] Then, the statistical information is transferred from the source domain to the target domain. The mean and covariance of the features in the target domain are calibrated by finding statistical information with similar distributions in the source domain, as expressed below:

[0230]

[0231] in It is a parameter that adjusts the dispersion of the calibrated features.

[0232] Finally, sampling is performed from the calibrated distribution to obtain... The optimized features are as follows:

[0233]

[0234] Distribution calibration based on statistical information can improve small sample adaptability by compensating for insufficient target domain data through source domain statistical priors; it can also enhance cross-scenario generalization performance, improve distribution alignment, significantly reduce the JS divergence of source and target domain feature distributions, and improve positioning error; it can also optimize computational efficiency and strengthen anti-interference ability by automatically identifying and excluding abnormal samples through statistical hypothesis testing, thus achieving modality missing fault tolerance.

[0235] The combination of statistical calibration and channel exchange network (CEN) provides high-quality feature distribution, which CEN then finely integrates. The calibrated feature distribution makes CEN's channel importance judgment more accurate. It complements contrast loss, which optimizes the feature space geometry, and statistical calibration further finely adjusts the distribution level. Together, they ensure the transferability of features across different scenarios. In conjunction with the regressor, the calibrated feature distribution makes the regressor learning more stable, reduces the impact of outliers on the final location prediction, and improves the smoothness and continuity of the localization results.

[0236] This embodiment uses a distribution calibration method based on statistical information. By making full use of prior knowledge from the source domain, it effectively solves the problem of model overfitting in small sample scenarios and provides a stable and reliable feature foundation for cross-scenario localization.

[0237] Furthermore, the metric learning-based mapping calibration includes:

[0238] Select the most similar and least similar features from the source domain samples to the target domain samples;

[0239] Learn a transfer function such that, after mapping the target domain features, the transfer function minimizes the distance to similar source domain features and maximizes the distance to dissimilar source domain features.

[0240] The original target domain features are concatenated with the mapped features to obtain the final optimized features.

[0241] After calibrating the statistical data in both domains, it is necessary to further use a transfer function based on metric learning to calibrate the feature distribution. This is because the boundaries between samples from different reference points RP in the feature space are fuzzy and uncertain, making it difficult for the model to learn an accurate feature distribution.

[0242] To effectively calibrate the feature distribution, transferable knowledge is defined in the form of a transfer function, which can reshape the original feature distribution into one more suitable for model learning. Furthermore, metric learning-based methods can further understand the feature relationships between the source and target domains. Samples from the same reference group (RP) or adjacent RPs have closer metric distances in the feature space. Conversely, samples from different RPs are more distant. The detailed process consists of the following two steps:

[0243] First, the features in the target domain are defined as follows: Based on a similarity algorithm, a feature is selected from the source domain that is most similar to a feature in the target domain and features of its neighboring RPs. Several most similar features are defined as... The least dissimilar feature is defined as .

[0244] Next, based on metric learning and transfer learning, a transfer function is used. This is used to represent feature mapping. Based on the principles of metric learning, a metric space needs to be learned, and a distance metric function needs to be constructed. This function can effectively distinguish different locations of samples. Therefore, a transfer function is defined. The expression is as follows:

[0245]

[0246] The expression for the calibrated feature space is:

[0247]

[0248] in It is the final optimized feature of the target domain. It consists of the original sample and the calibrated sample.

[0249] After this processing, unlike the original feature space, the new feature space simultaneously takes into account the discriminative power of both domains. Finally, it is input into the pre-trained model for fine-tuning, giving the localization model stronger generalization performance.

[0250] Furthermore, in the shared encoder, the visual image encoder adopts the DINO architecture based on self-supervised knowledge distillation, and the wireless signal image encoder adopts the improved LeNet-5 architecture, in which the first convolutional layer is replaced with a single-channel convolutional layer.

[0251] Two general encoders were designed for both RSSI and image data, which are heterogeneous datasets. Both encoders are capable of extracting depth distance metrics.

[0252] For image data, a general visual encoder should learn lightweight features with spatial information. Furthermore, visual encoders need to address the problem of small sample sizes. For these reasons, traditional visual encoders cannot extract effective and transferable features. Therefore, embodiments of this disclosure use the state-of-the-art "DINO" architecture as an image feature encoder. DINO is a self-supervised learning method that utilizes unlabeled knowledge distillation for pre-training. This encoder can effectively extract general visual features and solve the overfitting problem in cases of scarce training samples. Since DINO typically does not require labeled data, this reduces the dependence on labeled data and improves the encoder's generalization ability.

[0253] For RSSI data, in order to simulate the temporal dependence of continuous RSSI data, LeNet-5 is used as the feature encoder for RSSI, but the first layer of LeNet-5 is replaced with a single-channel convolutional network because RSSI has a single-channel data structure, while images do not have three channels: R, G, and B.

[0254] The metric learning-based mapping calibration method significantly improves the performance and practicality of cross-scene positioning systems by establishing an accurate sample relationship model and optimizing feature space transformation.

[0255] This disclosure achieves full fusion of heterogeneous data through spatiotemporal alignment and channel exchange fusion networks, fully leveraging the complementary characteristics of multimodal data and significantly improving positioning accuracy and robustness in complex environments. Utilizing dual-contrast loss and cross-domain feature transfer calibration mechanisms, the model can quickly adapt to the target domain with only a small number of samples, greatly reducing deployment costs in new scenarios and achieving excellent cross-scenario generalization capabilities. Simultaneously, the shared encoder and efficient fusion strategy significantly reduce computational complexity, ensuring real-time system response and providing a complete and feasible technical solution for high-precision, low-cost, universal indoor positioning services.

[0256] Example 2

[0257] This disclosure also provides a cross-scenario multimodal heterogeneous data fusion positioning system, such as... Figure 4 As shown, the system includes:

[0258] The data acquisition and preprocessing module 11 is configured to receive sensor data from at least two different modalities, including wireless signals and visual images, and to perform normalization and spatial interpolation processing on the wireless signal data to generate a wireless signal image so that the wireless signal data and the visual image data are aligned in the data dimension.

[0259] The pre-training module 12 is configured to pre-train the deep fusion network using labeled data from the source domain scene to obtain a pre-trained model. The deep fusion network includes a shared encoder and a channel exchange network (CEN). The shared encoder is used to extract features from the wireless signal image and the visual image and map them to a unified feature space. The CEN is used to exchange information between channels based on the importance of the features of each modality to a specific task.

[0260] The feature transfer calibration module 13 is configured to use the pre-trained model as prior knowledge in the target domain scene and calibrate the feature distribution of the target domain samples based on a small number of samples in the target domain to obtain a localization model. The calibration includes distribution calibration based on statistical information and mapping calibration based on metric learning.

[0261] The positioning module 14 is configured to input preprocessed multimodal data collected in real time in the target domain scene into the positioning model optimized by the feature transfer calibration step, and output the position coordinates of the target.

[0262] Furthermore, the data acquisition and preprocessing module 11 is specifically configured as follows:

[0263] Normalize the Received Signal Strength Indicator (RSSI) value to the range of 0-255;

[0264] The normalized wireless signal data is spatially interpolated using the Kriging interpolation method to generate the wireless signal image.

[0265] Furthermore, the exchange of information between channels based on the importance of each modality's characteristics to a specific task includes:

[0266] For each feature channel of each modality, a trainable scaling factor is calculated, which is a factor that affects the multi-scale output of the channel;

[0267] The scaling factor is compared with a preset threshold.

[0268] If the scaling factor is greater than the threshold, the features of the channel are normalized according to standard procedures.

[0269] If the scaling factor is less than or equal to the threshold, the feature of that channel is replaced with the weighted average of the features of the corresponding channels of other modalities, so as to realize information exchange between channels.

[0270] Furthermore, in the model training step, the loss function used for pre-training the deep fusion network is the dual contrast loss function, which is a weighted sum of heterogeneous contrast loss and homogeneous contrast loss;

[0271] The heterogeneous contrast loss is used to narrow the distance between different modalities at the same location in the feature space;

[0272] The isomorphic contrast loss is used to bring the samples of the same type in the source and target domains closer together in the feature space, and to increase the distance between samples of different types.

[0273] Furthermore, the heterogeneous contrast loss function calculates the matching probability in two directions: image to wireless signal and wireless signal to image, and uses the cross-entropy loss function to maximize the similarity between image-wireless signal pairs from the same location.

[0274] Furthermore, the isomorphic contrast loss function adopts the interval loss function, the expression of which is:

[0275]

[0276] in, For isomorphic contrast loss, N is the batch size, n represents the nth sample pair in the batch, y is the similarity label of whether the two samples belong to the same category, d is the distance between the samples in the feature space, and margin is a preset interval value.

[0277] Furthermore, the distribution calibration based on statistical information includes:

[0278] Calculate the mean and covariance matrix of the features of each category of samples in the source domain;

[0279] Based on KL divergence, we search for categories in the source domain that are similar to the feature distribution of samples in the target domain;

[0280] The statistical information of the found categories is combined with the features of the target domain samples to obtain the calibrated feature distribution parameters;

[0281] Optimized features are obtained by sampling from the calibrated distribution.

[0282] Furthermore, the metric learning-based mapping calibration includes:

[0283] Select the most similar and least similar features from the source domain samples to the target domain samples;

[0284] Learn a transfer function such that, after mapping the target domain features, the transfer function minimizes the distance to similar source domain features and maximizes the distance to dissimilar source domain features.

[0285] The original target domain features are concatenated with the mapped features to obtain the final optimized features.

[0286] Furthermore, in the shared encoder, the visual image encoder adopts the DINO architecture based on self-supervised knowledge distillation, and the wireless signal image encoder adopts the improved LeNet-5 architecture, in which the first convolutional layer is replaced with a single-channel convolutional layer.

[0287] The cross-scene multimodal heterogeneous data fusion positioning system of this disclosure is used to implement the cross-scene multimodal heterogeneous data fusion positioning method in the first embodiment of the method, so the description is relatively simple. For details, please refer to the relevant descriptions in the previous method embodiments, which will not be repeated here.

[0288] Figure 5 This is a block diagram of an electronic device provided in Embodiment 3 of this disclosure.

[0289] Reference Figure 5 This disclosure provides an electronic device, which includes: at least one processor 701; at least one memory 702; and one or more I / O interfaces 703 connected between the processor 701 and the memory 702; wherein the memory 702 stores one or more computer programs that can be executed by the at least one processor 701, and the one or more computer programs are executed by the at least one processor 701 to enable the at least one processor 701 to execute the above-described cross-scene multimodal heterogeneous data fusion positioning method.

[0290] This disclosure also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the aforementioned cross-scenario multimodal heterogeneous data fusion and localization method. The computer-readable storage medium may be volatile or non-volatile.

[0291] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described cross-scenario multimodal heterogeneous data fusion positioning method.

[0292] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0293] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0294] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0295] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0296] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0297] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0298] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0299] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0300] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0301] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. A cross-scenario multimodal heterogeneous data fusion localization method, characterized in that, The method includes: Data preprocessing steps: Receive sensor data from at least two different modalities, including wireless signals and visual images; normalize and spatially interpolate the wireless signal data to generate a wireless signal image, so that the wireless signal data and the visual image data are aligned in data dimension; Model training steps: The deep fusion network is pre-trained using labeled data from the source domain scene to obtain a pre-trained model; the deep fusion network includes a shared encoder and a channel exchange network (CEN). The shared encoder is used to extract features from the wireless signal image and the visual image and map them to a unified feature space. The CEN is used to exchange information between channels based on the importance of the features of each modality to a specific task. Feature transfer calibration steps: In the target domain scene, the pre-trained model is used as prior knowledge, and the feature distribution of the target domain samples is calibrated based on a small number of samples in the target domain to obtain the localization model. The calibration includes distribution calibration based on statistical information and mapping calibration based on metric learning. Localization steps: Input the preprocessed multimodal data collected in real time in the target domain scene into the localization model optimized by the feature transfer calibration step, and output the position coordinates of the target.

2. The fusion positioning method according to claim 1, characterized in that, The normalization and spatial interpolation processing of the wireless signal data includes: Normalize the Received Signal Strength Indicator (RSSI) value to the range of 0-255; The normalized wireless signal data is spatially interpolated using the Kriging interpolation method to generate the wireless signal image.

3. The fusion positioning method according to claim 1, characterized in that, The method of exchanging information between channels based on the importance of each modality to a specific task includes: For each feature channel of each modality, a trainable scaling factor is calculated, which is a factor that affects the multi-scale output of the channel; The scaling factor is compared with a preset threshold. If the scaling factor is greater than the threshold, the features of the channel are normalized according to standard procedures. If the scaling factor is less than or equal to the threshold, the feature of that channel is replaced with the weighted average of the features of the corresponding channels of other modalities, so as to realize information exchange between channels.

4. The fusion positioning method according to claim 1, characterized in that, In the model training step, the loss function used for pre-training the deep fusion network is the dual contrast loss function, which is a weighted sum of heterogeneous contrast loss and homogeneous contrast loss. The heterogeneous contrast loss is used to narrow the distance between different modalities at the same location in the feature space; The isomorphic contrast loss is used to bring the samples of the same type in the source and target domains closer together in the feature space, and to increase the distance between samples of different types.

5. The method according to claim 4, characterized in that, The heterogeneous contrast loss function calculates the matching probability in two directions: image to wireless signal and wireless signal to image, and uses the cross-entropy loss function to maximize the similarity between image-wireless signal pairs from the same location.

6. The fusion positioning method according to claim 4, characterized in that, The isomorphic contrast loss function adopts the interval loss function, and its expression is: ； in, For isomorphic contrast loss, N is the batch size, n represents the nth sample pair in the batch, y is the similarity label of whether the two samples belong to the same category, d is the distance between the samples in the feature space, and margin is a preset interval value.

7. The fusion positioning method according to claim 1, characterized in that, The distribution calibration based on statistical information includes: Calculate the mean and covariance matrix of the features of each category of samples in the source domain; Based on KL divergence, we search for categories in the source domain that are similar to the feature distribution of samples in the target domain; The statistical information of the found categories is combined with the features of the target domain samples to obtain the calibrated feature distribution parameters; Optimized features are obtained by sampling from the calibrated distribution.

8. The fusion positioning method according to claim 1, characterized in that, The metric learning-based mapping calibration includes: Select the most similar and least similar features from the source domain samples to the target domain samples; Learn a transfer function such that, after mapping the target domain features, the transfer function minimizes the distance to similar source domain features and maximizes the distance to dissimilar source domain features. The original target domain features are concatenated with the mapped features to obtain the final optimized features.

9. The fusion positioning method according to claim 1, characterized in that, In the shared encoder, the visual image encoder adopts the DINO architecture based on self-supervised knowledge distillation, and the wireless signal image encoder adopts the improved LeNet-5 architecture, in which the first convolutional layer is replaced with a single-channel convolutional layer.

10. A cross-scenario multimodal heterogeneous data fusion positioning system, characterized in that, The system includes: The data acquisition and preprocessing module is configured to receive sensor data from at least two different modalities, including wireless signals and visual images, and to perform normalization and spatial interpolation processing on the wireless signal data to generate a wireless signal image so that the wireless signal data and the visual image data are aligned in the data dimension. The pre-training module is configured to pre-train the deep fusion network using labeled data from the source domain scene to obtain a pre-trained model. The deep fusion network includes a shared encoder and a channel exchange network (CEN). The shared encoder is used to extract features from the wireless signal image and the visual image and map them to a unified feature space. The CEN is used to exchange information between channels based on the importance of the features of each modality to a specific task. The feature transfer calibration module is configured to use the pre-trained model as prior knowledge in the target domain scene and calibrate the feature distribution of the target domain samples based on a small number of samples in the target domain to obtain a localization model. The calibration includes distribution calibration based on statistical information and mapping calibration based on metric learning. The localization module is configured to input preprocessed multimodal data collected in real time in the target domain scene into the localization model optimized by the feature transfer calibration step, and output the position coordinates of the target.

11. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the cross-scene multimodal heterogeneous data fusion localization method as described in any one of claims 1-9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the cross-scenario multimodal heterogeneous data fusion positioning method as described in any one of claims 1-9.

13. A computer program product, characterized in that, Includes computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the cross-scenario multimodal heterogeneous data fusion positioning method as described in any one of claims 1-9.

Images