Multi-modal data fusion positioning method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a multi-modal data fusion positioning method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring measurement data of multiple modes of a target terminal; performing cross-modal feature fusion on the measurement data through a pre-trained cross-modal adversarial network to obtain cross-modal fusion features; the cross-modal adversarial network is obtained based on adversarial training of sample data; and carrying out positioning calculation on the cross-modal fusion feature to obtain positioning information of the target terminal. By adopting the method, the limitation of a traditional single data source can be broken through, data support is provided for subsequent fusion, the problem of heterogeneity of different modal data in feature dimensions, sampling frequencies and data formats can be solved, effective fusion of multi-modal measurement data is realized, positioning deviation caused by environmental interference of the single data source is effectively reduced, and the accuracy of multi-modal measurement data fusion is improved. And the positioning precision and stability in a complex environment are obviously improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a multimodal data fusion positioning method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the rapid development of 5G communication technology, the location of the terminal can be achieved by analyzing measurement data such as RSRP (Reference Signal Receiving Power), RSRQ (Reference Signal Receiving Quality), TA (time advanced), AOA (Angle of Arrival), and TDOA (Time Difference of Arrival) between the 5G base station and the terminal.

[0003] Traditional 5G positioning methods typically rely on measurement data from a single data source for location. For example, a terminal can receive RSRP and RSRQ from different 5G base stations and, in conjunction with a preset signal attenuation model, estimate the distance from the terminal to each 5G base station, thereby determining the terminal's specific location.

[0004] However, in complex environments such as indoors, underground, and densely populated high-rise buildings, a single data source is prone to a sharp drop in positioning accuracy due to signal obstruction, multipath effect, electromagnetic interference, and other reasons. Summary of the Invention

[0005] Therefore, it is necessary to provide a multimodal data fusion positioning method, device, computer equipment, computer-readable storage medium, and computer program product that can avoid relying on a single data source for positioning and improve positioning accuracy in complex environments, in order to address the above-mentioned technical problems.

[0006] Firstly, this application provides a multimodal data fusion localization method, including:

[0007] Acquire measurement data for multiple modalities of the target terminal;

[0008] The measurement data is fused using a pre-trained cross-modal adversarial network to obtain cross-modal fused features; the cross-modal adversarial network is obtained based on adversarial training on the sample data.

[0009] The location information of the target terminal is obtained by performing localization calculation on the cross-modal fusion features.

[0010] In one embodiment, acquiring measurement data of multiple modalities of the target terminal includes:

[0011] Acquire raw data of multiple modalities of the target terminal, wherein the raw data has a corresponding timestamp;

[0012] Based on the timestamp, the original data is time-aligned to obtain time-synchronized data in multiple modalities;

[0013] The time synchronization data is denoised to obtain measurement data for multiple modes.

[0014] In one embodiment, the cross-modal adversarial network includes a cross-modal generator, which includes a cross-modal fusion layer and multiple modality encoders for each modality.

[0015] The method of fusing cross-modal features into the measurement data using a pre-trained cross-modal adversarial network to obtain cross-modal fused features includes:

[0016] Modal features are obtained by feature encoding the measurement data using the modal encoder corresponding to the same modality;

[0017] The cross-modal fusion layer assigns corresponding fusion weights to each modal feature;

[0018] Based on the fusion weights, the modal features are mapped to a unified feature space to obtain cross-modal fusion features.

[0019] In one embodiment, assigning corresponding fusion weights to each modal feature through the cross-modal fusion layer includes:

[0020] The modal features are analyzed by the cross-modal fusion layer to obtain the environmental context, and a corresponding fusion weight is assigned to each modal feature based on the environmental context.

[0021] In one embodiment, the cross-modal generator further includes a unified representation generator, wherein mapping the modal features to a unified feature space based on the fusion weights to obtain cross-modal fusion features includes:

[0022] Through the cross-modal fusion layer, the modal features are mapped to a unified feature space based on the fusion weights to obtain preliminary fused features;

[0023] The unified representation generator performs feature standardization on the preliminary fusion features to obtain cross-modal fusion features.

[0024] In one embodiment, before performing cross-modal feature fusion on the measurement data using a pre-trained cross-modal adversarial network to obtain cross-modal fused features, the method further includes:

[0025] Acquire sample data from multiple modalities;

[0026] The sample data is input into the generator of the adversarial network to be trained for cross-modal feature fusion to obtain the predicted fused features;

[0027] The sample data and the predicted fusion features are input into the training multilayer discriminator of the adversarial network to be trained for classification, and a first discrimination result is obtained. The first discrimination result is used to indicate whether the input data matches the sample data.

[0028] Based on the first discrimination result and the predicted fusion features, the first target loss value is determined;

[0029] The model parameters of the generator to be trained are adjusted based on the first target loss value, and the sample data is input into the adjusted generator to be trained for cross-modal feature fusion until the first target loss value meets the training termination condition. The currently obtained adversarial network to be trained is then used as a cross-modal adversarial network.

[0030] In one embodiment, the step of inputting the sample data into the generator of the adversarial network to be trained for cross-modal feature fusion to obtain predicted fused features includes:

[0031] The sample data is input into the generator of the adversarial network to be trained for cross-modal feature fusion to obtain reference fused features;

[0032] The sample data and the reference fusion features are input into the multi-layer discriminator of the adversarial network to be trained for classification, and a second discrimination result is obtained. The second discrimination result is used to indicate whether the input data matches the sample data.

[0033] Based on the second discrimination result, a second target loss value is determined, and the model parameters of the multilayer discriminator to be trained are adjusted based on the second target loss value to obtain an updated multilayer discriminator to be trained.

[0034] The sample data is re-input into the generator to be trained for cross-modal feature fusion to obtain predicted fused features.

[0035] In one embodiment, the multi-layer discriminator to be trained includes a modality discriminator, the first discrimination result includes a modality discrimination result, and the step of inputting the sample data and the predicted fusion features into the multi-layer discriminator of the adversarial network to be trained for classification to obtain the first discrimination result includes:

[0036] The predicted fusion features are reconstructed to obtain reconstructed data corresponding to multiple modalities;

[0037] The sample data and the reconstructed data are input into the modality discriminator for classification to obtain a modality discrimination result. The modality discrimination result is used to indicate whether the input data matches the sample data.

[0038] In one embodiment, the multi-layer discriminator to be trained includes a fusion discriminator, and the first discrimination result includes the fusion discrimination result. The step of inputting the sample data and the predicted fusion features into the multi-layer discriminator of the adversarial network to be trained for classification to obtain the first discrimination result includes:

[0039] Obtain the true fusion features corresponding to the sample data;

[0040] The true fusion features and the predicted fusion features are input into the fusion discriminator for classification to obtain a fusion discrimination result. The fusion discrimination result is used to indicate whether the input data matches the true fusion features.

[0041] In one embodiment, the multi-layer discriminator to be trained includes a localization discriminator, and the first discrimination result includes a localization discrimination result. The step of inputting the sample data and the predicted fusion features into the multi-layer discriminator of the adversarial network to be trained for classification to obtain the first discrimination result includes:

[0042] Obtain the actual location corresponding to the sample data;

[0043] Based on the predicted fusion features, a localization calculation is performed to obtain the predicted localization.

[0044] The actual location and the predicted location are input into the location discriminator for classification to obtain a location discrimination result. The location discrimination result is used to indicate whether the input data matches the actual location.

[0045] In one embodiment, the first discrimination result includes a modality discrimination result, a fusion discrimination result, and a localization discrimination result. The step of determining a first target loss value based on the first discrimination result and the predicted fusion features includes:

[0046] The modality discrimination result, fusion discrimination result, and localization discrimination result are weighted and summed to obtain the adversarial loss;

[0047] The predicted fusion features are reconstructed to obtain reconstructed data corresponding to multiple modalities, and the error between the sample data and the reconstructed data is calculated to obtain the reconstruction loss;

[0048] Obtain the true fusion features corresponding to the sample data, and calculate the error between the true fusion features and the predicted fusion features to obtain the consistency loss;

[0049] Obtain the true location corresponding to the sample data, and perform location calculation based on the prediction fusion features to obtain the predicted location. Calculate the error between the true location and the predicted location to obtain the location loss.

[0050] The first target loss value is obtained by weighted summation of the adversarial loss, the reconstruction loss, the consistency loss, and the localization loss.

[0051] Secondly, this application also provides a multimodal data fusion positioning device, comprising:

[0052] The acquisition module is used to acquire measurement data of multiple modes of the target terminal;

[0053] The fusion module is used to perform cross-modal feature fusion on the measurement data through a pre-trained cross-modal adversarial network to obtain cross-modal fused features; the cross-modal adversarial network is obtained based on adversarial training on the sample data;

[0054] The positioning module is used to perform positioning calculations on the cross-modal fusion features to obtain the positioning information of the target terminal.

[0055] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0056] Acquire measurement data for multiple modalities of the target terminal;

[0057] The measurement data is fused using a pre-trained cross-modal adversarial network to obtain cross-modal fused features; the cross-modal adversarial network is obtained based on adversarial training on the sample data.

[0058] The location information of the target terminal is obtained by performing localization calculation on the cross-modal fusion features.

[0059] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0060] Acquire measurement data for multiple modalities of the target terminal;

[0061] The measurement data is fused using a pre-trained cross-modal adversarial network to obtain cross-modal fused features; the cross-modal adversarial network is obtained based on adversarial training on the sample data.

[0062] The location information of the target terminal is obtained by performing localization calculation on the cross-modal fusion features.

[0063] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0064] Acquire measurement data for multiple modalities of the target terminal;

[0065] The measurement data is fused using a pre-trained cross-modal adversarial network to obtain cross-modal fused features; the cross-modal adversarial network is obtained based on adversarial training on the sample data.

[0066] The location information of the target terminal is obtained by performing localization calculation on the cross-modal fusion features.

[0067] The aforementioned multimodal data fusion positioning method, apparatus, computer equipment, computer-readable storage medium, and computer program product, by acquiring multimodal measurement data from the target terminal, break through the limitations of traditional single data sources and provide data support for subsequent fusion. Then, the measurement data is input into a pre-trained cross-modal adversarial network for cross-modal feature fusion, which can solve the heterogeneity problem of different modal data in terms of feature dimensions, sampling frequency, and data format, and realize the effective fusion of multimodal measurement data. Furthermore, the positioning calculation of the cross-modal fused features can effectively reduce the positioning deviation caused by environmental interference from a single data source and significantly improve the positioning accuracy and stability in complex environments. Attached Figure Description

[0068] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0069] Figure 1 This is a flowchart of a multimodal data fusion localization method in one embodiment;

[0070] Figure 2 This is a system architecture diagram of a multimodal data fusion localization method in one embodiment;

[0071] Figure 3 This is a flowchart illustrating the preprocessing of raw data in one embodiment.

[0072] Figure 4 This is an architecture diagram of a cross-modal generative adversarial network in one embodiment;

[0073] Figure 5 This is a schematic diagram of the training process of a cross-modal generative adversarial network in one embodiment.

[0074] Figure 6 This is a decision-making flowchart for an adaptive fusion strategy in one embodiment;

[0075] Figure 7 This is a structural block diagram of a multimodal data fusion positioning device in one embodiment;

[0076] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0077] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0078] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0079] Traditional 5G positioning methods typically rely on measurement data from a single data source for location. For example, a terminal can receive RSRP and RSRQ from different 5G base stations and, in conjunction with a preset signal attenuation model, estimate the distance from the terminal to each 5G base station, thereby determining the terminal's specific location.

[0080] However, in complex environments such as indoors, underground, and densely populated high-rise areas, a single data source is prone to a sharp drop in positioning accuracy due to signal obstruction, multipath effects, electromagnetic interference, and other factors. Therefore, in an exemplary embodiment, such as... Figure 1 As shown, this application provides a multimodal data fusion localization method, including the following steps:

[0081] Step 101: Obtain measurement data for multiple modes of the target terminal.

[0082] In order to break the limitation of traditional 5G positioning relying on a single data source, this application can acquire measurement data of multiple modalities, providing comprehensive and reliable information support for subsequent accurate positioning.

[0083] Measurement data can come from various data sources related to the positioning of the target terminal, including but not limited to signal strength parameters such as RSRP, RSRQ, and CQI (Channel Quality Indicator) generated by the interaction between the 5G base station and the target terminal; location correlation parameters such as TA, AOA, and TDOA (Time Difference of Arrival) reflecting the spatial geometric relationship between the 5G base station and the target terminal; motion data from the target terminal's own IMU (Inertial Measurement Unit); environmental perception data such as GPS (Global Positioning System) data and GNSS (Global Navigation Satellite System) data received by sensors; and visual data such as image data and depth information.

[0084] RSRP is a parameter representing the strength of a wireless signal in LTE (Long-Term Evolution) networks. Within a certain symbol, the target terminal averages the signal power received on all resource particles carrying the reference signal, and the result is RSRP.

[0085] RSRQ is a parameter used in LTE networks to evaluate radio signal quality. It provides a basis for ranking different LTE candidate cells by comprehensively considering the received power of the reference signal and the interference and noise levels in the channel. RSRQ can serve as an important input parameter for cell handover and reselection decisions, thereby ensuring that the target terminal always connects to the optimal serving cell, improving communication quality and user experience.

[0086] CQI is a parameter used to characterize the quality of the current channel. Its value is closely related to the signal-to-noise ratio of the channel, and it typically ranges from 0 to 31. A higher CQI value indicates better channel conditions, and the system can support higher data transmission rates.

[0087] TA refers to the difference between the actual arrival time of the wireless signal from the target terminal to the 5G base station and the ideal arrival time assuming the distance between the target terminal and the base station is zero. The 5G base station continuously monitors the arrival time of the wireless signal from the target terminal and adjusts the signal transmission timing on the downlink SACCH (Slow Associated Control Channel) according to its changes to ensure uplink synchronization.

[0088] AOA refers to the angle between the wireless signal and a reference direction of the receiving antenna array when the wireless signal arrives at the receiving end from the transmitting end. This angle is usually expressed in two dimensions: azimuth and elevation. It is an important measurement parameter in wireless positioning technology.

[0089] TDOA refers to the time difference in arrival time of a signal emitted by a signal source when it is received by multiple receiving points at known locations (such as 5G base stations or target terminals) due to the different signal propagation paths. TDOA-based positioning technology does not require strict time synchronization and has high positioning accuracy.

[0090] GPS is a high-precision radio navigation positioning system based on artificial Earth satellites, which can provide users with accurate location, speed, and time information.

[0091] GNSS is a general term for satellite navigation systems that can achieve global coverage and provide all-weather three-dimensional coordinates, velocity, and time information at any location on the Earth's surface or in near-Earth space.

[0092] An inertial measurement unit (IMU) is a device that measures an object's three-axis attitude angles (or angular rates) and acceleration. It typically includes three single-axis accelerometers and three single-axis gyroscopes. The accelerometers detect the object's three-axis acceleration signals in the carrier's coordinate system, while the gyroscopes measure the carrier's angular velocity signals relative to the navigation coordinate system. The object's attitude changes in three-dimensional space can be calculated from the acceleration and angular velocity signals.

[0093] This application can be applied to any base station, server, or terminal (including the target terminal). The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc.

[0094] Step 102: Cross-modal feature fusion is performed on the measurement data through a pre-trained cross-modal adversarial network to obtain cross-modal fused features; the cross-modal adversarial network is obtained based on adversarial training on the sample data.

[0095] Typically, measurement data from different data sources exhibit differences in modal characteristics, sampling frequencies, and data formats, making direct fusion and utilization difficult. Therefore, in this step, the measurement data can be input into a pre-trained cross-modal adversarial network to obtain cross-modal fusion features, thereby addressing the heterogeneity of multimodal data and achieving effective information integration.

[0096] Cross-modal adversarial networks (CANs) are trained on adversarial data and typically consist of a generator and a discriminator. They can effectively transform measurement data from different modalities into unified cross-modal fusion features, providing high-quality and usable fusion information for subsequent localization calculations.

[0097] During the training phase, the generator and discriminator continuously improve their performance through a game of continuous competition. The generator learns how to fuse multimodal measurement data into cross-modal fusion features that contain the core information of each modality, while the discriminator learns to distinguish between real measurement data and generated cross-modal fusion features. The dynamic competition between the two ultimately enables the generator to have a stable cross-modal fusion capability.

[0098] Furthermore, when measurement data is input into the cross-modal adversarial network, the generator can perform feature fusion on measurement data from different modalities, and the final output cross-modal fused features can provide high-quality feature input for subsequent localization calculations.

[0099] Step 103: Perform localization calculation on the cross-modal fusion features to obtain the localization information of the target terminal.

[0100] In this step, by using appropriate positioning calculation methods, the positioning-related information in the cross-modal fusion features can be fully extracted, thereby transforming the fused abstract features into positioning results and obtaining the positioning information of the target terminal.

[0101] The specific method for localization calculation can be selected based on the needs of the scenario, and this application does not impose any specific restrictions on it.

[0102] For example, in static or low-dynamic scenarios, such as indoor robot point navigation, the weighted least squares method can often be used to estimate the distance from the target terminal to each known reference point based on cross-modal fusion features, and assign a corresponding weight to each distance estimate. The higher the weight, the more reliable the distance estimate for that reference point. Then, a point that minimizes the weighted sum of the errors between all distance estimates can be found as the positioning information of the target terminal.

[0103] In addition, in dynamic scenarios, such as mobile terminal navigation, Bayesian filtering algorithms are more suitable. First, cross-modal fusion features are used to predict the probability distribution of the next position of the target terminal. Then, new cross-modal fusion features are used as evidence to update the predicted probability distribution. Bayes' formula is used to calculate the new distribution after obtaining the new evidence. The position with the highest probability can be used as the positioning information of the target terminal.

[0104] In this way, compared with positioning calculation based on measurement data from a single data source, cross-modal fusion features can be used to locate the target terminal more comprehensively and accurately, providing a strong guarantee for the target terminal to achieve high-precision positioning in complex environments.

[0105] like Figure 2 The diagram shown is a system architecture diagram of a multimodal data fusion localization method in a specific embodiment of this application, wherein:

[0106] First, the multimodal data preprocessing module collects measurement data from various sources, including signal data provided by 5G base stations, motion data provided by IMU sensors, wireless signal data provided by WiFi / Bluetooth, auxiliary information provided by GPS, and visual data such as images provided by visual sensors.

[0107] Then, the measurement data enters the cross-modal adversarial generative network (CM-GAN). Through adversarial training of the cross-modal generator and the multi-layer discriminator, the cross-modal adversarial generative network can achieve effective fusion of measurement data and generate cross-modal fusion features that can accurately reflect localization-related information.

[0108] Subsequently, the cross-modal fusion features are transmitted to the positioning solution module, where the corresponding positioning algorithm is used to calculate the positioning information. The positioning information is usually high-precision location information, which can support positioning needs in various scenarios.

[0109] Because the sampling frequencies of different modal data differ, direct fusion can lead to positioning errors due to time misalignment. To address this issue, in an exemplary embodiment, step 101 includes:

[0110] Acquire raw data from multiple modalities of the target terminal; the raw data has corresponding timestamps.

[0111] Time synchronization data in multiple modalities is obtained by aligning the original data based on timestamps.

[0112] Denoising was performed on the time synchronization data to obtain measurement data for multiple modes.

[0113] In other words, the first thing acquired is the raw data from multiple modalities of the target terminal. Each raw data point is accompanied by a precise timestamp during generation to record the specific moment of data acquisition, providing a basis for subsequent timing calibration. For example, RSRP might sample 10 times per second, while AOA, due to its higher measurement complexity, only samples 5 times per second. Therefore, RSRP and AOA are not synchronized in time.

[0114] Then, through timestamp-based synchronization processing, the raw data of each modality can be aligned to a unified time axis, resulting in time-synchronized data for each modality. This ensures that the time synchronization of different modalities at the same moment corresponds to the same state of the target terminal. For example, for raw data with a low sampling frequency, missing values ​​can be supplemented using interpolation; for raw data with a high sampling frequency, aggregation can be performed according to a fixed time window.

[0115] Then, the time-synchronized data can be denoised to obtain usable measurement data for multiple modalities, laying a data quality foundation for subsequent cross-modal fusion. Denoising processing includes, but is not limited to, filtering and outlier removal, etc. For example, for time-synchronized data with strong temporal sequence characteristics, Kalman filtering or moving average filtering can be used to smooth numerical fluctuations; for time-synchronized data with geometric parameters, outlier detection algorithms can be used to remove extreme values ​​that exceed reasonable ranges, and so on.

[0116] In this way, the obtained measurement data not only retains the core characteristics of the original data of each modality, but also has the characteristics of low noise and high reliability, which can provide high-quality input for feature fusion of cross-modal adversarial networks.

[0117] like Figure 3 The diagram shown is a flowchart of preprocessing raw data in a specific embodiment of this application, wherein:

[0118] The raw multimodal data first needs to be standardized to unify the units of measurement for different modalities, facilitating subsequent processing. Then, time synchronization is performed to ensure that raw data collected from different sources and at different times are aligned in the time dimension, resulting in time-aligned data. Next, Kalman filtering is used to remove noise interference from the time-aligned data. Then, feature extraction is performed to extract useful measurement data for tasks such as localization from the denoised time-aligned data. Following this, quality assessment is conducted to determine if the measurement data meets the required quality. Finally, after preprocessing, the data can be output, providing clean, standardized, and valuable measurement data for subsequent cross-modal generative adversarial networks.

[0119] In one exemplary embodiment, the cross-modal adversarial network includes a cross-modal generator, which includes a cross-modal fusion layer and multiple modality encoders for each modality.

[0120] Therefore, step 102 includes steps 201 to 203. Wherein:

[0121] Step 201: Modal features are obtained by feature encoding the measurement data using a modal encoder corresponding to the same modality.

[0122] Step 202: Assign corresponding fusion weights to each modal feature through a cross-modal fusion layer;

[0123] Step 203: Based on the fusion weights, the modal features are mapped to a unified feature space to obtain cross-modal fusion features.

[0124] In this exemplary embodiment, the cross-modal generator of the cross-modal adversarial network generates cross-modal fusion features through a modality-specific modality encoder and a dynamic fusion strategy, which preserves the characteristics of each modality and breaks down the barriers of data heterogeneity.

[0125] Specifically, since the measurement data of different modalities have significant differences in data format and semantic connotation, the cross-modal generator is equipped with a dedicated modal encoder for each modality. In this embodiment, the measurement data of each modality is first feature-encoded by the modal encoder of each modality to obtain the modal features of that modality.

[0126] Modal encoders can transform measurement data into high-dimensional abstract modal features through nonlinear transformations of neural network layers. For example, modal encoders can include, but are not limited to, 5G signal encoders, used to process radio frequency indicators such as RSRP, RSRQ, and CQI to extract signal strength and quality features; inertial measurement unit encoders, used to process acceleration, gyroscope, and magnetometer data to extract motion state features; environmental perception encoders, used to process WiFi, Bluetooth, and GPS data to extract position-related features; and visual encoders, used to process image data to extract visual positioning features.

[0127] Secondly, the localization value of different modalities varies in complex environments, and a fixed-weight fusion method cannot adapt to changes in the scenario. The cross-modal fusion layer dynamically calculates the fusion weights of each modal feature through an attention mechanism. For example, when RSRP data fluctuates drastically at a certain moment, the fusion layer automatically reduces its weight; while when AOA data is stable, its weight is increased. This dynamic allocation mechanism ensures that the fusion process focuses on more reliable modal information in the current scenario, avoiding interference from low-quality data in the fusion results.

[0128] It is understandable that the reliability of measurement data of different modalities is not constant in complex environments. For example, in indoor occluded scenarios, AOA may deviate significantly due to signal reflection, while the relative change of RSRP can still reflect the distance trend; while in open outdoor environments, TA time difference measurement accuracy is higher.

[0129] Based on this, the cross-modal fusion layer can be pre-trained to learn to assign corresponding fusion weights to each modal feature. In this way, when the measurement data of a certain modality is less affected by environmental interference, the cross-modal fusion layer will automatically give it a higher fusion weight, making it dominant in the subsequent fusion process; when the measurement data of a certain modality is noisy, its fusion weight will be reduced to reduce interference with the fusion result.

[0130] Furthermore, modal features can be mapped to a unified feature space based on fusion weights to obtain cross-modal fused features. Under the dynamic control of fusion weights, modal features of different dimensions are projected onto the same preset high-dimensional feature space, i.e., the unified feature space, and feature fusion is completed through weighted summation.

[0131] In this way, the heterogeneity barrier between modes can be eliminated, enabling multimodal measurement data that were originally difficult to correlate directly to work together. The output cross-modal fusion features not only condense the effective information of each mode, but also filter out noise interference through fusion weight adjustment, providing accurate and comprehensive feature input for subsequent localization calculation.

[0132] In one exemplary embodiment, step 202 includes:

[0133] By performing environmental analysis on modal features through a cross-modal fusion layer, an environmental context is obtained, and corresponding fusion weights are assigned to each modal feature based on the environmental context.

[0134] Specifically, the cross-modal fusion layer can use the feature analysis module of the neural network to perform environmental analysis on the features of each modality to obtain environmental context, such as "indoor multi-occlusion environment", "outdoor high-rise dense area", "underground weak signal environment", etc., to provide clear scenario basis for the subsequent allocation of fusion weights.

[0135] For example, if the 5G RSRP feature exhibits high-frequency and drastic fluctuations, and the RSRP values ​​of different base stations are very similar, then the environmental context can be "indoor multi-obstruction environment" or "dense high-rise building environment"; if the AOA feature has low dispersion, the values ​​are concentrated in a fixed range, and the mapping relationship between the TA feature and distance is stable, then the environmental context can be "outdoor open environment".

[0136] Building upon this, the cross-modal fusion layer can assign corresponding fusion weights to each modal feature based on the environmental context. Specifically, the cross-modal fusion layer can pre-define a rule base that matches different environmental contexts, or learn the dynamic mapping relationship between environmental context and fusion weights through adversarial training.

[0137] For example, in an indoor multi-obstructed environment, the reliability of modal features such as AOA and RSRP decreases because 5G signals are easily interfered with by wall reflections. The cross-modal fusion layer automatically reduces the fusion weight of these modal features. Meanwhile, if Bluetooth beacons or inertial measurement data are present, these modal features are less affected by obstruction, and the cross-modal fusion layer can increase their fusion weight. Conversely, when the environmental context switches to an outdoor open environment, the accuracy of 5G TA and AOA features significantly improves. The cross-modal fusion layer can increase the fusion weight of these two types of modal features while decreasing the fusion weight of Bluetooth features, which are more adaptable to indoor environments.

[0138] This ensures that the contribution of each modal feature in the cross-modal fusion features matches the adaptability to the current environment. It avoids noise interference from low-reliability modal features in the fusion results, while fully leveraging the localization value of high-reliability modal features. This allows the cross-modal fusion features to accurately adapt to the real-time scenario and provide high-quality support for subsequent localization calculations.

[0139] In one exemplary embodiment, the cross-modal generator further includes a unified representation generator, and step 203 includes:

[0140] Through the cross-modal fusion layer, modal features are mapped to a unified feature space based on fusion weights to obtain preliminary fused features;

[0141] The initial fused features are standardized using a unified representation generator to obtain cross-modal fused features.

[0142] Previously, the cross-modal fusion layer had assigned dynamic weights to the features of each modality through environmental analysis. This step will use these weights as a basis to project the feature vectors of different modalities into a unified feature space of the same preset dimension.

[0143] In this exemplary embodiment, the modal features are mapped to a unified feature space based on the fusion weights by the cross-modal fusion layer to obtain preliminary fusion features. Then, the cross-modal generator can transform the preliminary fusion features into cross-modal fusion features that are more adapted to the positioning requirements by adjusting the unified representation generator. This solves the problems of distribution differences and information redundancy that may exist in the preliminary fusion features, so that the final cross-modal fusion features retain the multimodal collaborative value and have higher consistency and relevance.

[0144] Specifically, the cross-modal fusion layer has assigned fusion weights to each modal feature based on environmental analysis, and then weighted and integrated each modal feature according to the fusion weights. This process achieves the initial aggregation of multimodal features. However, due to the inherent differences in the original modal features, the initial fusion features may have problems such as some dimensions having excessively large values ​​and some dimensions having their information obscured.

[0145] Building upon this, a unified representation generator can be used to standardize the initial fusion features, yielding the final cross-modal fusion features. The core function of the unified representation generator is to eliminate feature distribution bias. Through batch normalization or layer normalization techniques, it adjusts the values ​​of each dimension of the initial fusion features to a standard distribution, preventing a particular modality from becoming overly dominant in the cross-modal fusion features due to an excessively large numerical range. This ensures that modal features with smaller numerical ranges can also contribute valuable information.

[0146] In this way, the final output of cross-modal fusion features not only has unified dimensions and balanced numerical distribution, but also accurately focuses on key positioning information, providing better feature input for subsequent positioning calculations and effectively supporting high-precision positioning in complex environments.

[0147] In an exemplary embodiment, prior to step 102, the method provided in this application further includes steps 301 to 305. Wherein:

[0148] Step 301: Obtain sample data for multiple modalities;

[0149] Step 302: Input the sample data into the generator to be trained of the adversarial network to perform cross-modal feature fusion and obtain the predicted fused features;

[0150] Step 303: Input the sample data and predicted fusion features into the multi-layer discriminator of the adversarial network to be trained for classification, and obtain the first discrimination result. The first discrimination result is used to indicate whether the input data matches the sample data.

[0151] Step 304: Based on the first discrimination result and the predicted fusion features, determine the first target loss value;

[0152] Step 305: Adjust the model parameters of the generator to be trained based on the first target loss value, and input the sample data into the adjusted generator to be trained for cross-modal feature fusion until the first target loss value meets the training termination condition. Then, use the currently obtained adversarial network to be trained as a cross-modal adversarial network.

[0153] In this exemplary embodiment, the training of the cross-modal adversarial network is completed through adversarial training based on sample data, which allows the generator to be trained to gradually master the fusion rules of multimodal measurement data under the supervision of multi-layer discriminators, and finally form a cross-modal fusion feature that can stably output high-quality fusion features.

[0154] First, the training begins with step 301, which involves acquiring sample data for multiple modalities. The sample data must cover modalities consistent with the actual positioning scenario and include diverse environmental scenarios such as indoor, underground, and densely populated high-rise areas. At the same time, the data undergoes preprocessing such as time alignment and noise reduction to ensure data quality, enabling the adversarial network to be trained to adapt to positioning needs in different environments.

[0155] Subsequently, in step 302, the sample data is input into the generator to be trained to generate predicted fusion features. At this point, although the generator to be trained has a complete structure, the model parameters have not yet been optimized, and the output predicted fusion features may not be perfect. They are only used to provide specific objects for supervision of the subsequent multi-layer discriminator to be trained.

[0156] In step 303, the sample data and the predicted fused features are input together into the multilayer discriminator to be trained for classification. If the fusion effect of the generator to be trained is good, the multilayer discriminator to be trained will have difficulty distinguishing between the predicted fused features and the sample data, that is, the first discrimination result output will classify the predicted fused features and the sample data as the same category; if the fusion effect of the generator to be trained is poor, the multilayer discriminator to be trained can easily distinguish between the predicted fused features and the sample data, that is, the first discrimination result output will classify the predicted fused features and the sample data as different categories.

[0157] Based on the first discrimination result and the predicted fusion feature, step 304 calculates the first target loss value. The first target loss value can be determined from two aspects: first, whether the multi-layer discriminator to be trained can accurately distinguish between the predicted fusion feature and the sample data; and second, the difference between the predicted fusion feature and the sample data. The final first target loss value can quantify the fusion capability defect of the current generator to be trained.

[0158] Then, in step 305, based on the first target loss value, the model parameters of the generator to be trained are adjusted through the backpropagation algorithm, so that the generator to be trained can be improved in a targeted manner when generating predicted fusion features in the next iteration. Through iteration, the first target loss value is reduced to a preset threshold or the maximum number of iterations is reached. At this point, the adversarial network to be trained has a stable cross-modal fusion capability and can be used as the final cross-modal adversarial network in actual positioning scenarios to provide reliable support for the fusion of measurement data.

[0159] Typically, training a cross-modal adversarial network is achieved through alternating training of a generator to be trained and a multi-layer discriminator to be trained. In an exemplary embodiment, step 302 includes:

[0160] The sample data is input into the generator of the adversarial network to be trained for cross-modal feature fusion to obtain reference fused features;

[0161] The sample data and reference fused features are input into the training multi-layer discriminator of the adversarial network to be trained for classification, and a second discrimination result is obtained. The second discrimination result is used to indicate whether the input data matches the sample data.

[0162] The second target loss value is determined based on the second discrimination result, and the model parameters of the multilayer discriminator to be trained are adjusted based on the second target loss value to obtain the updated multilayer discriminator to be trained.

[0163] The sample data is re-input into the generator to be trained for cross-modal feature fusion to obtain the predicted fused features.

[0164] Specifically, the model parameters of the generator to be trained are first temporarily fixed, and the sample data is input into the generator to obtain reference fusion features. At this time, the model parameters of the generator to be trained have not been fully optimized, and its output reference fusion features are only used to provide materials for the training of the multi-layer discriminator to be trained, so that the multi-layer discriminator to be trained can establish a distinguishing criterion between the sample data and the reference fusion features.

[0165] Next, the sample data and reference fused features are input into the multilayer discriminator to be trained for classification, resulting in a second discrimination result. During this process, the multilayer discriminator gradually grasps the reasonable distribution patterns of sample data under different environments by analyzing the differences between a large amount of sample data and the reference fused features.

[0166] Then, the second target loss value determined based on the second discrimination result becomes the basis for optimizing the multilayer discriminator to be trained. If the multilayer discriminator misclassifies a real sample as a "mismatch" or gives a high score to a low-quality reference fusion feature, the second loss value increases; conversely, the second loss value decreases. By adjusting the model parameters of the multilayer discriminator to be trained through backpropagation, it continuously improves the discrimination accuracy between sample data and reference fusion features, ultimately obtaining an updated multilayer discriminator to be trained. At this point, the multilayer discriminator to be trained can more accurately identify sample data.

[0167] Then, the model parameters of the multi-layer discriminator to be trained can be temporarily fixed, the sample data can be re-input into the generator to be trained to generate predicted fusion features, and the process can return to step S303 to adjust the model parameters of the generator to be trained. Since the updated multi-layer discriminator to be trained can more sensitively capture feature defects, the generator to be trained will be optimized more specifically based on the feedback from the multi-layer discriminator to be trained in subsequent training.

[0168] By alternating training, the quality of the predicted fusion features output by the generator to be trained is gradually improved, ultimately enabling the entire cross-modal adversarial network to have reliable cross-modal fusion capabilities, providing strong support for actual localization scenarios.

[0169] In an exemplary embodiment, the multilayer discriminator to be trained includes a modality discriminator, the first discrimination result includes a modality discrimination result, and step 303 includes:

[0170] The predicted fusion features are reconstructed to obtain reconstructed data corresponding to multiple modalities;

[0171] The sample data and reconstructed data are input into the modality discriminator for classification, and the modality discrimination result is obtained. The modality discrimination result is used to indicate whether the input data matches the sample data.

[0172] In this exemplary embodiment, the multi-layer discriminator to be trained determines whether the predicted fusion features can still identify the original sample data through the modality discriminator, ensuring that the predicted fusion features output by the generator to be trained can accurately retain the original characteristics of each single modality and avoid losing important modality-specific information during the feature fusion process.

[0173] Specifically, the predicted fusion features are first reconstructed to obtain reconstructed data corresponding to multiple modalities. The predicted fusion features are abstract feature vectors formed by the generator to be trained after performing cross-modal fusion on multimodal sample data. Reconstruction is the process of restoring this abstract feature vector to reconstructed data of the same type and format as the original sample data through the decoding process, providing an intuitive basis for comparison in subsequent modality-level discrimination.

[0174] For example, if the sample data contains three modalities: RSRP, AOA, and TA, then after reconstructing the predicted fusion features, three sets of reconstructed data will be obtained: reconstructed RSRP, reconstructed AOA, and reconstructed TA. The dimensions and format of each set of reconstructed data are consistent with the corresponding modalities in the sample data.

[0175] Subsequently, the sample data and reconstructed data are input together into the modality discriminator for classification, yielding the modality discrimination results. The core function of the modality discriminator is to verify the authenticity of the reconstructed data modally. For each modality, the modality discriminator compares the feature patterns of the original sample data and the reconstructed data to determine whether the reconstructed data faithfully reflects the characteristics of the sample data.

[0176] The final output modality discrimination result is usually the matching probability between the sample data and the reconstructed data of each modality. The higher the probability, the higher the matching degree between the reconstructed data and the sample data of that modality. Conversely, it indicates that the information of that modality may be lost or distorted during the fusion process.

[0177] This supervisory mechanism based on modality discriminators provides precise guidance for optimizing the generator to be trained. If the modality discrimination result of a certain modality is consistently low, it indicates that the generator to be trained has defects in processing the sample data of that modality, and the corresponding model parameters need to be adjusted accordingly. If the modality discrimination results of all modalities are high, it indicates that the predicted fusion features have not only integrated multimodal information, but also fully preserved the core characteristics of each modality.

[0178] In this way, the training of cross-modal adversarial networks can focus on the information integrity of each modality. The resulting cross-modal fusion features can take into account multimodal collaboration while ensuring that the localization value of a single modality is not weakened, providing more comprehensive and reliable feature support for subsequent localization calculations.

[0179] In another exemplary embodiment, the multi-layer discriminator to be trained includes a fusion discriminator, the first discrimination result includes the fusion discrimination result, and step 303 includes:

[0180] Obtain the true fusion features corresponding to the sample data;

[0181] The true fusion features and the predicted fusion features are input into the fusion discriminator for classification, and the fusion discriminant result is obtained. The fusion discriminant result is used to indicate whether the input data matches the true fusion features.

[0182] In this exemplary embodiment, the multi-layer discriminator to be trained evaluates the quality and consistency of the predicted fusion features through the fusion discriminator, ensuring that the predicted fusion features output by the generator to be trained reasonably combine multimodal sample data, forming an overall feature distribution that conforms to the localization logic, and avoiding confusion or conflict of multimodal information in the fusion process.

[0183] Specifically, the first step is to obtain the true fusion features corresponding to the sample data. True fusion features are standard features obtained by ideally fusing multimodal sample data. Their generation logic is usually based on prior knowledge or verification results in high-precision positioning scenarios, which can provide a clear benchmark for judging the quality of predicted fusion features.

[0184] Subsequently, the real and predicted fusion features are input together into the fusion discriminator for classification, yielding the fusion discrimination result. The core task of the fusion discriminator is to determine, from an overall perspective, whether the distribution pattern of the predicted fusion features closely resembles that of the real fusion features, including whether the weight ratio of each modality is reasonable and whether the correlation between feature dimensions is consistent with the real scene. The final output fusion discrimination result is presented in the form of probability values. A higher probability indicates a higher overall consistency between the predicted and real fusion features, while a lower probability indicates a problem with information integration during the fusion process.

[0185] This supervision mechanism based on the fusion discriminator provides overall guidance for the optimization of the generator to be trained. If the fusion discrimination result is consistently low, it indicates that there is a defect in the generator's overall integration logic of multimodal information; if the fusion discrimination result is high, it indicates that the predicted fusion features have the positioning value of real fusion features as a whole.

[0186] In this way, the resulting cross-modal fusion features can accurately integrate multimodal information and conform to the physical laws of the actual positioning scenario, providing high-quality overall feature input for subsequent positioning calculations.

[0187] In another exemplary embodiment, the multi-layer discriminator to be trained includes a localization discriminator, the first discrimination result includes a localization discrimination result, and step 303 includes:

[0188] Obtain the actual location corresponding to the sample data;

[0189] Predicted localization is obtained by performing localization calculation based on predictive fusion features;

[0190] The actual location and predicted location are input into the location discriminator for classification, and the location discrimination result is obtained. The location discrimination result is used to indicate whether the input data matches the actual location.

[0191] In this exemplary embodiment, the multi-layer discriminator to be trained determines whether the predicted fusion features contain sufficient location-related information through the localization discriminator, ensuring that the predicted fusion features output by the generator to be trained can support high-precision localization calculations and more effectively serve the localization target.

[0192] Specifically, the first step is to obtain the actual location corresponding to the sample data. Actual location is the actual location information of the terminal corresponding to the sample data, usually obtained through high-precision positioning methods, providing the most direct basis for judging the effectiveness of predictive fusion features.

[0193] Subsequently, localization is calculated based on the predictive fusion features to obtain the predicted localization. This process simulates the calculation logic in real-world localization scenarios: using the same calculation method as real localization, location information is extracted from the predictive fusion features output by the generator to be trained, and the predicted localization of the terminal is calculated. For example, if real localization is calculated using distance and orientation information from the predictive fusion features, then the predicted localization also uses the same algorithm, calculating based on the corresponding information in the predictive fusion features, ensuring that the two are comparable.

[0194] Then, the actual location and predicted location are input into the localization discriminator for classification, and the localization discrimination result is obtained. The core task of the localization discriminator is to determine the degree of agreement between the predicted location and the actual location. The final output localization discrimination result reflects this matching degree in the form of probability. The closer the probability is, the more reliable the localization result supported by the predicted fusion features is.

[0195] This supervisory mechanism based on the localization discriminator provides result-oriented guidance for optimizing the generator to be trained. If the localization discrimination result has a high accuracy, it indicates that the predicted fusion features can effectively support accurate localization. In this way, it can be ensured that the finally generated cross-modal fusion features not only conform to the data at the data level, but also output high-precision results in actual localization scenarios.

[0196] In this application, the multi-layer discriminator to be trained may include multiple types of modality discriminators, fusion discriminators, and localization discriminators. Correspondingly, in an exemplary embodiment, the first discrimination result includes modality discrimination result, fusion discrimination result, and localization discrimination result. Step 304 includes:

[0197] The adversarial loss is obtained by weighted summation of the modality discrimination results, fusion discrimination results, and localization discrimination results.

[0198] The predicted fusion features are reconstructed to obtain reconstructed data corresponding to multiple modalities, and the error between the sample data and the reconstructed data is calculated to obtain the reconstruction loss;

[0199] Obtain the true fusion features corresponding to the sample data, and calculate the error between the true fusion features and the predicted fusion features to obtain the consistency loss;

[0200] Obtain the true location corresponding to the sample data, and perform localization calculation based on the prediction fusion features to obtain the predicted location. Calculate the error between the true location and the predicted location to obtain the localization loss.

[0201] The first objective loss value is obtained by weighted summation of the adversarial loss, reconstruction loss, consistency loss, and localization loss.

[0202] In this exemplary embodiment, the calculation of the first target loss value is achieved through the weighted integration of multi-dimensional losses. This ensures that the predicted fusion features are consistent with the sample data in terms of modal details, overall distribution, and localization effect. At the same time, the adversarial mechanism drives the generator to continuously improve its fusion capabilities, ultimately achieving high-quality output of cross-modal fusion features.

[0203] Specifically, the adversarial loss is first obtained by weighted summation of the modality discrimination result, the fusion discrimination result, and the localization discrimination result. This adversarial loss drives the adversarial game between the generator and the multi-layer discriminator, enabling the generator to produce increasingly realistic cross-modal features. These three discrimination results reflect the quality of the predicted fusion features from different perspectives: the modality discrimination result focuses on the completeness of single-modality information, the fusion discrimination result focuses on the overall synergy of multi-modal information, and the localization discrimination result is directly related to the final localization effect. The adversarial loss, by assigning different weights to the three, transforms the opinions of the modality discriminator, the fusion discriminator, and the localization discriminator into quantitative indicators.

[0204] Building upon this foundation, the fidelity of modal information is enhanced. After reconstructing the predicted fusion features to obtain the reconstructed data for each modality, these data are compared with the sample data. The differences are measured using metrics such as mean squared error or cosine similarity, and the reconstruction loss is calculated. The reconstruction loss ensures that the predicted fusion features can reconstruct the original sample data, preventing the generator to be trained from losing the core information of any modality during the fusion process and avoiding sacrificing the localization value of a single modality due to an excessive pursuit of overall fusion.

[0205] Meanwhile, by calculating the error between the predicted fusion features and the real fusion features corresponding to the sample data, the consistency loss is obtained. The consistency loss can solve the semantic alignment problem between sample data of different modalities and ensure the semantic consistency of the predicted fusion features output by the generator to be trained in different modalities.

[0206] Furthermore, the predicted localization based on predictive fusion feature calculation is compared with the actual localization corresponding to the sample data, and the difference is quantified using coordinate error to obtain the localization loss. The localization loss is used to directly optimize the final localization information, ensuring that the optimization direction of the generator to be trained always serves the core need to improve localization accuracy.

[0207] Furthermore, the adversarial loss, reconstruction loss, consistency loss, and localization loss are weighted and summed to obtain the first objective loss value, which comprehensively reflects the multi-dimensional deficiencies of the predicted fusion features. The generator to be trained continuously reduces the first objective loss value through backpropagation, gradually achieving the fusion goal of complete modal information, reasonable overall distribution, and accurate localization results, providing clear and comprehensive optimization guidance for high-quality training of cross-modal adversarial networks.

[0208] For example, such as Figure 4 The diagram shown is an architecture diagram of a cross-modal generative adversarial network in a specific embodiment of this application.

[0209] First, different types of sample data, such as 5G signal data, IMU data, environmental perception data, and visual data, will be encoded by the corresponding modal encoders to be trained (E1, E2, E3, E4) to transform the measurement data into sample modal features that are more suitable for subsequent fusion.

[0210] Next, these encoded modal features of different samples enter the cross-modal fusion layer to be trained. An attention mechanism is used to distinguish and weight the importance of different modal features of samples, so as to obtain the preliminary fusion features of the samples.

[0211] The initial fusion features of the samples are transmitted to the unified representation generator G to be trained, resulting in standardized predicted fusion features.

[0212] Then, the multi-layer discriminator to be trained discriminates the generated predicted fusion features. The multi-layer discriminator to be trained includes a modality discriminator D1, a fusion discriminator D2, and a localization discriminator D3.

[0213] like Figure 5 The diagram shows the training process of a cross-modal generative adversarial network. During the adversarial training iteration, the generator G and the multi-layer discriminator D to be trained play against each other and train alternately through loss calculation and parameter update until the generator can generate sufficiently realistic predicted fusion features and the multi-layer discriminator to be trained can hardly distinguish the predicted fusion features from the sample data. This results in the overall performance of the cross-modal generative adversarial network reaching a good level.

[0214] The loss function for the first target loss value can be expressed as:

[0215] L_total = L_adv + λ1L_recon + λ2L_consistency + λ3L_position

[0216] In this loss function, L_adv represents the adversarial loss, which drives the generator to produce high-quality predicted fusion features; L_recon represents the reconstruction loss, which ensures the information integrity of the predicted fusion features; L_consistency represents the consistency loss, which ensures the semantic alignment of different modal information in the predicted fusion features; L_position represents the localization loss, which optimizes the localization accuracy target based on the predicted fusion features; L_total represents the first target loss value; and λ1, λ2, and λ3 represent the weights corresponding to the reconstruction loss, consistency loss, and localization loss, respectively. In this loss function, the weight corresponding to the adversarial loss L_adv is set to 1 by default.

[0217] like Figure 6The diagram shows the decision-making flowchart of the adaptive fusion strategy in a specific embodiment of this application. First, environmental perception is performed to acquire measurement data. Next, data quality assessment is conducted to determine if the data quality meets the standards. If not, data preprocessing and cleaning are performed, followed by a return to the data quality assessment stage. If the standards are met, modal weights are calculated through a cross-modal fusion layer to determine the initial fusion weights assigned to each modal feature. Then, environmental analysis is performed on the modal features to obtain the environmental context. Based on the environmental context, a fusion strategy is selected, and weights are adjusted and optimized, dynamically adjusting the fusion weights of each modal feature. During this process, a feedback adjustment mechanism is used to optimize the preceding modal weight calculation and fusion strategy selection stages. Finally, the optimal fusion result is output. The entire process, through continuous evaluation, adjustment, and feedback, ensures that the optimal fusion strategy is adaptively selected according to the actual situation to achieve high-quality cross-modal feature fusion.

[0218] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0219] Based on the same inventive concept, this application also provides a multimodal data fusion positioning device for implementing the multimodal data fusion positioning method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the multimodal data fusion positioning device provided below can be found in the limitations of the multimodal data fusion positioning method described above, and will not be repeated here.

[0220] In one exemplary embodiment, such as Figure 7 As shown, a multimodal data fusion positioning device is provided, comprising:

[0221] The acquisition module 401 is used to acquire measurement data of multiple modes of the target terminal;

[0222] The fusion module 402 is used to perform cross-modal feature fusion on the measurement data through a pre-trained cross-modal adversarial network to obtain cross-modal fused features; the cross-modal adversarial network is obtained based on adversarial training on the sample data;

[0223] The positioning module 403 is used to perform positioning calculations on the cross-modal fusion features to obtain the positioning information of the target terminal.

[0224] Each module in the aforementioned multimodal data fusion positioning device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0225] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 3 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs in the non-volatile storage media to run. The database stores measurement data and / or positioning information. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When executed by the processor, the computer program implements a multimodal data fusion positioning method.

[0226] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0227] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0228] Acquire measurement data for multiple modalities of the target terminal;

[0229] The measurement data is fused using a pre-trained cross-modal adversarial network to obtain cross-modal fused features; the cross-modal adversarial network is obtained based on adversarial training on the sample data.

[0230] The location information of the target terminal is obtained by performing localization calculation on the cross-modal fusion features.

[0231] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0232] Acquire measurement data for multiple modalities of the target terminal;

[0233] The measurement data is fused using a pre-trained cross-modal adversarial network to obtain cross-modal fused features; the cross-modal adversarial network is obtained based on adversarial training on the sample data.

[0234] The location information of the target terminal is obtained by performing localization calculation on the cross-modal fusion features.

[0235] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0236] Acquire measurement data for multiple modalities of the target terminal;

[0237] The measurement data is fused using a pre-trained cross-modal adversarial network to obtain cross-modal fused features; the cross-modal adversarial network is obtained based on adversarial training on the sample data.

[0238] The location information of the target terminal is obtained by performing localization calculation on the cross-modal fusion features.

[0239] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0240] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0241] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0242] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A multi-modal data fusion positioning method, characterized in that, The method comprises: acquiring measurement data of multiple modalities of a target terminal; performing cross-modality feature fusion on the measurement data by a pre-trained cross-modality generative adversarial network to obtain cross-modality fusion features; the cross-modality generative adversarial network is obtained based on adversarial training on sample data; performing positioning calculation on the cross-modality fusion features to obtain positioning information of the target terminal.

2. The method of claim 1, wherein, The acquiring of the measurement data of multiple modalities of the target terminal comprises: acquiring original data of multiple modalities of the target terminal, the original data having corresponding time stamps; performing time alignment on the original data based on the time stamps to obtain time-synchronized data of multiple modalities; performing denoising processing on the time-synchronized data to obtain the measurement data of multiple modalities.

3. The method of claim 1, wherein, The cross-modality generative adversarial network comprises a cross-modality generator, the cross-modality generator comprising a cross-modality fusion layer and multiple modality encoders respectively for each modality; The performing of the cross-modality feature fusion on the measurement data by the pre-trained cross-modality generative adversarial network to obtain the cross-modality fusion features comprises: performing feature encoding on the measurement data by the modality encoders corresponding to the same modality to obtain modality features; assigning corresponding fusion weights to each modality feature by the cross-modality fusion layer; mapping the modality features to a unified feature space based on the fusion weights to obtain the cross-modality fusion features.

4. The method of claim 3, wherein, The assigning of the corresponding fusion weights to each modality feature by the cross-modality fusion layer comprises: performing environment analysis on the modality features by the cross-modality fusion layer to obtain environmental contexts, and assigning corresponding fusion weights to each modality feature based on the environmental contexts.

5. The method of claim 3, wherein, The cross-modality generator further comprises a unified representation generator, and the mapping of the modality features to the unified feature space based on the fusion weights to obtain the cross-modality fusion features comprises: mapping the modality features to the unified feature space based on the fusion weights by the cross-modality fusion layer to obtain preliminary fusion features; performing feature standardization processing on the preliminary fusion features by the unified representation generator to obtain the cross-modality fusion features.

6. The method of claim 1, wherein, Before the performing of the cross-modality feature fusion on the measurement data by the pre-trained cross-modality generative adversarial network to obtain the cross-modality fusion features, the method further comprises: acquiring sample data of multiple modalities; inputting the sample data into a to-be-trained generator of a to-be-trained generative adversarial network to perform cross-modality feature fusion to obtain predicted fusion features; inputting the sample data and the predicted fusion features into a to-be-trained multi-layer discriminator of the to-be-trained generative adversarial network to perform classification to obtain a first discrimination result, the first discrimination result being used to indicate whether the input data matches the sample data; determining a first target loss value based on the first discrimination result and the predicted fusion features; adjusting model parameters of the to-be-trained generator based on the first target loss value, and inputting the sample data into the adjusted to-be-trained generator to perform cross-modality feature fusion until the first target loss value meets a training end condition, and taking the current to-be-trained generative adversarial network as the cross-modality generative adversarial network.

7. The method of claim 6, wherein, The sample data is input into a to-be-trained generator of a to-be-trained generative adversarial network for cross-modal feature fusion to obtain predicted fusion features. The sample data is input into a to-be-trained generator of a to-be-trained generative adversarial network for cross-modal feature fusion to obtain reference fusion features. The sample data and the reference fusion features are input into a to-be-trained multi-layer discriminator of the to-be-trained generative adversarial network for classification to obtain a second discrimination result, which is used to indicate whether the input data matches the sample data. A second target loss value is determined based on the second discrimination result, and model parameters of the to-be-trained multi-layer discriminator are adjusted based on the second target loss value to obtain an updated to-be-trained multi-layer discriminator. The sample data is re-input into the to-be-trained generator for cross-modal feature fusion to obtain predicted fusion features.

8. The method of claim 6, wherein, The to-be-trained multi-layer discriminator includes a modal discriminator, and the first discrimination result includes a modal discrimination result. The sample data and the predicted fusion features are input into the to-be-trained multi-layer discriminator of the to-be-trained generative adversarial network for classification to obtain a first discrimination result, which includes: The predicted fusion features are reconstructed to obtain reconstructed data corresponding to multiple modalities. The sample data and the reconstructed data are input into the modal discriminator for classification to obtain a modal discrimination result, which is used to indicate whether the input data matches the sample data.

9. The method of claim 6, wherein, The to-be-trained multi-layer discriminator includes a fusion discriminator, and the first discrimination result includes a fusion discrimination result. The sample data and the predicted fusion features are input into the to-be-trained multi-layer discriminator of the to-be-trained generative adversarial network for classification to obtain a first discrimination result, which includes: The real fusion features corresponding to the sample data are obtained. The real fusion features and the predicted fusion features are input into the fusion discriminator for classification to obtain a fusion discrimination result, which is used to indicate whether the input data matches the real fusion features.

10. The method of claim 6, wherein, The to-be-trained multi-layer discriminator includes a positioning discriminator, and the first discrimination result includes a positioning discrimination result. The sample data and the predicted fusion features are input into the to-be-trained multi-layer discriminator of the to-be-trained generative adversarial network for classification to obtain a first discrimination result, which includes: The real positioning corresponding to the sample data is obtained. The predicted positioning is obtained based on the predicted fusion features. The real positioning and the predicted positioning are input into the positioning discriminator for classification to obtain a positioning discrimination result, which is used to indicate whether the input data matches the real positioning.

11. The method of claim 6, wherein, The first discrimination result includes a modal discrimination result, a fusion discrimination result, and a positioning discrimination result. The first target loss value is determined based on the first discrimination result and the predicted fusion features, which includes: The modal discrimination result, the fusion discrimination result, and the positioning discrimination result are weighted and summed to obtain an adversarial loss. reconstructing the predicted fusion feature to obtain reconstruction data corresponding to multiple modalities, and calculating an error between the sample data and the reconstruction data to obtain a reconstruction loss; obtaining a real fusion feature corresponding to the sample data, and calculating an error between the real fusion feature and the predicted fusion feature to obtain a consistency loss; obtaining a real positioning corresponding to the sample data, and performing positioning calculation based on the predicted fusion feature to obtain a predicted positioning, and calculating an error between the real positioning and the predicted positioning to obtain a positioning loss; performing weighted summation on the adversarial loss, the reconstruction loss, the consistency loss and the positioning loss to obtain a first target loss value.

12. A multi-modal data fusion positioning apparatus, characterized by, The device comprises: an acquisition module configured to acquire measurement data of multiple modalities of a target terminal; a fusion module configured to perform cross-modal feature fusion on the measurement data by using a pre-trained cross-modal adversarial network to obtain cross-modal fusion features; the cross-modal adversarial network is obtained based on adversarial training on sample data; a positioning module configured to perform positioning calculation on the cross-modal fusion features to obtain positioning information of the target terminal.

13. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 11.

14. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 11.

Images