A direct localization method based on multimodality and self-attention

Abstract

This invention relates to a multimodal self-attention direct localization (MMSA-DPD) method for multiple radiation sources, belonging to the field of satellite positioning technology. It achieves high-precision localization of multiple radiation sources by intercepting their radiation signals, addressing the challenges of target monitoring in complex electromagnetic environments. Traditional model-driven methods are limited by model mismatch and high-dimensional search, while existing data-driven methods still have limitations in feature utilization efficiency and scene generalization ability. To address the limitations of existing methods, this method first constructs a multimodal heterogeneous feature extraction network for direct location determination (DPD) tasks. The system integrates four types of heterogeneous information: time-frequency features of the signal, time-domain waveform, receiver station status, and array geometry, forming a highly complementary and comprehensive feature representation. Then, a self-attention-based adaptive mechanism is introduced to achieve deep interaction and dynamic fusion of multimodal features, simultaneously completing implicit inference of the number of radiation sources and accurate regression of the target position. Furthermore, a mask supervision strategy is designed to support end-to-end joint optimization in scenarios with dynamically changing radiation source numbers. Compared with existing representative methods, MMSA-DPD significantly improves localization accuracy under low signal-to-noise ratio and multiple radiation source conditions.

Description

Technical Field

[0001] This invention relates to a multimodal self-attention direct localization (MMSA-DPD) method for direct localization of multiple radiation sources, belonging to the field of satellite positioning technology. Background Technology

[0002] Passive localization technology for radiation sources achieves location based on the reception and processing of target radiation signals. Since it does not require actively emitting detection signals, it offers advantages such as strong concealment and long operating range.

[0003] Traditional radiation source localization typically employs a two-step method: first, it estimates intermediate parameters such as Time Difference of Arrival (TDOA) and Angle of Arrival (AOA), and then calculates the location based on geometric relationships. However, this process results in independent estimation of intermediate parameters, leading to the loss of joint information from the original signal, and its performance is limited under low signal-to-noise ratio (SNR) and complex channel conditions.

[0004] To address this limitation, existing research has proposed the Direct Position Determination (DPD) framework. By constructing a unified cost function, it performs global optimization directly on the original signal space, avoiding information loss and error propagation in the "two-step method," thus opening a new avenue for improving positioning robustness under low signal-to-noise ratio conditions. However, classical DPD methods are mostly model-driven, requiring the construction of accurate propagation and signal models. Therefore, the common limitation of these methods lies in their reliance on pre-defined mathematical models (such as array manifolds and signal correlation structures). In real-world complex environments with model mismatch (such as array errors and non-line-of-sight propagation), their positioning accuracy and robustness face significant challenges. Furthermore, the optimization design of the cost function often involves high computational complexity, leading to high-dimensional data search when facing multiple radiation sources.

[0005] In recent years, deep learning has provided new ideas for solving problems such as model mismatch sensitivity, complex cost function design, and high-dimensional search in distributed localization (DPD). Early research often used neural networks as auxiliary tools for region pre-screening or error correction to help construct distributed cost functions, but their core localization engines still did not break away from the optimization framework based on specific signal models.

[0006] To completely eliminate the dependence on explicit cost functions, although there has been research on end-to-end localization architectures centered on neural networks, existing deep learning-based methods still face challenges. These include insufficient feature utilization, relying heavily on single-modal signal representations (such as using only time-frequency maps or array snapshots), failing to systematically fuse multi-dimensional heterogeneous information about the signal across time, frequency, space, and state; and insufficient dynamic adaptability, as their feature fusion strategies are usually fixed and preset, unable to dynamically adjust the dependence on different feature sources based on real-time scene conditions of the input signal (such as signal-to-noise ratio and source signal characteristics), thus limiting the model's robustness and generalization ability in complex dynamic environments. These problems are particularly prominent in low signal-to-noise ratio environments.

[0007] This invention constructs a framework that can fully integrate multimodal information and achieve adaptive weighted fusion, thereby improving the positioning accuracy in low signal-to-noise ratio and multi-radiation source scenarios. Summary of the Invention

[0008] The purpose of this method is to address the shortcomings and deficiencies of existing deep learning-based localization methods, which rely heavily on single signal representations and lack the ability to systematically fuse and dynamically adapt to multi-source heterogeneous information. To address these shortcomings, a new method, Multimodal Self-Attention Direct Localization (MMSA-DPD), is proposed.

[0009] The innovations of this invention are: constructing a multimodal heterogeneous feature extraction network, proposing a structured feature integration strategy, and unifying and fusing four types of heterogeneous information—time-frequency, time-domain, receiver station status, and array structure—to form robust and complete multimodal features. Simultaneously, an adaptive joint inference mechanism based on self-attention is designed to achieve deep multimodal interaction, eliminating the need for prior knowledge of the number of targets and simultaneously completing implicit inference of the number of radiation sources and accurate regression of target locations.

[0010] The present invention is achieved using the following technical solution.

[0011] A multimodal self-attention direct localization (MMSA-DPD) method includes the following steps: Step 1: Data Preprocessing. The input contains three types of raw data: complex signals received by each array element, ephemeris data (position, velocity) of the receiving station, and the precise distribution of the array elements. To ensure the stability and efficiency of network training, this data needs to be preprocessed.

[0012] Specifically, step 1 includes the following steps: Step 1.1: Time-frequency graph output processing.

[0013] To capture the time difference of arrival (TDOA) and frequency difference of arrival (FDOA) information in the signal, a short-time Fourier transform (STFT) is performed on the received signal of each array element to generate a time-frequency diagram. The STFT window type and length are fixed to ensure the stability of the results. The time-frequency diagram is obtained through STFT, and finally, each receiving station generates its own time-frequency diagram, which is then normalized.

[0014] Step 1.2: Time-Domain Snapshot Extraction. To utilize the instantaneous phase difference (including AOA information) of the signal arriving at different array elements, a snapshot of the complex signal of all array elements at the initial moment is extracted. The I (real part) and Q (imaginary part) data of all array elements at each receiving station are arranged in two dimensions to form a time-domain snapshot matrix, and then normalized. The real and imaginary parts are separated and combined to form 2-channel data.

[0015] Step 1.3: System status information normalization. The position and velocity of the receiving station, as well as the position of each array element relative to the center of the receiving station, are normalized to eliminate the influence of dimensions and constrain the data range to the interval [-1,1] to facilitate network learning.

[0016] Step 2: Network Model Design. Construct a model parameterized by a deep neural network.

[0017] Specifically, step 2 includes the following steps: Step 2.1: Construct a nonlinear mapping function parameterized by a deep neural network to achieve end-to-end regression target location estimation for multimodal data under low signal-to-noise ratio conditions.

[0018] Step 2.2: Construct the network's minimum joint loss function, which includes coordinate regression loss and classification loss.

[0019] Step 2.3: To address the issue of the variable number of radiation sources in the training data, a masking mechanism is introduced. First, the coordinate labels of each sample are padded to a fixed dimension, with any insufficient space filled with special values. During training, a Boolean mask is dynamically generated by comparing the true coordinates with the padded values ​​to identify valid radiation sources.

[0020] The localization error loss calculates the mean square error only for the effective radiation sources marked by the mask, thus achieving unified supervision of variable-length outputs. The number classification loss directly uses the true number obtained from mask statistics as the supervision signal. The masking mechanism enables the model to jointly optimize the location regression and number classification tasks end-to-end, while maintaining adaptability to the uncertainty of the number of radiation sources in real-world scenarios.

[0021] Step 3: Input the data from Step 1 into the MMSA-DPD network. The MMSA-DPD network adopts a multi-branch parallel processing architecture.

[0022] Specifically, step 3 includes the following steps: Step 3.1: Time-Frequency Map Feature Extraction Module. The time-frequency map already possesses good time-frequency feature representation. This module employs a three-layer convolutional neural network (CNN) for progressive feature extraction and compression. The network sequentially uses convolutional kernels for feature mapping and intersperses max-pooling layers with a stride of 2 for downsampling, gradually reducing the spatial size while adjusting the number of channels. The final feature map, after flattening, outputs a 512-dimensional feature vector.

[0023] Step 3.2: Temporal Quick Feature Extraction Module. This module is designed to fully preserve and extract subtle phase information between array elements. First, the data is reconstructed into a format suitable for one-dimensional convolution. The core part adopts a structure containing a large initial convolutional layer and two residual blocks. The initial convolutional kernel aims to fuse the I / Q information of all array elements at once, and the residual blocks avoid the gradient vanishing problem in deep networks through skip connections, ensuring that the key low-level phase information is preserved. Finally, a 32-dimensional feature vector is output.

[0024] Step 3.3: Receiving Station State Feature Extraction Module. The receiving station state information (position, velocity) is structured vector data. This module uses a lightweight two-branch convolutional structure to capture the correlation between features. The first layer uses a convolutional kernel to learn the relationship between different state dimensions within the same receiving station. The second layer uses a convolutional kernel to learn the spatial distribution relationship of the same state dimension between different receiving stations. Finally, a 32-dimensional feature vector is output.

[0025] Step 3.4: Element position information describes the physical geometry of the array. This module uses a multilayer perceptron (MLP) for processing, whose fully connected characteristics excel at learning global nonlinear mappings between low-dimensional coordinates. The network first maps the 3D coordinates of each element to a high-dimensional space (64-dimensional) to enhance feature representation. Then, it aggregates all element information through fully connected layers to generate a 256-dimensional global array geometric feature vector.

[0026] Step 4: Feature Fusion Based on Self-Attention. Design a feature fusion module based on the self-attention mechanism. This mechanism dynamically calculates the global dependencies between all feature dimensions in a data-driven manner and generates adaptive fusion weights accordingly.

[0027] Specifically, step 4 includes the following steps: Step 4.1: Use the concatenated hybrid feature vector as input. Project it into query, key, and value vectors using three independent learnable linear transformation matrices, respectively.

[0028] Step 4.2: Since the input has no sequence dimension, construct the feature dimension and the associated computation space of the feature dimensions. Perform matrix multiplication between the query matrix and the expanded key matrix to obtain the attention weights for each feature dimension, and then perform scaling. For each input sample, the attention weight matrix is ​​dynamically generated. Step 4.3: The output fused feature is a new feature representation that aggregates global context information and is adaptively optimized. This feature is then fed into two subsequent parallel fully connected network branches: the coordinate regression branch and the radiation source number classification branch.

[0029] The coordinate regression branch employs a five-layer fully connected network for progressive dimensionality reduction and supports training with a variable number of radiation sources through a masking mechanism. The quantity classification branch consists of a three-layer fully connected network, outputting a logical value for the number of radiation sources, with its true label determined by the number of valid positions in the mask. The mask provides dual supervision: identifying valid positions for coordinate regression and providing category labels for quantity classification. By sharing the mask, the two branches are jointly optimized within a unified framework, simultaneously achieving radiation source number inference and position estimation, enabling the model to adapt to localization scenarios with an unknown number of radiation sources in an end-to-end manner. Beneficial effects

[0030] The method of the present invention has the following advantages compared with the prior art: 1. This invention constructs a multimodal heterogeneous feature extraction network for DPD tasks. A structured feature integration strategy is proposed to systematically represent and fuse four types of heterogeneous information: time-frequency features of the signal, time-domain waveform, receiver station status, and array geometry. This forms a highly complementary and comprehensive multimodal feature representation, providing high-dimensional and robust feature primitives for subsequent high-precision positioning.

[0031] 2. This invention designs an adaptive joint inference mechanism based on self-attention. By introducing a self-attention module, deep interaction and adaptive fusion of multimodal features are achieved, and its global modeling capability is utilized to simultaneously complete the implicit inference of the number of radiation sources and the accurate estimation of target location. This mechanism does not rely on prior information about the number of targets, effectively addressing the dual challenges of unknown target number and high-precision positioning in complex scenarios with low signal-to-noise ratio and multiple radiation sources.

[0032] 3. This invention achieves good localization performance in scenarios with multiple radiation sources and low signal-to-noise ratios. Systematic simulations were conducted within a signal-to-noise ratio range of -15 dB to -5 dB. The results show that the proposed method maintains high robustness in both noisy and multi-target environments, and its localization accuracy is significantly better than existing traditional methods. Furthermore, interpolation and extrapolation tests within the training interval demonstrate good generalization ability, further validating the engineering applicability of the proposed method. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the process in the method and embodiments of the present invention; Figure 2 This is a network structure diagram of the time-frequency graph feature extraction module; Figure 3 This is a diagram of the initial convolutional layer network structure for the temporal snapshot feature extraction module; Figure 4 This is a diagram of the residual block network structure of the temporal snapshot feature extraction module; Figure 5 This is a network structure diagram of the receiving station status feature extraction module; Figure 6 The network structure diagram of the array element geometric feature extraction module; Figure 7 This is a network structure diagram of the feature fusion module. Detailed Implementation

[0034] The technical solution of the method of the present invention will be clearly and completely described below with reference to the accompanying drawings and embodiments. Example

[0035] This embodiment illustrates the specific implementation process of the method of the present invention.

[0036] Step 1: Perform data preprocessing on the received radiation source signal, extract the complex signal snapshots of all array elements at time t=0, and obtain the time-frequency diagram through short-time Fourier transform. Then, normalize the time-domain snapshots, time-frequency diagrams, receiver position and velocity, and array element positions.

[0037] Step 2: Input the data processed in Step 1 into different pre-trained network modules for feature extraction.

[0038] Step 2.1: For the time-frequency graph, a CNN network is used, with the specific structure as follows: Figure 2 As shown, TDOA / FDOA related features are extracted.

[0039] Step 2.2: For temporal snapshots, a network combining convolutional layers and residual fast loops is used, with the specific structure as follows: Figure 3 and Figure 4 As shown, phase difference and AOA-related features are extracted.

[0040] Step 2.3: For the receiving station status information (position, velocity), a two-branch convolutional structure is used, the specific structure is as follows: Figure 5 As shown, geometric and dynamic constraints are provided.

[0041] Step 2.4: The spatial distribution of array elements is processed using a multilayer perceptron (MLP), the specific structure of which is as follows: Figure 6 As shown, an array manifold prior is provided.

[0042] Step 3: Concatenate the various features extracted in Step 2, and combine them with a self-attention mechanism to achieve feature fusion. The specific structure is as follows: Figure 7 As shown.

[0043] Step 3.1: The four types of features extracted in Step 2 are concatenated and input into the feature fusion processing module based on the self-attention mechanism. The attention weight calculation covers all feature dimensions, so that a certain dimension of the time-frequency feature can directly interact with a certain dimension of the array geometry feature, thereby capturing deep correlations such as the energy distribution at a specific frequency point and the gain in a specific direction of the array, while realizing dynamic weight allocation.

[0044] Step 3.2: Feed the fused features obtained in Step 3.1 into the two subsequent parallel fully connected network branches, namely the coordinate regression branch and the radiation source number classification branch.

Claims

1. A multi-modal self-attention direct positioning method applied to a passive positioning scene, characterized in that, include: Constructing a multimodal heterogeneous fusion tensor: The received complex signal is subjected to short-time Fourier transform to generate a time-frequency map, the initial moment of the complex signal is extracted to form a time-domain phase matrix, and the receiver ephemeris and array element distribution are normalized; the above time-frequency map, time-domain phase matrix, ephemeris vector and array geometry vector are spliced ​​according to a preset topology to form a unified input tensor containing four-dimensional feature channels of time-frequency energy, instantaneous phase, motion state and spatial topology; The unified input tensor is input into a multimodal self-attention direct localization network, the network comprising: The multi-branch parallel feature extraction module uses convolutional neural networks, residual networks, and multilayer perceptrons to extract time-frequency features, phase features, state features, and geometric features from different feature channels of the unified input tensor. The self-attention feature fusion module concatenates the extracted multimodal features and calculates the global dependency weights through a query, key, and value projection mechanism to generate an adaptive fused context feature representation. The dual-task joint inference module, based on the context feature representation, synchronously performs coordinate regression and radiation source number classification through a shared masking mechanism, and outputs the target location estimate and the number of radiation sources. The masking mechanism dynamically identifies effective radiation sources during the training phase, jointly optimizes the location regression loss and the quantity classification loss, and achieves end-to-end localization without prior knowledge of the number of targets.

2. The method of claim 1, wherein, The multi-branch parallel feature extraction module includes: The time-frequency branch employs multi-layer convolution and pooling structures to perform spatial downsampling and feature compression on the time-frequency graph. The temporal branch uses a structure that includes a large initial convolutional layer and residual connection blocks to extract the instantaneous phase difference features between array elements. The state branch adopts a dual-branch convolutional structure to learn the state dimension association within a single station and the spatial distribution association between multiple stations, respectively. The geometric branch uses a fully connected mapping structure to map the coordinates of low-dimensional matrix elements to a high-dimensional space and aggregate them into global geometric features.

3. The method of claim 1, wherein, The self-attention feature fusion module is specifically used for: Project the multimodal feature vectors into a query matrix, a key matrix, and a value matrix; With no sequence dimension input, we construct the association computation space between feature dimensions and generate a dynamic attention weight matrix through the interaction of the query matrix and the key matrix. The value matrix is ​​weighted and aggregated using the dynamic attention weight matrix to obtain a feature representation that incorporates global context information.

4. The method according to claim 1, characterized in that, The masking mechanism in the dual-task joint inference module specifically includes: Fill the sample coordinate labels to a fixed dimension, and fill any insufficient parts with mask values; During training, Boolean masks are generated by comparing the real coordinates with the mask values ​​to identify the locations of effective radiation sources; The coordinate regression loss calculates the mean square error based solely on the valid positions marked by the Boolean mask, while the quantity classification loss uses the valid quantity counted by the Boolean mask as the supervision label, thus achieving joint backpropagation optimization of the two tasks.

Images