Photoelectric-radar space-time cooperative pulse route identification method

Abstract

The invention belongs to the technical field of multi-modal data fusion identification, designs a photoelectric-radar space-time cooperative pulse route identification method, realizes efficient fusion and utilization of multi-modal data, and achieves the effect of accurately identifying a target when a modal part is lost or interfered. Compared with an existing method, the method can be applied to the condition of mode missing, and the application scene is wider. Time sequence-space joint embedding is specially used for space-time alignment feature extraction processing of photoelectric and radar data, the feature extraction capability of the radar data is higher, and a multi-modal data fusion recognition task is facilitated. Meanwhile, a gating routing dynamic selection mechanism is introduced, robust features can be clearly learned from missing modal data, and missing modal masks are selectively added through gating in the reasoning stage to improve the interpretability of the result.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal data fusion and recognition, and in particular relates to a pulse routing recognition method based on a gated routing dynamic selection mechanism applicable to partially missing or disturbed modalities. Background Art

[0002] In recent years, the development of recognition tasks has shown a trend towards multimodal fusion. Multimodal fusion technology integrates data from different modalities (such as images, voice, and text) to mine complementary information between them, thereby achieving more comprehensive and accurate recognition. Multimodal fusion recognition technology places high demands on data from different modalities, which must have certain temporal, spatial, or semantic correlations. Since the temporal and spatial information of optoelectronic image data and radar data is typically misaligned, how to comprehensively utilize optoelectronic and radar data is a challenge that needs to be addressed. Summary of the Invention

[0003] In view of this, the present invention provides a pulse routing identification method for photoelectric-radar spatiotemporal coordination.

[0004] The present invention can be achieved by the following technical means:

[0005] A pulse routing identification method for photoelectric-radar spatiotemporal coordination includes the following steps:

[0006] Step 1: Preprocess the data of different modes separately. For radar track data, fixed-length radar track points are obtained by truncation or interval sampling. If the number of radar track points is less than the set value, zeros are added to process it to a fixed length. For photoelectric image data, samples are cropped and scaled so that the target is located in the center of the image, and the surrounding environment is cropped.

[0007] Step 2: Feature extraction is performed on data of different modalities using a proprietary encoder model. For radar track data, radar track embedding is obtained through time-position embedding and longitude-latitude embedding, and then the radar track embedding is encoded using a radar-specific transformer encoder to obtain radar track features. For optoelectronic image data, optoelectronic infrared features and optoelectronic visible light features are encoded using a dedicated ResNet network and a dedicated ResNet network, respectively, to obtain optoelectronic infrared features and optoelectronic visible light features, collectively referred to as optoelectronic pixel features. Then, a cross-attention mechanism across heterogeneous modalities is used to align optoelectronic pixel features with radar track features, resulting in an optoelectronic feature vector after optoelectronic-radar spatiotemporal alignment.

[0008] Step 3: A gated routing dynamic selection mechanism is used to select routes based on radar track characteristics, optoelectronic infrared characteristics, and optoelectronic visible light characteristics. A missing mask is selectively added through gating to obtain a gated routing selection result consisting of the missing mask.

[0009] Step 4: Fuse the radar track features, optoelectronic infrared features, and optoelectronic visible light features to obtain a fused feature vector. Input the gated routing selection result and the fused feature vector into a heterogeneous topology adaptive fusion model composed of a transformer encoder to obtain the modal recognition result after gated routing selection, realize the recognition of any input combination from single modality to multimodality, and obtain the final multimodal fusion recognition result.

[0010] Furthermore, the specific process of time position embedding in step 2 is as follows: Assumptions is a timestamp in the input timestamp sequence, represents the index of the vector, Used to distinguish odd and even positions, Indicates the maximum length of the preset sequence, and the time-related coding is expressed as: ; By using sine and cosine to distinguish between odd and even positions, then for timestamps The time code is represented as A vector of dimensions: ; Therefore, the encoding of all timestamps of a radar track is Expressed as: .

[0011] Furthermore, the specific process of embedding longitude and latitude in step 2 is as follows: Assume that the longitude and latitude of the track are respectively and To do this, first convert the degrees to radians: ; Then convert the latitude and longitude of the track points into the ECEF coordinate system using and Represents the WGS-84 earth ellipsoid parameters, and the calculation formula is expressed as: ; For the three coordinate system parameters 、 and , respectively, using one-dimensional convolution for embedding and encoding the coordinates dimension, and use LeakyReLU nonlinear activation function for nonlinear processing; use Represents the embedded one-dimensional convolution operation, using Represents the operation of the activation function, then the embedding of longitude and latitude is expressed as: .

[0012] Furthermore, the specific process of step 3 is: use 、 and They represent radar track features, photoelectric infrared features, and photoelectric visible light features respectively. The missing modal data is represented by a feature vector of all 0s. The fused features are expressed as: ; According to the dynamic selection mechanism of gated routing, a mask of all negative infinity is added when the modal input is 0. The setting of the mask for any modality is expressed as follows: ; Among them, dim represents the dimension of feature setting; use 、 and They represent the radar feature corresponding mask, the photoelectric infrared feature mask, and the photoelectric visible light feature mask respectively. Then the fusion mask, i.e., the gated routing selection result, is expressed as: .

[0013] Furthermore, after the missing mask is selectively added through gating in step 3, the missing mask can be corrected through manual intervention to obtain a gated routing result consisting of the missing mask.

[0014] The beneficial effects of the present invention are:

[0015] 1. This paper designs a joint time-space embedding method, which converts radar time series data into high-level semantic embedding information through a time-series attention model, and aligns it with photoelectric image features through spatiotemporal position encoding, so that the extracted features are suitable for multimodal fusion recognition algorithms.

[0016] 2. The method of the present invention designs a dynamic gating function for the scenario where the input modality is partially missing. In the training phase, an adversarial mask simulation strategy is adopted to force the model to learn robust representations of different modalities under the condition of random modality missing. In the inference phase, the missing modality mask is selectively added through gating according to the modality of the actual input data to automatically activate the optimal feature transfer path of the input modality.

[0017] 3. This paper proposes a shared-private topology. The private branch designs proprietary encoder models for different modalities, while the shared branch implements a cross-modal attention mechanism to achieve equal-dimensional latent space mapping of multimodal features. This architecture, through topological consistency constraints, ensures complete sharing of model parameters when inputs are combined from different modalities. It supports any input combination from single modality to multimodality, and the fixed number of model parameters eliminates the need to train multiple sub-models for different modal combinations, significantly reducing computational resource requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is the overall flow chart of the pulse routing identification method of the present invention.

[0019] Figure 2 This is a processing principle diagram of the pulse routing identification method of the present invention. DETAILED DESCRIPTION

[0020] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is described in further detail below.

[0021] A pulse routing identification method based on photoelectric-radar spatiotemporal coordination, such as Figure 1 and Figure 2 As shown in Figure 1, the processing is divided into three parts, including the optoelectronic-radar spatiotemporal alignment attention module (processing of steps 1 and 2), the gated routing dynamic selection module (processing of step 3), and the parameter-sharing heterogeneous topology adaptation module (processing of step 4).

[0022] The specific steps include:

[0023] Step 1: Preprocess the data of different modes separately. For radar track data, obtain radar track points of fixed length by truncation or interval sampling. If the number of radar track points is less than the set value, fill it with zeros to process it to a fixed length. For photoelectric image data, crop and scale the samples so that the target is located in the center of the image, and crop the surrounding environment to reduce environmental interference.

[0024] Step 2: Use a proprietary encoder model to extract features from data of different modalities. For radar track data, radar track embedding is obtained through time position embedding and longitude and latitude embedding, and then the radar track embedding is encoded using a radar-specific transformer encoder to obtain radar track features. For photoelectric image data, photoelectric infrared-specific ResNet network and photoelectric visible light-specific ResNet network are used to encode, respectively, to obtain photoelectric infrared features and photoelectric visible light features, collectively referred to as photoelectric pixel features. Then, the alignment of photoelectric pixel features to radar track features is achieved through a cross-heterogeneous modal cross-attention mechanism to obtain the photoelectric feature vector after photoelectric-radar spatiotemporal alignment. By designing a special time position encoding, it is a vector about specific time information. The vector of dimension is related only to the timestamp interval and has nothing to do with its position in the sequence.

[0025] The specific process of time position embedding is as follows: Assumptions Is a timestamp in the input timestamp sequence. If represents the index of the vector, Used to distinguish odd and even positions, Indicates the maximum length of the preset sequence, so the time-related coding can be expressed as: ; By using sine and cosine to distinguish between odd and even positions, then for timestamps The time code can be expressed as A vector of dimensions: ; Therefore, the encoding of all timestamps of a track It can be expressed as: .

[0026] The specific process of embedding longitude and latitude is as follows: Assume that the longitude and latitude of the track are respectively and To do this, first convert the degrees to radians: ; Then convert the latitude and longitude of the track points into the ECEF coordinate system using and Represents the WGS-84 earth ellipsoid parameters, and the calculation formula is expressed as: ; For the three coordinate system parameters 、 and , respectively, using one-dimensional convolution for embedding and encoding the coordinates dimension, and use LeakyReLU nonlinear activation function for nonlinear processing; use Represents the embedded one-dimensional convolution operation, using Represents the operation of the activation function, then the embedding of longitude and latitude is expressed as: .

[0027] In step 3, the dynamic selection mechanism of gated routing is used to select according to the radar track characteristics, optoelectronic infrared characteristics, and optoelectronic visible light characteristics. The missing mask is selectively added through gating, thereby automatically activating the optimal feature transfer path of the input modality and generating the radar mask and optoelectronic mask. At the same time, the missing mask can also be corrected through manual intervention to obtain the gated routing selection result composed of the missing mask.

[0028] use 、 and They represent radar track features, photoelectric infrared features, and photoelectric visible light features respectively. The missing modal data is represented by a feature vector of all 0s. The fused features are expressed as: ; According to the dynamic selection mechanism of gated routing, a mask of all negative infinity is added when the modal input is 0. The setting of the mask for any modality is expressed as follows: ; Among them, dim represents the dimension of feature setting; use 、 and They represent the radar feature corresponding mask, the photoelectric infrared feature mask, and the photoelectric visible light feature mask respectively. Then the fusion mask, i.e., the gated routing selection result, is expressed as: .

[0029] Step 4: Fuse the radar track features, optoelectronic infrared features, and optoelectronic visible light features to obtain a fused feature vector. Input the gated routing selection result and the fused feature vector into a heterogeneous topology adaptive fusion model composed of a transformer encoder to obtain the modal recognition result after gated routing selection, realize the recognition of any input combination from single modality to multimodality, and obtain the final multimodal fusion recognition result.

[0030] For the parameter-sharing heterogeneous topology adaptive module, the private branch designs proprietary encoder models for different modalities to extract features from different modal data. The shared branch designs a cross-modal attention mechanism to achieve equal-dimensional latent space mapping of multimodal features for identification of fused features. The shared part ensures complete sharing of model parameters for inputs of different modal combinations through topological consistency constraints and dynamic masks generated by gated routing. It supports any input combination from single modality to multimodality, and the number of model parameters is fixed, eliminating the need to train multiple sub-models for different modal combinations. The final model output is the identified category. During training, the cross-entropy loss is calculated between this category and the true category, allowing backpropagation to continuously optimize the model.

[0031] In summary, the present invention adopts a pulse routing recognition method of photoelectric-radar spatiotemporal coordination, and the method of the present invention can be used to provide support and basis for solving multimodal fusion recognition technology.

Claims

1. A pulse routing identification method for photoelectric-radar spatiotemporal coordination, characterized in that: The steps include: Step 1: Preprocess the data of different modes separately. For radar track data, fixed-length radar track points are obtained by truncation or interval sampling. If the number of radar track points is less than the set value, zeros are added to process it to a fixed length. For photoelectric image data, samples are cropped and scaled so that the target is located in the center of the image, and the surrounding environment is cropped. Step 2: Feature extraction is performed on data of different modalities using a proprietary encoder model. For radar track data, radar track embedding is obtained through time position embedding and longitude and latitude embedding, and then the radar track embedding is encoded using a radar-specific transformer encoder to obtain radar track features. For photoelectric image data, photoelectric infrared dedicated ResNet network and photoelectric visible light dedicated ResNet network are used to encode, respectively, to obtain photoelectric infrared features and photoelectric visible light features, collectively referred to as photoelectric pixel features. Then, the photoelectric pixel features are aligned with the radar track features through a cross-heterogeneous modality cross-attention mechanism to obtain the photoelectric feature vector after photoelectric-radar spatiotemporal alignment. Step 3: A gated routing dynamic selection mechanism is used to select routes based on radar track characteristics, optoelectronic infrared characteristics, and optoelectronic visible light characteristics. A missing mask is selectively added through gating to obtain a gated routing selection result consisting of the missing mask. Step 4: Fuse the radar track features, optoelectronic infrared features, and optoelectronic visible light features to obtain a fused feature vector. Input the gated routing selection result and the fused feature vector into a heterogeneous topology adaptive fusion model composed of a transformer encoder to obtain the modal recognition result after gated routing selection, realize the recognition of any input combination from single modality to multimodality, and obtain the final multimodal fusion recognition result.

2. The method for pulse routing identification based on spatiotemporal coordination of photoelectric and radar according to claim 1, characterized in that: The specific process of time position embedding in step 2 is: Assumptions is a timestamp in the input timestamp sequence, represents the index of the vector, Used to distinguish odd and even positions, Indicates the maximum length of the preset sequence, and the time-related coding is expressed as: ； By using sine and cosine to distinguish between odd and even positions, then for timestamps The time code is represented as A vector of dimensions: ； Therefore, the encoding of all timestamps of a radar track is Expressed as: 。 3. The method for pulse routing identification based on spatiotemporal coordination of photoelectric and radar according to claim 2, characterized in that: The specific process of embedding longitude and latitude in step 2 is as follows: Assume that the longitude and latitude of the track are respectively and To do this, first convert the degrees to radians: ； Then convert the latitude and longitude of the track points into the ECEF coordinate system using and Represents the WGS-84 earth ellipsoid parameters, and the calculation formula is expressed as: ； For the three coordinate system parameters 、 and , respectively, using one-dimensional convolution for embedding and encoding the coordinates dimension, and use LeakyReLU nonlinear activation function for nonlinear processing; use Represents the embedded one-dimensional convolution operation, using Represents the operation of the activation function, then the embedding of longitude and latitude is expressed as: 。 4. The method for pulse routing identification based on spatiotemporal coordination of photoelectric and radar according to claim 1, characterized in that: The specific process of step 3 is as follows: use 、 and They represent radar track features, photoelectric infrared features, and photoelectric visible light features respectively. The missing modal data is represented by a feature vector of all 0s. The fused features are expressed as: ； According to the dynamic selection mechanism of gated routing, a mask of all negative infinity is added when the modal input is 0. The setting of the mask for any modality is expressed as follows: ； Among them, dim represents the dimension of feature setting; use 、 and They represent the radar feature corresponding mask, the photoelectric infrared feature mask, and the photoelectric visible light feature mask respectively. Then the fusion mask, i.e., the gated routing selection result, is expressed as: 。 5. The method for pulse routing identification based on spatiotemporal coordination of photoelectric and radar according to claim 1, characterized in that: After selectively adding the missing mask through gating in step 3, the missing mask can be corrected through manual intervention to obtain a gated routing result consisting of the missing mask.

Images