Optoelectronic-radar spatiotemporal coordination pulse routing identification method

Abstract

The application belongs to the technical field of multi-modal data fusion recognition, and designs a kind of photoelectric-radar space-time coordination pulse routing identification method, realizes the efficient fusion and utilization of multi-modal data, reaches the effect that target can be accurately identified when modal part is missing or disturbed.Compared with the prior art, the method can be applied to the case of modal missing, and is more widely applicable.The time-space joint embedding is dedicated to the space-time alignment feature extraction processing of photoelectric and radar data, and has stronger feature extraction capability for radar data, which is beneficial to multi-modal data fusion recognition task.Meanwhile, the application introduces a gating routing dynamic selection mechanism, which can learn robust features from missing modal data, and selectively add missing modal mask in the reasoning stage to improve the interpretability of the result.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of multi-modal data fusion recognition, and particularly relates to a pulse routing recognition method based on a gating routing dynamic selection mechanism suitable for modal partial loss or interference. BACKGROUND

[0002] In recent years, the development of recognition tasks has shown a trend of multi-modal fusion. Multi-modal fusion technology integrates data of different modalities (such as images, speech, text, etc.), mines complementary information between modalities, and thus achieves more comprehensive and accurate recognition. Multi-modal fusion recognition technology has high requirements for data of different modalities, and the data of different modalities should have certain correlation in time, space or semantics. The information of optical-electric image data and radar data is usually not aligned in time and space, so how to comprehensively utilize optical-electric data and radar data is a difficult problem to be solved. SUMMARY

[0003] Therefore, the application provides an optical-electric-radar spatiotemporal collaborative pulse routing recognition method.

[0004] The application can be implemented through the following technical means:

[0005] An optical-electric-radar spatiotemporal collaborative pulse routing recognition method comprises the following steps:

[0006] Step 1, preprocessing different modal data respectively; wherein for radar track data, fixed-length radar track points are obtained by truncation or interval sampling method, and if the radar track points are less than a set value, the fixed-length radar track points are processed by 0 padding; for optical-electric image data, sample cropping and scaling are performed so that the target is located in the center of the image and the surrounding environment range is cropped;

[0007] Step 2, respectively extracting features of different modal data by using a special encoder model, wherein for radar track data, radar track embedding is obtained by time position embedding and latitude and longitude embedding, and then a radar special transformer encoder is used to encode the radar track embedding to obtain radar track features; for optical-electric image data, optical-electric infrared special ResNet network and optical-electric visible light special ResNet network are used for encoding to obtain optical-electric infrared features and optical-electric visible light features, which are collectively referred to as optical-electric pixel features; then, the optical-electric pixel features are aligned to the radar track features by using a cross-heterogeneous modal cross-attention mechanism to obtain optical-electric features after spatiotemporal alignment of optical-electric-radar;

[0008] Step 3, dynamically select according to the characteristics of radar track, photoelectric infrared and photoelectric visible light respectively through the gated routing dynamic selection mechanism, add missing masks selectively through gated selection, and obtain the gated routing selection result composed of missing masks;

[0009] Step 4, fuse the radar track features, photoelectric infrared features and photoelectric visible light features to obtain the fusion feature vector; input the gated routing selection result and the fusion feature vector into the heterogeneous topology adaptive fusion model composed of the transformer encoder to obtain the modality recognition result after gated routing selection, realize the recognition of any input combination of single modality to multi-modality, and obtain the final multi-modality fusion recognition result.

[0010] Further, the time position embedding specific process in step 2 is:

[0011] Suppose is a certain timestamp in the input timestamp sequence, use to represent the index of the vector, to distinguish between odd and even positions, to represent the maximum length of the preset sequence, and the time-related encoding is represented as:

[0012] ;

[0013] Distinguish between odd and even positions by sine and cosine, and the time encoding of the timestamp is represented as a vector of dimension:

[0014] ;

[0015] Therefore, the encoding of all timestamps of a radar track is represented as:

[0016] .

[0017] Further, the latitude and longitude embedding specific process in step 2 is:

[0018] Suppose the longitude and latitude of the track are represented by and respectively, first convert the degree system to the radian system:

[0019] ;

[0020] Then convert the latitude and longitude of the track point to the ECEF coordinate system, use and to represent the WGS-84 Earth ellipsoid parameters, and the calculation formula is represented as:

[0021] ;

[0022] For three coordinate system parameters 、 and , embedding is carried out using one-dimensional convolution, the coordinate code is encoded dimension, and nonlinear processing is carried out using a LeakyReLU nonlinear activation function; using to represent the one-dimensional convolution operation of embedding, using to represent the operation of the activation function, then the embedding of longitude and latitude is represented as:

[0023] .

[0024] Further, the specific process of step 3 is as follows:

[0025] Using 、 and respectively represent the radar track feature, the photoelectric infrared feature and the photoelectric visible light feature, wherein the feature vector of all 0 is used to represent the missing of the modal data, then the fusion feature is represented as:

[0026] ;

[0027] According to the dynamic selection mechanism of the gated routing, an all-negative infinite mask is added to the modal input of 0, and the setting of the mask of any modal is as follows:

[0028] ;

[0029] Wherein, dim represents the dimension of the feature setting;

[0030] Using 、 and respectively represent the radar feature corresponding mask, the photoelectric infrared feature mask and the photoelectric visible light feature mask, then the fusion mask, that is, the gated routing selection result is represented as:

[0031] .

[0032] Further, after the missing mask is selectively added in step 3 through the gate, the missing mask can be corrected through manual intervention to obtain the gated routing selection result composed of the missing mask.

[0033] The beneficial effects of the present application are as follows:

[0034] 1. The application designs a time-space joint embedding method, which converts the time series data of the radar into high-level semantic embedding information through a time series attention model, and aligns the space-time position coding with the photoelectric image features, so that the extracted features are suitable for multi-modal fusion recognition algorithm.

[0035] 2. The method of the application designs a dynamic gating function for the input modal part missing scene, adopts an antagonistic mask simulation strategy in the training stage, forces the model to learn the robust representation of different modalities under the condition of random modal missing, and in the inference stage, according to the modal of the actual input data, the missing modal mask is selectively added through gating to automatically activate the optimal feature transmission path of the input modal.

[0036] 3. The application proposes a shared-private topology structure, the private branch designs a special encoder model for different modalities, and the shared part designs a cross-modal attention mechanism to realize the isometric hidden space mapping of multi-modal features. The architecture ensures that the model parameters are completely shared when different modalities are combined, supports any input combination from single modality to multi-modality, the model parameter quantity is fixed, and multiple sub-models do not need to be trained for different modal combinations, which significantly reduces the demand for computing resources. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is the overall flowchart of the pulse routing recognition method of the application.

[0038] Figure 2 is the processing principle diagram of the pulse routing recognition method of the application. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical scheme and advantages of the application more clear, the application will be further described in detail below.

[0040] A photoelectric-radar space-time cooperative pulse routing recognition method, as shown in Figure 1 and Figure 2 is divided into three parts, including a photoelectric-radar space-time alignment attention module (processing of steps 1 and 2), a gating routing dynamic selection module (processing of step 3) and a parameter sharing heterogeneous topology adaptive module (processing of step 4).

[0041] Specifically, the following steps are included:

[0042] Step 1, the data of different modalities are preprocessed respectively; wherein for the radar track data, fixed length radar track points are obtained by truncation or interval sampling method, if the radar track points are less than the set value, the fixed length is processed by 0 filling; for the photoelectric image data, the samples are cropped and scaled, so that the target is located in the middle of the image, and the surrounding environment range is cropped, reducing the environmental interference.

[0043] Step 2, feature extraction is performed on data of different modalities using dedicated encoder models respectively, wherein for radar track data, radar track embedding is obtained through time position embedding and latitude and longitude embedding, and then a radar dedicated transformer encoder is used to encode the radar track embedding to obtain radar track features; for photoelectric image data, photoelectric infrared and visible light features are obtained respectively by encoding photoelectric infrared and visible light using photoelectric infrared and visible light dedicated ResNet networks, collectively referred to as photoelectric pixel features; then cross-heterogeneous modal cross-attention mechanism is used to realize alignment of photoelectric pixel features to radar track features, to obtain photoelectric-radar spatio-temporal aligned photoelectric feature vectors; a special time position encoding is designed, which is a vector about specific time information dimension, only related to time stamp interval, and independent of its position in the sequence.

[0044] wherein the time position embedding process is as follows:

[0045] Suppose is a certain time stamp in the input time stamp sequence, if is used to represent the index of the vector, is used to distinguish between odd and even positions, represents the maximum length of the preset sequence, so the time-related encoding can be represented as:

[0046] ;

[0047] The sine and cosine are used to distinguish between odd and even positions, so the time encoding of the time stamp can be represented as a vector of dimension:

[0048] ;

[0049] Therefore, the encoding of all time stamps of a track can be represented as:

[0050] .

[0051] wherein the latitude and longitude embedding process is as follows:

[0052] Suppose the longitude and latitude of the track are represented by and respectively, first convert the degree system to the radian system:

[0053] ;

[0054] Then convert the latitude and longitude of the track point to the ECEF coordinate system using and The WGS-84 Earth ellipsoid parameters are represented, and the calculation formula is represented as:

[0055] ;

[0056] For three coordinate system parameters , and , one-dimensional convolution is used for embedding, the coordinate code is embedded in the dimension, and the LeakyReLU nonlinear activation function is used for nonlinear processing; using to represent the one-dimensional convolution operation of embedding, using to represent the operation of the activation function, then the embedding of the longitude and latitude is represented as:

[0057] .

[0058] Step 3, through the dynamic selection mechanism of the gate routing, respectively according to the selection of the radar track feature, the photoelectric infrared feature and the photoelectric visible light feature, through the gate selection, the missing mask is selectively added, so as to automatically activate the optimal feature transmission path of the input mode, generate the radar mask and the photoelectric mask, and the missing mask can also be corrected through manual intervention, and the gate routing selection result composed of the missing mask is obtained.

[0059] Using , and to represent the radar track feature, the photoelectric infrared feature and the photoelectric visible light feature respectively, wherein the feature vector of all 0 is used to represent the missing of the mode data, and the fusion feature is represented as:

[0060] ;

[0061] According to the dynamic selection mechanism of the gate routing, a mask of all negative infinity is added to the input of 0 of the mode, and the setting of the mask of any mode is represented as follows:

[0062] ;

[0063] Wherein, dim represents the dimension of the feature setting;

[0064] Using , and to represent the radar feature mask, the photoelectric infrared feature mask and the photoelectric visible light feature mask respectively, then the fusion mask, that is, the gate routing selection result is represented as:

[0065] .

[0066] Step 4, fuse the radar track features, photoelectric infrared features and photoelectric visible light features to obtain a fused feature vector; input the gating routing selection result and the fused feature vector into a heterogeneous topology adaptive fusion model composed of a transformer encoder to obtain a modal recognition result after gating routing selection, realize recognition of any input combination of single modal to multi-modal, and obtain a final multi-modal fusion recognition result.

[0067] For the parameter-shared heterogeneous topology adaptive module, the private branch designs a special encoder model for different modalities to extract features of different modal data, and the shared part designs a cross-modal attention mechanism to realize equal-dimensional hidden space mapping of multi-modal features for identifying the fused features. The shared part ensures complete sharing of model parameters when different modal combinations are input through the topological structure consistency constraint and the dynamic mask generated by the gating routing, supports any input combination of single modal to multi-modal, the model parameter quantity is fixed, and multiple sub-models do not need to be trained for different modal combinations. The final model output is the recognized class, and the cross entropy loss between this class and the real class is calculated in the training process, so as to continuously optimize the model through back propagation.

[0068] In summary, the present application adopts a photoelectric-radar spatiotemporal coordination pulse routing recognition method, and the present application method can be used to solve the multi-modal fusion recognition technology to provide support and basis.

Claims

1.A method for photoelectric-radar spatiotemporal coordination pulse routing identification, characterized in that, Comprising the following steps: Step 1, pre-process the data of different modalities respectively; wherein for radar track data, fixed-length radar track points are obtained by truncation or interval sampling method, and if the radar track points are less than the set value, 0 is added to process it into a fixed length; for photoelectric image data, sample cropping and scaling are performed to make the target located in the center of the image and crop the surrounding environment range; Step 2, respectively use the special encoder model to extract features for data of different modalities, wherein for radar track data, radar track embedding is obtained by time position embedding and latitude and longitude embedding, and then radar track features are obtained by using radar special transformer encoder to encode the radar track embedding; for photoelectric image data, photoelectric infrared feature and photoelectric visible light feature are obtained by photoelectric infrared special ResNet network and photoelectric visible light special ResNet network, respectively, which are collectively referred to as photoelectric pixel feature; Then, the photoelectric pixel feature is aligned to the radar track feature through the cross-heterogeneous modal cross-attention mechanism to obtain the photoelectric-radar spatio-temporal aligned photoelectric feature vector; Step 3, the radar track feature, the photoelectric infrared feature and the photoelectric visible light feature are selected according to the situation through the gate routing dynamic selection mechanism, and the missing mask is selectively added through the gate selection to obtain the gate routing selection result composed of the missing mask; Step 4, fuse the radar track feature, the photoelectric infrared feature and the photoelectric visible light feature to obtain a fusion feature vector; input the gate routing selection result and the fusion feature vector into the heterogeneous topology adaptive fusion model composed of the transformer encoder to obtain the modality recognition result after the gate routing selection, realize the recognition of any input combination of single modality to multi-modality, and obtain the final multi-modality fusion recognition result; The specific process of step 3 is: Using , and represent radar track features, electro-optical infrared features and electro-optical visible features, respectively, where missing modal data is represented using an all-zero feature vector, then the fused feature is represented as: According to the gate routing dynamic selection mechanism, a mask of all negative infinity is added to the modality input of 0, and the setting of the mask of any modality is as follows: Wherein, dim represents the dimension of the feature setting; Using , and represent the radar feature corresponding mask, the electro-optical infrared feature mask and the electro-optical visible light feature mask respectively, then the fusion mask, i.e. the gating routing selection result, is represented as: ； After selectively adding the missing mask through the gate selection, the missing mask can be corrected by manual intervention to obtain the gate routing selection result composed of the missing mask. 2.The photoelectric-radar spatio-temporal coordinated pulse routing identification method according to claim 1, characterized in that, The specific process of time position embedding in step 2 is: Let us assume is a certain timestamp in the input timestamp sequence, with denoting the index of the vector, used to distinguish between odd and even positions, denoting the maximum length of the preset sequence, then the time-dependent encoding is given by: By distinguishing between odd and even positions by sine and cosine, the time encoding of the timestamp is represented as a vector of dimension Thus for the encoding of all time stamps of a radar track is represented as: 。 3. The photoelectric-radar spatiotemporal coordinated pulse routing identification method according to claim 2, characterized in that, The specific process of latitude and longitude embedding in step 2 is: Assume that the longitude and latitude of a track are represented using and degrees, first convert to radians: Then the latitude and longitude of the track points are converted to ECEF coordinate system, using and represent the WGS-84 Earth ellipsoid parameters, and the calculation formula is represented as: For three coordinate system parameters , and , one-dimensional convolution is used for embedding, the coordinate encoding dimension, and non-linear processing is performed using a LeakyReLU non-linear activation function; using to represent the one-dimensional convolution operation of embedding, using to represent the operation of the activation function, then the embedding of longitude and latitude is represented as: 。

Images