Moving target identification method

Abstract

The invention relates to a moving target identification method, which comprises the following steps of: respectively carrying out feature extraction on acquired multi-modal data to obtain a first modal feature, a second modal feature and a third modal feature; fusing the second modal feature and the third modal feature with frequency characteristics into a fused modal feature through low-rank multi-modal fusion; performing time extraction and position embedding on the first modal feature and the fusion modal feature in sequence to obtain a first processing feature and a second processing feature; cross-modal interaction of the first processing feature and the second processing feature is established by using a cross-modal learning model, a third processing feature and a fourth processing feature are output, and the cross-modal learning model is constructed based on an improved multi-modal Transform model; and performing prediction according to the third processing feature and the fourth processing feature to obtain a target recognition result. According to the invention, the accuracy of moving target identification can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of target recognition, and in particular to a moving target recognition method. Background Art

[0002] The recognition of moving targets in outdoor scenes needs to take into account the particularity of the environment: interference such as image field of view occlusion, wind noise or animals passing by causes the single signal recognition method to have poor accuracy and robustness, so it is necessary to use different types of signals and data fusion methods to complete the recognition task. In practical applications, composite sensor nodes are used to collect three types of signals: image, sound and vibration, and then the signal features are extracted for classification to avoid the low recognition accuracy caused by a single type of signal.

[0003] The existing field moving target recognition technology has the following disadvantages: (1) Since the acoustic shock signal can be collected synchronously and continuously while the image signal is collected by interval shooting, the information of the image signal and the acoustic shock signal is not equal. Therefore, the image mode is not selected for signal fusion. However, the image information is rich and the recognition accuracy is high. The lack of image mode has a great loss on the recognition performance; (2) The moving target has a long movement time in the node monitoring area. The existing technology generally intercepts the target signal in a short time for recognition, without considering the complete signal characteristics and signal changes of the target signal over a long period of time, which may cause recognition errors; (3) The fusion method of different modes is relatively simple, and there is no mutual learning of implicit information, resulting in limited recognition accuracy. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a moving target recognition method, which can effectively improve the accuracy of moving target recognition.

[0005] The technical solution adopted by the present invention to solve the technical problem is: to provide a moving target recognition method, comprising the following steps:

[0006] Performing feature extraction on the collected multimodal data respectively to obtain first modal features, second modal features and third modal features;

[0007] The second modal feature and the third modal feature having frequency characteristics are fused into a fused modal feature through low-rank multimodal fusion;

[0008] Performing time extraction and position embedding on the first modal feature in sequence to obtain a first processed feature;

[0009] Sequentially performing time extraction and position embedding on the fused modal features to obtain second processing features;

[0010] Establishing a cross-modal interaction between the first processing feature and the second processing feature by using a cross-modal learning model, and outputting a third processing feature and a fourth processing feature, wherein the cross-modal learning model is constructed based on an improved multimodal Transformer model;

[0011] A moving target recognition result is obtained by predicting the third processing feature and the fourth processing feature.

[0012] Further, the using a cross-modal learning model to establish a cross-modal interaction between the first processing feature and the second processing feature, and outputting a third processing feature and a fourth processing feature, includes:

[0013] The first processing feature receives information of the second processing feature through a first cross-modal learning model to obtain the third processing feature;

[0014] The second processing feature receives information of the first processing feature through a second cross-modal learning model to obtain the fourth processing feature.

[0015] Further, the predicting and obtaining the target recognition result according to the third processing feature and the fourth processing feature includes:

[0016] The third processing feature and the fourth processing feature are processed by the self-attention transformer respectively to output a fifth processing feature and a sixth processing feature;

[0017] The moving target recognition result is obtained by performing weight assignment on the fifth processing feature and the sixth processing feature using a fully connected layer.

[0018] Furthermore, the first cross-modal learning model and the second cross-modal learning model are respectively one of a first improved multimodal Transformer model, a second improved multimodal Transformer model, and a third improved multimodal Transformer model.

[0019] Furthermore, the first improved multimodal Transformer model includes:

[0020] A first FFN module, including layer normalization, a position feed-forward layer and a residual connection, is used to enhance the first input feature to obtain a first intermediate feature;

[0021] A first cross-modal attention module, including layer normalization, a multi-head attention mechanism and a residual connection, is used to obtain a second intermediate feature by fusing information of the second input feature with the first intermediate feature;

[0022] The second FFN module includes layer normalization, a position feedforward layer and a residual connection, and is used to perform feature enhancement on the second intermediate feature and then output the first input feature that receives information of the second feature.

[0023] Furthermore, the second improved multimodal Transformer model includes:

[0024] The second cross-modal attention module includes root mean square layer normalization, multi-head attention mechanism and residual connection, which is used to fuse the first input feature with the information of the second input feature to obtain the third intermediate feature;

[0025] The third FFN module includes a root mean square layer normalization, a position feedforward layer and a residual connection, and is used to perform feature enhancement on the third intermediate feature and then output the first input feature that receives the information of the second feature.

[0026] Furthermore, the third improved multimodal Transformer model includes:

[0027] The third cross-modal attention module includes root mean square layer normalization, multi-head attention mechanism and gating layer, which is used to fuse the first input feature with the information of the second input feature to obtain the fourth intermediate feature;

[0028] The fourth FFN module includes a root mean square layer normalization, a position feedforward layer and a gating layer, and is used to perform feature enhancement on the fourth intermediate feature and then output the first input feature that receives the information of the second feature.

[0029] Furthermore, the first modal feature is obtained by the following method:

[0030] The target detection algorithm is used to extract the area containing the target to be detected in each collected image to obtain multiple target area images;

[0031] The target area image is serialized to obtain the first modality feature.

[0032] Furthermore, the second modal feature is obtained by the following method:

[0033] Dividing the collected sound signal into a plurality of sound data frames of set length;

[0034] Mel-frequency cepstral coefficients of a set dimension of each of the sound data frames are extracted to obtain the second modal feature.

[0035] Furthermore, the third modal feature is obtained by the following method:

[0036] Dividing the collected vibration signal into a plurality of vibration data segments of set length;

[0037] The time domain features of the first set dimension and the frequency domain features of the second set dimension of each vibration data segment are extracted, and the time domain features and the frequency domain features are concatenated to obtain the third modal features.

[0038] Beneficial Effects

[0039] Due to the adoption of the above technical solution, the present invention has the following advantages and positive effects compared with the prior art:

[0040] (1) The present invention solves the fusion problem by building a multimodal fusion framework and introducing a Transformer structure, thereby solving the problem that the acoustic shock signal data sequence is relatively long and the image modality and acoustic shock modality information are not equal;

[0041] (2) By introducing the cross-modal Transformer, the present invention can learn from any two modal information with different time dimensions to obtain enhanced information representation, thereby obtaining more accurate detection results;

[0042] (3) The present invention introduces a low-rank multimodal fusion (LMF) method to fuse the features of different modalities, and obtains a fused modality as an input for modal mutual learning; the acoustic shock signal is two modalities with strong correlation. The low-rank multimodal fusion (LMF) method can effectively combine the features of the two modalities to obtain a new fused modality, which not only reduces the computational complexity, but also enables the subsequent cross-modal learning (Crossmodal Transformer) to be carried out between the image modality and an enhanced fusion modality, which is conducive to mutual learning and fusion between different modalities;

[0043] (4) The present invention divides the signal features of three different modalities into two paths: image features and acoustic shock fusion features, and extracts information, performs cross-modal learning and self-attention Transformer processing respectively. The two paths of features learn from each other's implicit information and then predict the recognition results, thereby improving the accuracy of target recognition.

[0044] (5) The present invention improves the network structure of the Crossmodal Transformer module and introduces three improved structures: a. A Position-wise Feed Forward module is added before the Multi-head module to reduce the local truncation error; b. Root mean square layer normalization (RMSLN) is used instead of layer normalization (LayerNorm) to enable the model to have implicit learning rate adaptability; c. A gating layer is used instead of the standard residual connection, the layer normalization is reordered, and a Markov training mechanism is used. The above structures can improve the accuracy of target recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a schematic diagram of a flow chart of an embodiment of the present invention;

[0046] Figure 2 is a schematic diagram of an LMF model according to an embodiment of the present invention;

[0047] Figure 3 is a schematic diagram of a Crossmodal attention module according to an embodiment of the present invention;

[0048] Figure 4 is a schematic diagram of a Crossmodal Transformer model according to an embodiment of the present invention;

[0049] Figure 5 Schematic diagram of a first improved Crossmodal Transformer model according to an embodiment of the present invention;

[0050] Figure 6 is a schematic diagram of a second improved Crossmodal Transformer model according to an embodiment of the present invention;

[0051] Figure 7 It is a schematic diagram of a third improved Crossmodal Transformer model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0053] The embodiment of the present invention relates to a method for identifying outdoor moving targets based on multimodal feature fusion, such as Figure 1 As shown in the figure, the input signals are three types of signals: image, sound and vibration. The overall solution mainly includes six stages: feature extraction, modal fusion, information extraction, cross-modal learning, self-attention mechanism and result prediction, and finally outputs the target recognition result. Among them, the acoustic shock signal is a synchronized long-term data sequence, and the image signal is a number of frames of pictures in the same time. The technical solutions for each stage are as follows:

[0054] 1. Feature extraction

[0055] For a frame of image signal, the yolov5 algorithm is used to detect the target in the image and intercept the target area, and the target area is converted into an 8-bit color image and serialized as an image signal feature matrix; for a sound signal, it is divided into multiple frames of 0.5 seconds per frame, and 23-dimensional Mel Frequency Cepstral Coefficients (MFCC) are extracted from each frame to form a two-dimensional sound signal feature matrix; for a vibration signal, it is divided into multiple signals of 5 seconds per segment, and 16-dimensional time domain features and 4-dimensional frequency domain features are extracted from each segment, which are spliced ​​into 20-dimensional time-frequency features to form a two-dimensional vibration signal feature matrix. The feature matrices of the above three signals are used as the input of the next stage of multimodal fusion.

[0056] 2. Modal Fusion

[0057] In the modal fusion stage, the low-rank multimodal fusion (LMF) method is introduced to fuse the sound and vibration modal features to form the acoustic-vibration fusion mode F.

[0058] like Figure 2 As shown, each mode is represented by z. In order to use a tensor to simulate the interaction between any subset of modes, before performing the outer product, a 1 is added to the end of each mode representation z to obtain z. m . So the input tensor Z is calculated by the representation of a single mode: z m is the input representation with 1 appended.

[0059] Input Tensor Passing a linear layer g(·) produces a vector representation:

[0060]

[0061] Where W is the weight and b is the bias. The LMF method considers W as d h M-order tensors, each of which can be expressed as k=1,...,d h , there is a pattern that decomposes exactly into vectors:

[0062]

[0063] The smallest R that makes the decomposition valid is called the rank of the tensor.

[0064] Collection of vectors It is called the rank R decomposition factor of the original tensor, fix R to r, and then use r decomposition factors To reconstruct a low-rank version of

[0065] These vectors can be reassembled into M modality-specific low-rank factors. Let Then the low-rank factor corresponding to mode m is

[0066] Then the low-rank weight tensor can be reconstructed using the following formula:

[0067]

[0068] Based on the decomposition of W, The fusion representation h can be obtained as follows:

[0069]

[0070] in Represented as the element-wise product of a series of tensors.

[0071] Therefore, the fusion of the two modes of acoustic shock is expressed as:

[0072]

[0073] The above-mentioned acoustic shock fusion modality and image modality serve as input for the next stage of information extraction.

[0074] (III) Information Extraction

[0075] Information extraction includes two steps: temporal convolution and position embedding. First, the input sequence passes through a 1D temporal convolution layer:

[0076]

[0077] Among them k {V,F} is the size of the convolution kernel of the modality {V, F}, and d is a common dimension that maps the feature dimensions of different modalities to d, making crossmodal attention dot-products available. The sequence obtained by convolution contains local information of the sequence, ensuring that each element in the input sequence has sufficient knowledge of its neighboring elements.

[0078] In order to ensure that the sequence carries time information, the representation containing time information Add position embedding (PE):

[0079]

[0080] in Calculate the embedding for each position index as follows:

[0081]

[0082] where i = 1, ..., T and It is the feature representation of different modalities containing low-level position information, which serves as the input for the next stage of cross-modal learning.

[0083] 4. Cross-modal learning

[0084] Cross-modal learning introduces an improved Crossmodal Transformer method to enable mutual learning between the image modality and the acoustic shock fusion modality. The core module of the Crossmodal Transformer, Crossmodal attention, is a way to fuse cross-modal information and provide latent adaptation across modalities, such as Figure 3 from mode β to mode α in .

[0085] Depend on Figure 3 It can be seen that X α and X β Is the input sequence, multiplied by the weights respectively to get: The latent adaptation from β to α is represented as the cross-modal attention Y α :

[0086]

[0087] Based on the Crossmodal attention module, the Crossmodal Transformer module can be designed to enable one modality to receive information from another modality. The standard Crossmodal Transformer model mainly consists of two layers: Multi-head (Crossmodal attention) and Position-wise Feed-forward, such as Figure 4 shown.

[0088] The present invention proposes three improved Crossmodal Transformer models.

[0089] (1) The first improved Crossmodal Transformer model is as follows Figure 5 As shown in the figure, the model structure is changed to three layers: Position-wise Feed-forward, Multi-head (Crossmodal attention) and Position-wise Feed-forward. The position feed-forward is divided into two levels, front and back, which can effectively reduce the local truncation error and improve the cross-modal learning performance.

[0090] Taking modality 2 to modality 1 as an example: Each Crossmodal Transformer consists of D layers of cross-modal attention blocks, and the Crossmodal Transformer calculates the 1:D layer as follows:

[0091]

[0092] Where f(·) represents the parameterized position-wise feed-forward layer, LN(·) represents layer normalization, and CM 2→1 Indicates that modality 1 receives information from modality 2 through crossmodal attention.

[0093] (2) The second improved Crossmodal Transformer model is as follows Figure 6 As shown in the figure, root mean square layer normalization (RMSLN) is used instead of layer normalization (LayerNorm), and the input is normalized according to the root mean square (RMS) statistic, so that the model has rescaling invariance and implicit learning rate adaptability, thereby improving cross-modal learning performance.

[0094] Given an input vector The feedforward network projects it into an output vector through a linear transformation As shown below:

[0095]

[0096] Among them, w i is the weight vector of the ith output neuron, b i is a bias scalar, f(·) is a nonlinear function, Represents the weighted sum of the neuron input, which is also the goal of normalization. Root mean square layer normalization (RMSLN) is as follows:

[0097]

[0098] in is a vector The ith value of , as a normalized replacement for the layer activation, is the gain parameter.

[0099] (3) The third improved Crossmodal Transformer model is as follows Figure 7 As shown, a gating layer is used instead of a standard residual connection to normalize and reorder the layers, and a Markov training mechanism is used.

[0100] Taking mode 2 to mode 1 as an example: Crossmodal Transformer calculates the 1:D layer as follows:

[0101]

[0102] Among them, g(·) represents the gating layer, f(·) represents the parameterized position-wise feed-forward layer, LN(·) represents layer normalization, CM 2→1 Indicates that modality 1 receives information from modality 2 through crossmodal attention.

[0103] Using the above three improved cross-modal Transformer models, cross-modal interaction is established between the image modality and the acoustic shock fusion modality as the input of the next stage.

[0104] 5. Self-Attention Mechanism

[0105] Each modality learns the other two modalities to obtain two information representations. The image modality is obtained by CrossmodalTransformer Z V The Self-Attention Transformer collects temporal information for prediction. The network structure of the Self-Attention Transformer is similar to that of the Crossmodal Transformer, but it does not need to learn other modalities, so there is no input from another modality.

[0106] (VI) Result prediction

[0107] Two modal information representation Z V ,Z F After the Self-Attention Transformer, the output result is obtained, and the output result is weighted by the fully connected layer to obtain the output, that is, the target recognition result.

[0108] To demonstrate the beneficial effects of the present invention, algorithm experiments were conducted based on Python language, PyTorch framework and collected outdoor moving target signals. (1) The method of the present invention (LMF-ImprovedMulT) was compared with different multimodal fusion methods, (2) different fusion structures were compared under the method of the present invention, and (3) different Crossmodal Transformer structures were compared under the method of the present invention and the determined fusion structure. The algorithm experiments can prove that the multimodal feature fusion method (LMF-ImprovedMulT) proposed in the present invention is superior to the existing multimodal fusion method and other structures of the method in target recognition performance, and achieves a recognition accuracy of 95.71% in the vehicle moving target experiment.

Claims

1. A moving target recognition method, characterized in that: The following steps are involved: Performing feature extraction on the collected multimodal data respectively to obtain first modal features, second modal features and third modal features; The second modal feature and the third modal feature having frequency characteristics are fused into a fused modal feature through low-rank multimodal fusion; Performing time extraction and position embedding on the first modal feature in sequence to obtain a first processed feature; Sequentially performing time extraction and position embedding on the fused modal features to obtain second processing features; Establishing a cross-modal interaction between the first processing feature and the second processing feature by using a cross-modal learning model, and outputting a third processing feature and a fourth processing feature, wherein the cross-modal learning model is constructed based on an improved multimodal Transformer model; A moving target recognition result is obtained by predicting the third processing feature and the fourth processing feature.

2. The method according to claim 1, characterized in that The using a cross-modal learning model to establish a cross-modal interaction between the first processing feature and the second processing feature, and outputting a third processing feature and a fourth processing feature, includes: The first processing feature receives information of the second processing feature through a first cross-modal learning model to obtain the third processing feature; The second processing feature receives information of the first processing feature through a second cross-modal learning model to obtain the fourth processing feature.

3. The method according to claim 2, characterized in that The predicting and obtaining the target recognition result according to the third processing feature and the fourth processing feature includes: The third processing feature and the fourth processing feature are processed by the self-attention transformer respectively to output a fifth processing feature and a sixth processing feature; The moving target recognition result is obtained by performing weight assignment on the fifth processing feature and the sixth processing feature using a fully connected layer.

4. The method according to claim 2, characterized in that: The first cross-modal learning model and the second cross-modal learning model are respectively one of a first improved multimodal Transformer model, a second improved multimodal Transformer model, and a third improved multimodal Transformer model.

5. The method according to claim 4, characterized in that The first improved multimodal Transformer model includes: a first FFN module, including layer normalization, a position feedforward layer and a residual connection, used to perform feature enhancement on the first input feature to obtain a first intermediate feature; A first cross-modal attention module, including layer normalization, a multi-head attention mechanism and a residual connection, is used to obtain a second intermediate feature by fusing information of the second input feature with the first intermediate feature; The second FFN module includes layer normalization, a position feedforward layer and a residual connection, and is used to perform feature enhancement on the second intermediate feature and then output the first input feature that receives information of the second feature.

6. The method according to claim 4, characterized in that The second improved multimodal Transformer model includes: a second cross-modal attention module, including root mean square layer normalization, a multi-head attention mechanism and a residual connection, which is used to fuse the first input feature with the information of the second input feature to obtain a third intermediate feature; The third FFN module includes a root mean square layer normalization, a position feedforward layer and a residual connection, and is used to perform feature enhancement on the third intermediate feature and then output the first input feature that receives the information of the second feature.

7. The method according to claim 4, characterized in that The third improved multimodal Transformer model includes: a third cross-modal attention module, including root mean square layer normalization, a multi-head attention mechanism and a gating layer, which is used to fuse the first input feature with the information of the second input feature to obtain a fourth intermediate feature; The fourth FFN module includes a root mean square layer normalization, a position feedforward layer and a gating layer, and is used to perform feature enhancement on the fourth intermediate feature and then output the first input feature that receives the information of the second feature.

8. The method according to claim 1, characterized in that The first modal feature is obtained by the following method: The target detection algorithm is used to extract the area containing the target to be detected in each collected image to obtain multiple target area images; The target area image is serialized to obtain the first modality feature.

9. The method according to claim 1, characterized in that: The second modal feature is obtained by the following method: Dividing the collected sound signal into a plurality of sound data frames of set length; Mel-frequency cepstral coefficients of a set dimension of each of the sound data frames are extracted to obtain the second modal feature.

10. The method according to claim 1, characterized in that The third modal feature is obtained by the following method: Dividing the collected vibration signal into a plurality of vibration data segments of set length; The time domain features of the first set dimension and the frequency domain features of the second set dimension of each vibration data segment are extracted, and the time domain features and the frequency domain features are concatenated to obtain the third modal features.

Images