A method and system for object detection based on feature alignment

Abstract

The present invention provides a target detection method and system based on feature alignment for unaligned visible light and infrared image pairs. The present invention uses the infrared modality as the reference modality, the visible light modality as the offset modality, fixes the reference modality, and only operates on the offset modality throughout the process. The present invention adopts a coarse-to-fine feature alignment strategy, predicts and corrects the offset through a feature offset prediction module as a preliminary alignment means, and performs secondary regression on each candidate frame through a region of interest alignment module to further correct the offset between modal features. In addition, an illumination-guided feature weighting module is introduced to re-weight the region of interest features and adaptively adjust the contributions of different modalities. The method can perform feature alignment on two modalities with large deviations while using only one modality annotation, effectively improving the accuracy of visible light and infrared image fusion target detection, and showing stronger robustness for scenes with spatial offsets between modalities.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a target detection method and system based on feature alignment. Background Art

[0002] Target detection is the basis for automatic analysis and understanding of complex scenes, and plays an important role in intelligent security, human-computer interaction, smart cities, etc. Infrared images and visible light images have good complementary characteristics. By mining the complementary information of the two, the target detection accuracy of visible light and infrared image fusion can be improved.

[0003] However, directly fusing features of images from two modalities often requires aligning the image pairs, because the spatial offset between the modalities will degrade the detector performance. This is because, on the one hand, the input images of different modalities do not match in corresponding positions, and the features to be fused also have spatial offsets. The meaning of the features on the feature map is unclear during fusion, which makes it difficult for the network to infer and leads to unstable target classification and positioning. On the other hand, the shared annotations of the two modalities will introduce annotation bias, and the network cannot distinguish which modality the annotation information belongs to, making it difficult to regress the position of the detection box.

[0004] See the instruction manual Figure 1 , Figure 1 The figure shows the positional offset phenomenon in the unregistered visible light-thermal infrared image pair. The solid frame represents the location of the annotation in the infrared image, and the dotted frame represents the location of the target in the visible light image. It can be found that although the sensors of the two modes capture images of the same scene, the images are not registered, so the frames at the same position cannot cover the same target, resulting in spatial misalignment, and even causing the target to be framed in one mode and the background to be framed in the other mode.

[0005] Most previous works require accurate annotation information of both modalities to supervise the network strongly, or migrate information between modalities without directly using it, resulting in information loss. For data that captures the same scene but only has single-modal annotations, re-labeling the other modality will consume a lot of manpower, material resources and time resources, but migrating modal information will result in information waste.

[0006] Therefore, a fast and convenient target detection algorithm based on visible light and infrared image fusion based on unregistered image pairs is urgently needed to improve the accuracy of target detection based on visible light and infrared image fusion. Summary of the invention

[0007] In view of the deficiencies in the prior art, the purpose of the present invention is to provide a feature alignment target detection method for unregistered visible light infrared image pairs. The method adopts a coarse-to-fine alignment strategy and can spatially align the features of two modalities using only one modality annotation to improve the accuracy of target detection by fusion of visible light and infrared images.

[0008] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0009] The present invention provides a target detection method based on feature alignment, comprising the following steps:

[0010] Step 1: Input the offset modality image and the reference modality image into two deep convolutional neural networks with the same structure to extract image features. The two networks do not share parameters.

[0011] Step 2: Input the extracted offset modal feature map and the reference modal feature map into the feature offset prediction module, predict the offset of each feature position of the offset modal feature map, and correct the feature position to preliminarily align the feature maps of the two modalities;

[0012] Step 3: Concatenate the corrected offset modal feature map and the reference modal feature map and input them into the region generation network to obtain the candidate box;

[0013] Step 4: input the candidate box into the region of interest alignment module, perform secondary regression on the candidate box of the offset modality, and obtain the corrected candidate box of the offset modality and the corrected region of interest features of the offset modality;

[0014] Step 5: Downsample the offset modality image and input it into the illumination perception network to predict the weights of the two modality features;

[0015] Step 6: Use the modal weights obtained in step 5 to re-weight the features of the regions of interest of the two modalities obtained in step 4, and send them to the detection head after cascading to obtain the location information of the target of interest, completing the training of the target detection model;

[0016] Step 7: Input the image to be tested into the trained target detection model to obtain the target detection result.

[0017] Preferably, step 2 specifically includes:

[0018] Step 21: Concatenate the feature map channels of the two modalities F C =CONCAT(F O ,F R ), CONCAT(·) represents channel concatenation;

[0019] Step 22: Extract features through the multi-head attention module and the reverse residual feedforward network, and predict the x and y offsets of each feature position on the feature map, the offset Among them, MHSA(·) stands for multi-head self-attention, IRFFN(·) stands for inverse residual feedforward network, represents a 1×1 convolutional layer, σ(·) represents the tanh activation function, and the threshold thr is used. W ,thr H As a hyperparameter, the offset of each feature in the x and y directions is ΔX = p x ·thr W ,ΔY=p y ·thr H ;

[0020] Step 23: The corrected feature position of the offset modal feature map is X'=X⊕ΔX, Y'=Y⊕ΔY. The corrected offset modal feature map (X',Y':V)←(X,Y:V) is obtained by transforming the feature value, where ⊕ represents element-by-element addition, (X',Y′:V)←(X,Y:V) means moving the value V of the feature point with coordinates X,Y to the new coordinates X',Y', and ← represents assignment to achieve correction of the feature position.

[0021] Preferably, step 2 further includes pre-training the feature offset prediction module, specifically including:

[0022] Each pixel of the offset modality feature map is randomly offset in the x-direction and the y-direction, and the offset image is downsampled as the true value of the feature map offset;

[0023] The loss function for pre-training the feature offset prediction module is Among them, p ,O g Represent the predicted value and true value of the offset respectively, i, j represent the row number and column number of the feature point, H, W represent the height and width of the offset modal feature map respectively, and n=HW represents the number of elements in the offset modal feature map.

[0024] Preferably, step 4 specifically includes:

[0025] Step 41: First, expand the context of the region of interest for the candidate boxes of the two modalities to obtain and

[0026] Step 42: Pool the expanded region of interest features to obtain the region of interest features of the offset mode and the region of interest characteristics of the reference modality Where RoIPooling(·) represents the region of interest pooling, FO and F R are the extracted offset modality image features and reference modality image features respectively;

[0027] Step 43: F ORoI ,F RRoI Perform channel cascading to predict 4 parameters for each offset modal candidate box Among them, c x 、c y Represents the change of the center point coordinates, c w 、c h The change in height and width, represents the candidate box regression layer, σ(·) represents the tanh activation function, and CONCAT(·) represents the channel cascade.

[0028] Step 44: Use 4 hyperparameters to constrain the change amount, then the corrected candidate box The coordinates are:

[0029]

[0030]

[0031]

[0032]

[0033] Hyperparameter thr x ,thr y ,thr w ,thr h Set to w, h, w / 2, h / 2 respectively, where w and h represent the width and height of the current candidate box, x o ,y o ,w o ,h o Represents the center coordinates, width and height of the initial candidate box.

[0034] Preferably, step 4 also includes a training step:

[0035] Use the corrected offset modal candidate box obtained in step 4 Calculate the corrected offset modal region of interest features Where RoIPooling(·) represents region of interest pooling;

[0036] Region of interest features for the modified offset mode And the reference modality region of interest feature F obtained in step 4 RRoI , calculate the channel average to get the response diagram of the two modes MRRoI The points with the largest response are P R =(x R ,y R )=argmax(M RRoI ), calculate the Euclidean distance between the two and use it as the loss function: This is done to shorten the distance between the maximum response points of the features in the regions of interest of the two modalities and guide network training.

[0037] The present invention also provides a target detection system based on feature alignment, comprising:

[0038] Extraction module: The offset modality image and the reference modality image are respectively input into two deep convolutional neural networks with the same structure to extract image features. The two networks do not share parameters.

[0039] Feature offset prediction module: input the extracted offset modal feature map and the reference modal feature map into the feature offset prediction module, predict the offset of each feature position of the offset modal feature map, and correct the feature position to preliminarily align the feature maps of the two modalities;

[0040] Region generation network: The corrected offset modal feature map is cascaded with the reference modal feature map and then input into the region generation network to obtain the candidate box;

[0041] Region of Interest Alignment Module: Input the candidate frame into the region of interest alignment module, perform secondary regression on the candidate frame of the offset modality, and obtain a corrected candidate frame of the offset modality and corrected region of interest features of the offset modality;

[0042] Feature weighting module: downsamples the offset modality image and inputs it into the illumination perception network to predict the weights of the two modality features;

[0043] Fusion module: Use the modal weights obtained by the feature weighting module to re-weight the ROI features of the two modalities obtained by the ROI alignment module, and then send them to the detection head after cascading to obtain the location information of the target of interest, thus completing the training of the target detection model;

[0044] Target detection module: input the image to be tested into the trained target detection model to obtain the target detection result.

[0045] Preferably, the feature shift prediction module specifically includes:

[0046] Concatenate the feature map channels of the two modalities F C =CONCAT(F O ,F R ), CONCAT(·) represents channel concatenation;

[0047] The multi-head attention module and the reverse residual feed-forward network are used to extract features and predict the x and y offsets of each feature position on the feature map. Among them, MHSA(·) stands for multi-head self-attention, IRFFN(·) stands for inverse residual feedforward network, represents a 1×1 convolutional layer, σ(·) represents the tanh activation function, and the threshold thr is used. W ,thr H As a hyperparameter, the offset of each feature in the x and y directions is ΔX = p x ·thr W ,ΔY=p y ·thr H ;

[0048] The corrected feature position of the offset modal feature map is X′=X⊕ΔX, Y′=Y⊕ΔY. The corrected offset modal feature map (X',Y':V)←(X,Y:V) is obtained by transforming the position of the eigenvalues, where ⊕ represents element-by-element addition, (X',Y':V)←(X,Y:V) means moving the value V of the feature point with coordinates X,Y to the new coordinates X',Y', and ← represents assignment to achieve the correction of the feature position.

[0049] Preferably, the feature shift prediction module further includes pre-training the feature shift prediction module, specifically including:

[0050] Each pixel of the offset modality feature map is randomly offset in the x-direction and the y-direction, and the offset image is downsampled as the true value of the feature map offset;

[0051] The loss function for pre-training the feature offset prediction module is Among them, p ,O g Represent the predicted value and true value of the offset respectively, i, j represent the row number and column number of the feature point, H, W represent the height and width of the offset modal feature map respectively, and n=HW represents the number of elements in the offset modal feature map.

[0052] Preferably, the region of interest alignment module includes:

[0053] Extension module: First, expand the context of the region of interest for the candidate boxes of the two modalities to obtain and

[0054] Pooling module: The offset modality image pools the expanded region of interest features to obtain the region of interest features of the offset modality and the region of interest characteristics of the reference modality Where RoIPooling(·) represents the region of interest pooling, F O and F R are the extracted offset modality image features and reference modality image features respectively;

[0055] Prediction module: F ORoI ,F RRoI Perform channel cascading to predict 4 parameters for each offset modal candidate box Among them, c x 、c y Represents the change of the center point coordinates, c w 、c h The change in height and width, F(·) represents the candidate box regression layer, σ(·) represents the tanh activation function, and CONCAT(·) represents the channel cascade.

[0056] Correction module: 4 hyperparameters are used to constrain the change amount, and the corrected candidate box The coordinates are:

[0057]

[0058]

[0059]

[0060]

[0061] Parameter thr x ,thr y ,thr w ,thr h Set to w, h, w / 2, h / 2 respectively, where w and h represent the width and height of the current candidate box, x o ,y o ,w o ,h o Represents the center coordinates, width and height of the initial candidate box.

[0062] Preferably, the region of interest alignment module further includes:

[0063] Pooling module: using the corrected offset modal candidate box obtained in the region of interest alignment module Calculate the corrected offset modal region of interest features Where RoIPooling(·) represents region of interest pooling;

[0064] Network training module: Features of the region of interest for the corrected offset mode And the reference modality region of interest feature F RRoI, calculate the channel average to get the response diagram of the two modes M RRoI The points with the largest response are Calculate the Euclidean distance between the two and use it as the loss function: This is done to shorten the distance between the maximum response points of the features in the regions of interest of the two modalities and guide network training.

[0065] Beneficial effects of the present invention:

[0066] 1. Based on the ability of infrared images to provide stable imaging in all weather conditions, the present invention selects the visible light modality as the offset modality and the infrared modality as the reference modality. The reference modality is fixed, and the features of the two modalities can be spatially aligned when only one modality is used for annotation, thereby effectively improving the accuracy of target detection and showing stronger robustness to offset input images.

[0067] 2. The present invention adopts a coarse-to-fine alignment strategy, which can spatially align the features of two modalities when only one modality is used for annotation, so as to improve the accuracy of target detection by fusion of visible light and infrared images.

[0068] 3. The present invention uses a multi-head self-attention mechanism to predict and correct the offset of the feature map of the offset modality as a preliminary alignment method. In the region of interest pooling stage, a secondary regression is performed on each candidate box to further correct the offset between modal features, effectively improving the accuracy of target registration.

[0069] 4. The present invention uses the modal weights obtained by the illumination-guided feature weighting module to re-weight the features of the two modalities, and then sends them to the detection head for classification and regression, adaptively adjusting the contributions of different modalities and improving the accuracy of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 This is an example diagram of the spatial offset between modes;

[0071] Figure 2 A deep convolutional neural network model diagram used in the target detection method provided by the present invention;

[0072] Figure 3 It is the feature offset prediction module in the present invention;

[0073] Figure 4 It is a region of interest alignment module in the present invention;

[0074] Figure 5 It is a feature weighting module based on illumination guidance in the present invention;

[0075] Figure 6 It is the feature map visualization result of the feature offset prediction module in the present invention;

[0076] Figure 7 This is an example diagram of the detection results of the present invention. DETAILED DESCRIPTION

[0077] The present invention will be further described below in conjunction with the embodiments shown in the accompanying drawings.

[0078] The present invention provides a method and system for object detection based on feature alignment. In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings.

[0079] The present invention provides a target detection method based on feature alignment, which uses a visible light modality as an offset modality, an infrared modality as a reference modality, and a fixed reference modality.

[0080] This is because infrared images have the ability to provide stable imaging in all weather conditions. The present invention uses the visible light mode as the offset mode and the infrared mode as the reference mode, fixes the reference mode, and only operates on the offset mode throughout the process.

[0081] Embodiment 1

[0082] The present invention provides a target detection method based on feature alignment, which specifically comprises the following steps:

[0083] Step 1: Input the offset modality image and the reference modality image into two deep convolutional neural networks with the same structure to extract image features. The two networks do not share parameters.

[0084] The input offset modal image and the reference modal image are an image pair, which are a visible light image and an infrared image taken simultaneously in the same scene.

[0085] Specifically, the offset modal image I O Input to the feature extractor of the deep convolutional neural backbone network to obtain the offset modality image feature F O =G O (I O ), the reference modality image I R Input to the feature extractor of the deep convolutional neural backbone network to obtain the reference modality image feature F R =G R (I R ). Wherein, subscript O represents the offset mode and subscript R represents the reference mode.

[0086] like Figure 2 As shown in Figure 1, Faster R-CNN is used as the basic framework to scale the offset modality image and the reference modality image to 640×512 pixels and input them into two deep convolutional neural backbone networks G with exactly the same structure. O(·),G R (·) is used to extract image features, and the two networks do not share parameters. In order to enhance the generalization ability of the network, the data augmentation strategy of scaling and random horizontal flipping is adopted in training.

[0087] Preferably, the feature extractor of the deep convolutional neural backbone network uses ResNet-101 (Residual Neural Network of 101 layers) + FPN (feature pyramid Network), and the initial parameter value uses the Faster R-CNN weights pre-trained on the COCO dataset, and only the weights of the feature extraction part are used.

[0088] Step 2: Input the extracted offset modal feature map and the reference modal feature map into the feature offset prediction module, predict the offset of each feature position of the offset modal feature map, and correct the feature position to preliminarily align the feature maps of the two modalities.

[0089] See also Figure 3 , extract the offset modal feature map F O and the reference modal characteristic diagram F R Input Figure 3 The feature offset prediction module M shown fop In (·), the offsets in the x and y directions are predicted for each feature position of the offset modal feature map, and the offsets of the feature positions are corrected to preliminarily align the feature maps of the two modalities.

[0090] Preferably, step 2 specifically includes:

[0091] Step 21: Concatenate the feature map channels of the two modalities F C =CONCAT(F O ,F R ), where CONCAT(·) represents channel concatenation;

[0092] Step 22: Extract features through the multi-head attention module and the reverse residual feedforward network, and predict the x and y offsets of each feature position on the feature map. The offset is Among them, MHSA(·) stands for multi-head self-attention, IRFFN(·) stands for inverse residual feedforward network, represents a 1×1 convolutional layer, σ(·) represents the tanh activation function, and p x ,p y ∈[-1,1] represents the output of this module.

[0093] In order to constrain the offset of the feature, a threshold thr is used W ,thr HAs a hyperparameter, the offset of each feature in the x and y directions is ΔX = p x ·thr W ,ΔY=p y ·thr H .

[0094] Step 23: The corrected feature position of the offset modal feature map is X'=X⊕ΔX, Y'=Y⊕ΔY. The corrected offset modal feature map (X',Y':V)←(X,Y:V) is obtained by transforming the position of the feature value, where ⊕ represents element-by-element addition, (X',Y':V)←(X,Y:V) means moving the value V of the feature point with coordinates X,Y to the new coordinates X',Y', and ← represents assignment to achieve correction of the feature position.

[0095] Preferably, the number of self-attentions in the multi-head self-attention module is set to 8, the reverse residual feedforward network contains 3 convolutional layers, batch normalization is used after each convolutional layer to normalize the features, and the GELU activation function is used after the first two convolutional layers and residual connections are performed, and the hyperparameter thr W ,thr H Set to 1 / 10 of the feature map width and height.

[0096] Compared with ResNet, the reverse residual feedforward network has fewer parameters and can reduce the amount of network computation.

[0097] In addition, step 2 also includes pre-training the feature offset prediction module, specifically including:

[0098] Each pixel point of the offset modal feature map is randomly offset in the x direction by [-thr W ,thr W ] and random offset in the y direction [-thr H ,thr H ] pixels, and downsample the offset vector map as the true value of the feature map offset.

[0099] The offset vector map is a vector map composed of the offset vectors of each pixel, and the offset vector map is downsampled by 1 / 8 as the true value of the feature map offset.

[0100] The loss function for pre-training the feature offset prediction module is Among them, p ,O g Represent the predicted value and true value of the offset respectively, i, j represent the row number and column number of the feature point, H, W represent the height and width of the offset modal feature map respectively, and n=HW represents the number of elements in the offset modal feature map.

[0101] Preferably, the feature offset prediction module is pre-trained using a FLIR-aligned dataset.

[0102] Step 3: Concatenate the corrected offset modal feature map with the reference modal feature map and input it into the region generation network to obtain the candidate box.

[0103] Specifically, the modified offset modal feature map obtained in step 2 is Compared with the reference modal characteristic diagram F R After cascading, the input region generation network is used to obtain the candidate box Proposal O ,Proposal R .

[0104] Step 4: Input the candidate box into the region of interest alignment module, perform secondary regression on the candidate box of the offset modality, and obtain the corrected offset modality candidate box and the corrected offset modality region of interest features.

[0105] See also Figure 4 , Figure 4 For the region of interest alignment module, the candidate box obtained in step 3 is input as follows Figure 4 The region of interest alignment module shown in the figure performs secondary regression on the candidate box of the offset mode to further correct the spatial offset of the feature and obtain the corrected candidate box of the offset mode. And the modified offset modal region of interest features

[0106] Furthermore, step 4 specifically includes:

[0107] Step 41: First, expand the context of the region of interest for the candidate boxes of the two modalities. For each candidate box, fix the center, expand the width and height to 3 times and 2 times the original respectively, and obtain the region of interest of the offset modality. and the region of interest of the reference modality Make it contain more contextual information.

[0108] Step 42: The offset modality image pools the expanded region of interest features to obtain the region of interest features of the offset modality and the region of interest characteristics of the reference modality Where RoIPooling(·) represents the region of interest pooling, F O and F R are the extracted offset modality image features and reference modality image features respectively.

[0109] Step 43: F ORoI ,F RRoI Perform channel cascading to predict 4 parameters for each offset modal candidate box Among them, c x 、c y Represents the change of the center point coordinates, c w 、c h The change in height and width, F(·) represents the candidate box regression layer, σ(·) represents the tanh activation function, and CONCAT(·) represents the channel cascade.

[0110] Step 44: Use 4 hyperparameters to constrain the change amount, then the corrected candidate box The coordinates are:

[0111]

[0112]

[0113]

[0114]

[0115] The candidate box regression layer consists of a 3×3 convolutional layer and two fully connected layers, and the hyperparameter thr x ,thr y ,thr w ,thr h Set to w, h, w / 2, h / 2 respectively, where w and h represent the width and height of the current candidate box, x o ,y o ,w o ,h o Represents the center coordinates, width and height of the initial candidate box.

[0116] Preferably, step 4 also includes training of a region of interest alignment module, that is, guiding network training by shortening the distance between the maximum response points of the two modal region of interest features of the corrected offset modal region of interest feature and the reference modal region of interest feature. Specifically, it includes:

[0117] Use the corrected offset modal candidate box obtained in step 4 Calculate the corrected offset modal region of interest features where RoIPooling(·) stands for region of interest pooling.

[0118] Region of interest features for the modified offset mode And the reference modality region of interest feature F RRoI , calculate the channel average to get the response diagram of the two modes M RRoI ,in, That is, the response graph is actually an H×W matrix, and the matrix elements are real numbers.

[0119] The points with the largest response are P R =(x R ,y R )=argmax(M RRoI ), calculate the Euclidean distance between the two and use it as the loss function: This is to shorten the distance between the maximum response points of the features of the regions of interest of the two modalities to guide network training.

[0120] Preferably, since there is no clear true label in this part, the loss function is Introduce weights to constrain the effect of the loss value, and set the weight to 0.5. Use the stochastic gradient descent (SGD) method to optimize the network weights, set the SGD momentum to 0.9, the weight decay coefficient to 0.0001, and the learning rate to 0.002.

[0121] Step 5: Downsample the offset modality image and input it into the illumination-aware network to predict the weights of the two modality features.

[0122] For details, see Figure 5 , Figure 5 In order to utilize the illumination-aware network N(·) to implement the feature weighting module based on illumination guidance so as to adaptively assign weights to the two modalities, the offset modality image I O Downsample and input the light perception network to obtain the probability C of the input image being day or night d =δ(N(I R )), C n =1-C d , and then the probability value is adjusted by the gate function to obtain a reasonable modal weight. The weight of the offset modal feature is The weight of the reference modal feature is W R =1-W O , where δ represents the softmax activation function, and α represents the learnable parameter, which is initially set to 1. The downsampling multiple is 1 / 8 of the original image. The illumination perception network N(·) contains 2 convolutional layers and 3 fully connected layers. Each convolutional layer is followed by a ReLU activation function layer and a 2×2 maximum pooling layer to activate and compress the features, and a softmax activation function is used after the last fully connected layer.

[0123] Preferably, step 5 also includes training of a light perception network.

[0124] Specifically, the loss function used in the training of the light perception network is The back-propagation method is used to train the light perception network N(·), where T is the total number of samples and y irepresents the classification label of the i-th input image, p i Predict the probability of daytime for the i-th input image. Use the stochastic gradient descent (SGD) method to optimize the network weights, set the SGD momentum to 0.9, the weight decay coefficient to 0.0001, and the learning rate to 0.002.

[0125] Step 6: Use the modal weights obtained in step 5 to re-weight the region of interest features of the two modalities obtained in step 4, and send them to the detection head after cascading to obtain the location information of the target of interest, completing the training of the target detection model.

[0126] Specifically, the modal weights obtained in step 5 are used to transform the region of interest features of the two modalities obtained in step 4 F RRoI After re-weighting, the concatenation is fed into the detection head (i.e., classification and regression network) D(·) to obtain the classification confidence and regression coordinates of the object of interest.

[0127] Among them, the fusion feature And calculate the category confidence p of the i-th anchor box i and the predicted regression value l i for (p i ,l i )=D(F F ), where CONCAT(·) represents channel concatenation.

[0128] Preferably, step 6 also includes training of the detection head, adjusting parameters according to the loss function to obtain a trained detection head, so that the network can correctly predict the position and category of the target of interest.

[0129] Specifically, the loss function in the training of the detection head is This enables the network to correctly predict the location and category of the target of interest, where N represents the total number of anchor boxes and p i and Represent the predicted category score and the true category label of the i-th anchor box, respectively, i and They represent the predicted regression value and true value of the i-th anchor box respectively, and λ is the weight factor, which is set to 1.

[0130] The loss function includes classification loss and regression loss, where the classification loss function uses the softmax cross entropy loss function. K is the total number of categories, y is the label, when the predicted category is different from the true category label When consistent otherwise is the softmax probability that anchor box i belongs to category n,

[0131] Regression loss uses smooth L1 Loss. Since the coordinates of the predicted box and the true box are not directly used when calculating the regression loss, the actual regression loss is

[0132]

[0133]

[0134]

[0135]

[0136] Among them, x, y, w, h represent the center coordinates, height and width of the box, x p ,x a ,x * They correspond to the predicted box, anchor box and true box values ​​regressed by the network, and the same is true for y, w, and h. The model is trained using the back-propagation method, and the network weights are optimized using the stochastic gradient descent (SGD) method. The SGD momentum is set to 0.9. The weight decay coefficient is 0.0001, and the learning rate is set to 0.002.

[0137] Step 7: Input the image to be tested into the trained target detection model to obtain the target detection result.

[0138] Specifically, the image to be tested is input into the trained target detection model, and the output of the structured network model can be obtained by performing forward propagation of the network. The output result of the network model is subjected to non-maximum suppression to obtain the final target detection result.

[0139] Preferably, the non-maximum suppression threshold is set to 0.5.

[0140] Embodiment 2

[0141] With respect to the target detection method based on feature alignment provided in the first embodiment of the present invention, this embodiment provides test results of the method to evaluate its performance.

[0142] During the test, the FLIR ADAS dataset was selected for accuracy testing. The FLIR ADAS dataset was released in 2018 and contains visible light images and infrared images, which were acquired by the FLIRBlackFly visible light camera with a resolution of 1280×1024 and the FLIR Tau2 infrared camera with a resolution of 640×512 mounted on the vehicle. The entire dataset contains a total of 10,228 pairs of one-to-one visible light-infrared image pairs, of which the training set has 8,862 pairs of image pairs and the test set has 1,366 pairs of image pairs. Since the visible light and infrared images have a large spatial deviation, the FLIR ADAS dataset only annotates the images of the infrared mode.

[0143] The FLIR ADAS dataset has four target categories labeled: people, bicycles, vehicles, and dogs. Since there are very few dogs in the dataset, only the first three target categories were retained, including 28,151 pedestrian instances, 46,692 vehicle instances, and 4,457 bicycle instances. In addition, there are a small number of infrared images that do not have corresponding visible light images. Unpaired images were removed during preprocessing, and 8,363 pairs of images were retained as training sets and 1,366 pairs of images were retained as test sets.

[0144] The feature map visualization results of the feature offset prediction module and the final detection results are shown in Figure 2. Figure 6 and Figure 7 shown. Figure 6 From top to bottom are visible light features, infrared features, and corrected visible light features. Figure 7 The solid line box represents the correct detection result, the dashed line box represents the missed detection, and the dotted line box represents the false alarm.

[0145] The analysis index used by this method to measure the detection accuracy is: Average Precision (AP).

[0146] The experiment evaluated the accuracy of each target category when the IoU threshold was 0.5 (AP50), the accuracy of all target categories when the IoU threshold was 0.5 (mAP50), and the average precision (mAP) calculated with 10 IoU thresholds such as 0.5:0.05:0.95. The test results are shown in Table 1.

[0147] Table 1 Experimental results of the visible light and infrared image fusion target detection algorithm on FLIR ADAS

[0148]

[0149]

[0150] The test results are shown in Table 1. From the quantitative and qualitative analysis of the detection accuracy, it can be seen that the detection accuracy of this method on the FLIRADAS dataset has reached a leading level.

[0151] Embodiment 3

[0152] Embodiment 3 of the present invention further provides a target detection system based on feature alignment, comprising:

[0153] Extraction module: The offset modality image and the reference modality image are respectively input into two deep convolutional neural networks with the same structure to extract image features. The two networks do not share parameters.

[0154] Feature offset prediction module: input the extracted offset modal feature map and the reference modal feature map into the feature offset prediction module, predict the offset of each feature position of the offset modal feature map, and correct the feature position to preliminarily align the feature maps of the two modalities;

[0155] Region generation network: The corrected offset modal feature map is cascaded with the reference modal feature map and then input into the region generation network to obtain the candidate box;

[0156] Region of Interest Alignment Module: Input the candidate frame into the region of interest alignment module, perform secondary regression on the candidate frame of the offset modality, and obtain a corrected candidate frame of the offset modality and corrected region of interest features of the offset modality;

[0157] Feature weighting module: downsamples the offset modality image and inputs it into the illumination perception network to predict the weights of the two modality features;

[0158] Fusion module: Use the modal weights obtained by the feature weighting module to re-weight the ROI features of the two modalities obtained by the ROI alignment module, and then send them to the detection head after cascading to obtain the location information of the target of interest, thus completing the training of the target detection model;

[0159] Target detection module: input the image to be tested into the trained target detection model to obtain the target detection result.

[0160] Preferably, the feature shift prediction module specifically includes:

[0161] Concatenate the feature map channels of the two modalities F C =CONCAT(F O ,F R ), CONCAT(·) represents channel concatenation;

[0162] The multi-head attention module and the reverse residual feed-forward network are used to extract features and predict the x and y offsets of each feature position on the feature map. Among them, MHSA(·) stands for multi-head self-attention, IRFFN(·) stands for inverse residual feedforward network, represents a 1×1 convolutional layer, σ(·) represents the tanh activation function, and the threshold thr is used. W ,thr H As a hyperparameter, the offset of each feature in the x and y directions is ΔX = p x ·thr W ,ΔY=p y ·thr H ;

[0163] The corrected feature position of the offset modal feature map is X'=X⊕ΔX, Y'=Y⊕ΔY. The corrected offset modal feature map (X′,Y′:V)←(X,Y:V) is obtained by transforming the position of the eigenvalues, where ⊕ represents element-by-element addition, (X′,Y′:V)←(X,Y:V) means moving the value V of the feature point with coordinates X,Y to the new coordinates X',Y', and ← represents assignment to achieve the correction of the feature position.

[0164] Preferably, the feature shift prediction module further includes pre-training the feature shift prediction module, specifically including:

[0165] Each pixel of the offset modal image is randomly offset in the x direction by [-thr W ,thr W ] and random offset in the y direction [-thr H ,thr H ] pixels, and downsample the shifted image as the true value of the feature map shift;

[0166] The loss function for pre-training the feature offset prediction module is Among them, p ,O g They represent the predicted value and true value of the offset respectively, i, j represent the row number and column number of the feature point, H, W represent the height and width of the feature map respectively, and n=HW represents the number of elements in the feature map.

[0167] Preferably, the region of interest alignment module includes:

[0168] Extension module: First, expand the context of the region of interest for the candidate boxes of the two modalities to obtain and

[0169] Pooling module: pool the expanded region of interest features to obtain the region of interest features of the offset mode and the region of interest characteristics of the reference modality Where RoIPooling(·) represents the region of interest pooling, F O and F R are the extracted offset modality image features and reference modality image features respectively;

[0170] Prediction module: F ORoI ,F RRoI Perform channel cascading to predict 4 parameters for each offset modal candidate box Among them, c x 、c y Represents the change of the center point coordinates, c w 、c h The change in height and width, F(·) represents the candidate box regression layer, σ(·) represents the tanh activation function, and CONCAT(·) represents the channel cascade.

[0171] Correction module: 4 hyperparameters are used to constrain the change amount, and the corrected candidate box The coordinates are:

[0172]

[0173]

[0174]

[0175]

[0176] Parameter thr x ,thr y ,thr w ,thr h Set to w, h, w / 2, h / 2 respectively, where w and h represent the width and height of the current candidate box, x o ,y o ,w o ,h o Represents the center coordinates, width and height of the initial candidate box.

[0177] Preferably, the region of interest alignment module further includes:

[0178] Pooling module: using the corrected offset modal candidate box obtained in the region of interest alignment module Calculate the corrected offset modal region of interest features Where RoIPooling(·) represents region of interest pooling;

[0179] Network training module: Features of the region of interest for the corrected offset mode And the reference modality region of interest feature F RRoI, calculate the channel average to get the response diagram of the two modes M RRoI The points with the largest response are P R =(x R ,y R )=argmax(M RRoI ), calculate the Euclidean distance between the two and use it as the loss function: This is done to shorten the distance between the maximum response points of the features in the regions of interest of the two modalities and guide network training.

[0180] It should be understood that parts not elaborated in detail in this specification belong to the prior art.

[0181] The protection scope of the present invention is not limited to the above-mentioned embodiments. Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the scope and spirit of the present invention. If these changes and modifications fall within the scope of the claims of the present invention and their equivalents, the present invention is intended to include these changes and modifications.

Claims

1. A target detection method based on feature alignment, characterized in that: The method comprises: Step 1: Input the offset modality image and the reference modality image into two deep convolutional neural networks with exactly the same structure to extract image features. The two networks do not share parameters. Step 2: Input the extracted offset modal feature map and the reference modal feature map into the feature offset prediction module, predict the offset of each feature position of the offset modal feature map, and correct the feature position to preliminarily align the feature maps of the two modalities; The offset is calculated as follows: Cascade the feature map channels of the two modalities , CONCAT(·) represents channel cascade; features are extracted through a multi-head attention module and a reverse residual feedforward network, and the offsets in the predicted x and y directions for each feature position on the feature map are calculated. , MHSA(·) stands for multi-head self-attention, IRFFN(·) stands for inverse residual feedforward network, (·) represents a 1×1 convolutional layer, and σ (·) represents the tanh activation function; Step 3: Concatenate the corrected offset modal feature map and the reference modal feature map and input them into the region generation network to obtain the candidate box; Step 4: input the candidate box into the region of interest alignment module, perform secondary regression on the candidate box of the offset modality, and obtain the corrected candidate box of the offset modality and the corrected region of interest features of the offset modality; Step 5: Downsample the offset modality image and input it into the illumination perception network to predict the weights of the two modality features and adaptively assign weights to the two modalities; Step 6: Use the modal weights obtained in step 5 to re-weight the features of the regions of interest of the two modalities obtained in step 4, and send them to the detection head after cascading to obtain the location information of the target of interest, completing the training of the target detection model; Step 7: Input the image to be tested into the target detection model obtained in step 6 to obtain the target detection result.

2. The method according to claim 1, characterized in that: Step 2 specifically includes: Step 21: Cascade the feature map channels of the two modalities , CONCAT(·) represents channel concatenation; Step 22: Extract features through the multi-head attention module and the reverse residual feedforward network, and predict the x and y offsets of each feature position on the feature map, the offset , where MHSA(·) stands for multi-head self-attention and IRFFN(·) stands for inverse residual feedforward network. (·) represents a 1×1 convolutional layer, σ (·) represents the tanh activation function; the threshold is used As a hyperparameter, the offset of each feature in the x and y directions is ; Step 23: The corrected feature position of the offset modal feature map is , the modified offset modal characteristic diagram is obtained by transforming the eigenvalue position ,in represents element-by-element addition, Indicates that the coordinates are The value of the feature point Move to new coordinates middle, Represents assignment to achieve correction of feature position.

3. The method according to claim 2, characterized in that: Step 2 also includes pre-training the feature offset prediction module, specifically including: Randomly shift each pixel of the offset modal feature map in the x direction And random offset in the y direction pixels, and downsample the shifted image as the true value of the feature map shift; The loss function for pre-training the feature offset prediction module is ,in Represent the predicted value and the true value of the deviation respectively, Represents the row and column numbers where the feature points are located. Respectively represent the height and width of the offset modal feature map, Represents the number of elements in the offset modal feature map.

4. The method according to claim 1, characterized in that: Step 4 specifically includes: Step 41: First, expand the context of the region of interest for the candidate boxes of the two modalities to obtain and ; Step 42: Pool the expanded region of interest features to obtain the region of interest features of the offset mode and the region of interest characteristics of the reference modality , where RoIPooling(·) represents region of interest pooling, and are the extracted offset modality image features and reference modality image features respectively; Step 43: Perform channel cascading to predict 4 parameters for each offset modal candidate box , where c x 、c y Represents the change of the center point coordinates, c w 、c h Represents the change in width and height of the candidate box respectively, (·) represents the candidate box regression layer, σ(·) represents the tanh activation function, and CONCAT(·) represents the channel cascade; Step 44: Use 4 hyperparameters to constrain the change amount, then the corrected candidate box The coordinates are: Hyperparameters Set as , and Represents the width and height of the current candidate box. Represents the center coordinates, width and height of the initial candidate box.

5. The method according to claim 4, characterized in that: Step 4 also includes a training step: Use the corrected offset modal candidate box obtained in step 4 , calculate the corrected offset modal region of interest features , where RoIPooling(·) represents region of interest pooling; Region of interest features for the modified offset mode And the reference modality region of interest features obtained in step 4 , calculate the channel average to get the response diagram of the two modes The points with the largest response are , calculate the Euclidean distance between the two and use it as the loss function: , in order to shorten the distance between the maximum response points of the features of the regions of interest of the two modalities and guide the network training.

6. A target detection system based on feature alignment, characterized in that: The system comprises: Extraction module: The offset modality image and the reference modality image are respectively input into two deep convolutional neural networks with exactly the same structure to extract image features. The two networks do not share parameters. Feature offset prediction module: input the extracted offset modal feature map and the reference modal feature map into the feature offset prediction module, predict the offset of each feature position of the offset modal feature map, and correct the feature position to preliminarily align the feature maps of the two modalities; The offset is calculated as follows: Cascade the feature map channels of the two modalities , CONCAT(·) represents channel cascade; features are extracted through a multi-head attention module and a reverse residual feedforward network, and the offsets in the predicted x and y directions for each feature position on the feature map are calculated. , MHSA(·) stands for multi-head self-attention, IRFFN(·) stands for inverse residual feedforward network, (·) represents a 1×1 convolutional layer, and σ (·) represents the tanh activation function; Region generation network: The corrected offset modal feature map is cascaded with the reference modal feature map and then input into the region generation network to obtain the candidate box; Region of Interest Alignment Module: Input the candidate box into the region of interest alignment module, perform secondary regression on the candidate box of the offset modality, and obtain a corrected offset modality candidate box and a corrected offset modality region of interest feature; Feature weighting module: downsamples the offset modality image and inputs it into the illumination perception network to predict the weights of the two modality features, and adaptively assigns weights to the two modalities; Fusion module: Use the modal weights obtained by the feature weighting module to re-weight the ROI features of the two modalities obtained by the ROI alignment module, and then send them to the detection head after cascading to obtain the location information of the target of interest, thus completing the training of the target detection model; Target detection module: Input the image to be tested into the target detection model obtained by the fusion module to obtain the target detection result.

7. The system according to claim 6, characterized in that: The feature offset prediction module specifically includes: Cascade the feature map channels of the two modalities , CONCAT(·) represents channel concatenation; The multi-head attention module and the reverse residual feed-forward network are used to extract features and predict the x and y offsets of each feature position on the feature map. , where MHSA(·) stands for multi-head self-attention and IRFFN(·) stands for inverse residual feedforward network. (·) represents a 1×1 convolutional layer, σ (·) represents the tanh activation function; the threshold is used As a hyperparameter, the offset of each feature in the x and y directions is ; The corrected feature position of the offset modal feature map is , the modified offset modal characteristic diagram is obtained by transforming the eigenvalue position ,in represents element-by-element addition, Indicates that the coordinates are The value of the feature point Move to new coordinates middle, Represents assignment to achieve correction of feature position.

8. The system according to claim 7, characterized in that: The feature shift prediction module further includes pre-training the feature shift prediction module, specifically including: Randomly shift each pixel of the offset modal feature map in the x direction And random offset in the y direction pixels, and downsample the shifted image as the true value of the feature map shift; The loss function for pre-training the feature offset prediction module is ,in Represent the predicted value and the true value of the deviation respectively, Represents the row and column numbers where the feature points are located. Respectively represent the height and width of the offset modal feature map, Represents the number of elements in the offset modal feature map.

9. The system according to claim 6, characterized in that: The region of interest alignment module comprises: Extension module: First, expand the context of the region of interest for the candidate boxes of the two modalities to obtain and ; Pooling module: The offset modality image pools the expanded region of interest features to obtain the region of interest features of the offset modality and the region of interest characteristics of the reference modality , where RoIPooling(·) represents region of interest pooling, and are the extracted offset modality image features and reference modality image features respectively; Prediction module: Perform channel cascading to predict 4 parameters for each offset modal candidate box :Among them, c x 、c y Represents the change of the center point coordinates, c w 、c h Represents the change in width and height of the candidate box respectively, (·) represents the candidate box regression layer, σ(·) represents the tanh activation function, and CONCAT(·) represents the channel cascade; Correction module: 4 hyperparameters are used to constrain the change amount, and the corrected candidate box The coordinates are: Hyperparameters Set as ,in and Represents the width and height of the current candidate box. Represents the center coordinates, width and height of the initial candidate box.

10. The system according to claim 9, characterized in that: The region of interest alignment module also includes: Pooling module: using the corrected offset modal candidate box obtained in the region of interest alignment module , calculate the corrected offset modal region of interest features , where RoIPooling(·) represents region of interest pooling; Network training module: Features of the region of interest for the corrected offset mode And the reference modality region of interest characteristics , calculate the channel average to get the response diagram of the two modes The points with the largest response are , calculate the Euclidean distance between the two and use it as the loss function: , in order to shorten the distance between the maximum response points of the features of the regions of interest of the two modalities and guide the network training.

Images