Visible light assisted infrared target detection method, device and medium

Abstract

The application discloses a visible light assisted infrared target detection method and device and a medium, and belongs to the field of infrared detection, and comprises the following steps: firstly, a certain number of infrared visible light feature co-occurrence matrices are constructed, and the purpose is to approximately represent the two-dimensional probability distribution between infrared features and visible light features; secondly, a conditional probability distribution migration module is proposed, and the purpose is to enable the feature extraction network to have a certain visible light feature extraction capability; finally, in the reasoning test stage, only the infrared image needs to be input, and the visible light image does not need to be input, and the reasoning process is completely consistent with the traditional reasoning process, and no additional time and space overheads are generated. The application has the advantages of high detection precision and fast reasoning speed.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of infrared detection, and more particularly to a visible light assisted infrared target detection method, device and medium. BACKGROUND

[0002] Target detection is a basic but very important task in the field of computer vision. Nowadays, high-speed parallel computing architectures represented by the NVIDIA series have gradually developed and grown up; at the same time, target detection data sets represented by VOC and COCO data sets have been proposed, so that the target detection algorithm based on deep learning has become the mainstream. Under this trend, intelligent technology, monitoring technology, security technology, military technology and many other technologies relying on computer vision, especially target detection field, have developed rapidly. At the same time, in addition to traditional cameras, infrared devices and other image acquisition devices have also been concerned. Due to its unique acquisition principle, the images collected by such image acquisition hardware can often make up for the information loss of visible light images to a certain extent. In complex climate environment (such as night and rainy day, etc.), infrared images often provide more information.

[0003] In recent years, due to the increasing richness of infrared acquisition devices and the increasing maturity of target detection algorithms, target detection methods based on infrared images have gradually attracted people's attention. At present, the industry has proposed corresponding target detection algorithms and systems for infrared images. However, these methods have obvious problems. Since the color and texture information of infrared images is relatively small, the effect of directly using infrared images to train the target detection model is often limited. SUMMARY

[0004] The purpose of the present application is to overcome the shortcomings of the prior art and provide a visible light assisted infrared target detection method, device and medium, which provides an infrared image target detection method with high detection accuracy and fast reasoning speed, without additional time and space overhead.

[0005] The purpose of the present application is achieved by the following scheme:

[0006] A visible light assisted infrared target detection method, comprising the following steps:

[0007] Constructing a feature co-occurrence matrix for approximately representing the two-dimensional probability distribution between infrared features and visible light features; and constructing a conditional probability distribution transfer module for enabling the feature extraction network to have the ability to extract visible light features, and using visible light images to assist infrared images for target detection;

[0008] Based on the feature co-occurrence matrix and the conditional probability distribution transfer module, a visible light assisted infrared target detection model combined with a target detection algorithm is constructed, and the corresponding target of the image is predicted through the feature extraction network and the target detection algorithm.

[0009] Further, the feature co-occurrence matrix is constructed, and specifically includes the following sub-steps:

[0010] S11, respectively training a feature extraction network of a visible light target detection model and a feature extraction network of an infrared target detection model;

[0011] S12, for each visible light infrared image in the training set, obtaining a corresponding visible light feature tensor and an infrared feature tensor through the corresponding feature extraction network;

[0012] S13, dimension reduction of the visible light feature tensor and the infrared feature tensor, thereby obtaining a visible light feature vector and an infrared feature vector, respectively;

[0013] S14, normalizing the feature vectors to change the value to the value range of 0 to 1;

[0014] S15, respectively performing discrete sampling statistics on each component of the feature vector to obtain a feature co-occurrence matrix.

[0015] Further, the conditional probability distribution transfer module is constructed, and specifically includes the following sub-steps:

[0016] S21, obtaining a corresponding feature tensor through the feature extraction network of the target detection model for each infrared image;

[0017] S22, inputting the feature tensor into the conditional probability prediction network to predict a corresponding mean vector and variance vector to obtain a loss;

[0018] S23, inputting the feature tensor into the neck network and the detection head network to obtain a predicted target detection result, and calculating a loss;

[0019] S24, adding the losses in steps S22 and S23 to optimize the entire infrared target detection model.

[0020] Further, the visible light assisted infrared target detection model combined with the target detection algorithm is constructed based on the feature co-occurrence matrix and the conditional probability distribution transfer module, and specifically includes the following sub-steps:

[0021] S31, in the model training stage, updating the parameters of the feature extraction network through the feature co-occurrence matrix and the conditional probability distribution transfer module;

[0022] S32, in the inference stage, the process is consistent with the target detection algorithm, but only infrared images need to be input, and no visible light images need to be input, and the feature extraction network and the decoding of the target detection model are used to predict the corresponding target of each image.

[0023] Further, the target detection algorithm includes a YOLOv3 target detection algorithm.

[0024] Further, the target corresponding to the predicted image includes a coordinate frame and a category of the target.

[0025] Further, in step S15, the number of constructed feature co-occurrence matrices is the same as the length of the dimension-reduced feature vector in step S13.

[0026] A visible light assisted infrared target detection device comprises:

[0027] The feature co-occurrence matrix construction module is configured to perform the following processes: first, training a feature extraction network of a visible light target detection model and a feature extraction network of an infrared target detection model respectively; second, for each visible light infrared image in the training set, obtaining corresponding visible light feature tensors and infrared feature tensors through the corresponding feature extraction networks; third, dimension-reducing the visible light feature tensors and the infrared feature tensors to obtain visible light feature vectors and infrared feature vectors respectively; fourth, performing a normalization operation on the feature vectors to change the values to a value range of 0 to 1; and fifth, performing discrete sampling statistics on each component of the feature vectors to obtain a feature co-occurrence matrix.

[0028] The conditional probability distribution migration module is configured to perform the following processes: first, obtaining a corresponding feature tensor for each infrared image through the feature extraction network of the target detection model; second, inputting the feature tensor into the conditional probability prediction network to obtain a loss by predicting a corresponding mean vector and a variance vector; third, inputting the feature tensor into the neck network and the detection head network to obtain a predicted target detection result and calculate a loss; and fourth, adding the two losses to optimize the entire infrared target detection model.

[0029] Further, the visible light assisted infrared target detection network construction module combined with the target detection algorithm is configured to predict a target corresponding to an image through the feature extraction network and the target detection algorithm.

[0030] A computer readable storage medium stores a computer program, the computer program is loaded and executed by a processor to perform the method of any one of the above.

[0031] The beneficial effects of the present application include:

[0032] The present application uses an existing target detection algorithm and realizes a method for assisting infrared image target detection by visible light image based on a transfer learning technology. On the basis of the infrared target detection method in recent years, the present application proposes an additional auxiliary task to train the feature extraction network part of the detection model. The auxiliary task can introduce the information of the visible light image, thereby improving the quality of feature extraction of the feature extraction network for the infrared image.

[0033] The application introduces visible light domain to ensure detection accuracy; meanwhile, the application only updates the parameters of the feature extraction network through auxiliary tasks, is completely consistent with the target detection model in the inference stage, does not introduce additional time and space overheads, and is convenient for deployment.

[0034] The application scheme can be combined with any target detection algorithm. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced as follows. Obviously, the drawings in the following description only constitute some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0036] Figure 1 It is a general flow chart of the prior art scheme one.

[0037] Figure 2 It is a general flow chart of the prior art scheme two.

[0038] Figure 3 It is a general flow chart of the visible light assisted infrared target detection method of the embodiment of the application.

[0039] Figure 4 It is a feature co-occurrence matrix construction schematic diagram of the embodiment of the application.

[0040] Figure 5 It is a conditional probability distribution query schematic diagram of the embodiment of the application.

[0041] Figure 6 It is a distribution prediction network model schematic diagram of the embodiment of the application. DETAILED DESCRIPTION

[0042] All features disclosed in all embodiments in the specification, or steps in all methods or processes implicitly disclosed, can be combined and / or extended, replaced, or modified in any way, except for mutually exclusive features and / or steps.

[0043] In view of the problems and status in the background, the inventors of the application further search and analyze and realize the following technical problems:

[0044] In the prior art, a Chinese patent application with publication number CN115331183A proposes an infrared target detection scheme for improving YOLOv5s, Figure 1 It is a general flow chart of the infrared target detection scheme for improving YOLOv5s.

[0045] The inventors of the present application further analyze and consider that the patent application scheme uses the modified YOLOv5 to train only on infrared images. However, due to the lack of texture information and color information, the model has limited feature extraction for infrared images. Training only on infrared images and performing feature enhancement and loss function design on infrared features often makes it difficult to obtain effective features. In addition, when the target is small, the features of the infrared target itself are very limited and need to be compared with the context. However, down-sampling the features and concatenating the detection heads will not have a significant improvement. Therefore, for infrared images, the existing method cannot meet the higher detection accuracy.

[0046] In the prior art, a Chinese patent application with publication number CN 115424028 A provides an infrared target detection scheme, Figure 2 A flowchart of the infrared target detection method based on the feature-enhanced lightweight SSD.

[0047] The inventors of the present application further analyze and consider that the patent application scheme does not transfer knowledge from the visible light domain because visible light images often have more rich color information and texture information, and the target detection algorithm, data set, and pre-training model corresponding to the visible light image are very mature. Therefore, the accuracy obtained by training the target detection model only on thermal infrared images is often limited.

[0048] To solve the further discovered technical problems, in the concept of the present application, the inventors consider that, compared with the traditional target detection task of visible light images, the target detection task of infrared images faces more challenges, mainly including low image resolution, lack of color information, insufficient edge contour information, high image noise, and high similarity between target and background making it difficult to distinguish. In addition, due to application requirements, there is a strong limitation on the time and space of model inference, which requires higher accuracy, inference speed, and model robustness of the algorithm, and also limits the use of some methods. The existing infrared image target detection algorithm either lacks visible light image assistance resulting in low detection accuracy, or the visible light assisted infrared transfer module requires additional time and space overhead resulting in slow inference speed, which all indicates that this problem needs further research. The present application proposes a visible light image assisted infrared image target detection method, which introduces visible light domain to ensure detection accuracy; at the same time, the present application scheme only updates the parameters of the feature extraction network through the auxiliary task, and the inference stage and the target detection model are completely consistent, which will not introduce additional time overhead and space overhead, and is easy to deploy. This has great significance and far-reaching influence on the construction of national economy, and has a great promoting effect on the development of infrared image target detection.

[0049] For the ready-made target detection algorithm, in addition to directly using the infrared image for the training of the target detection model, the feature extraction network of the target detection model should still be able to extract some visible light domain features when facing the infrared image for input. Therefore, the application proposes an additional auxiliary task to assist in training the feature extraction network of the target detection model. For each infrared image, the probability distribution of the corresponding visible light feature, i.e. the conditional probability distribution, is predicted to assist in training the feature extraction network, which specifically includes the following steps:

[0050] Firstly, a target detection model is trained using all visible light images in the training set, and the corresponding feature extraction network is saved; at the same time, a target detection model is trained using all infrared images in the training set, and the corresponding feature extraction network is saved;

[0051] Secondly, for each visible light image in the training set, the corresponding feature is extracted by the visible light feature extraction network in the first step; at the same time, for each infrared image in the training set, the corresponding feature is extracted by the visible light feature extraction network in the first step;

[0052] Thirdly, the visible light feature extracted in the second step is reduced to a fixed-length vector; at the same time, the infrared feature extracted in the second step is reduced to a fixed-length vector;

[0053] Fourthly, the visible light feature vector and the infrared feature vector obtained in the third step are normalized, so that the value range of each component of the vector is between 0 and 1;

[0054] Fifthly, a certain number of feature co-occurrence matrices are constructed, and the number is the same as the length of the feature vector after dimension reduction in the third step; for the visible light feature vector and the infrared feature vector after dimension reduction and normalization of each image, the corresponding value is discretized and sampled, and is counted into the corresponding feature co-occurrence matrix;

[0055] Sixthly, each row in the feature co-occurrence matrix counted in the fifth step is normalized, and each element is divided by the sum of all elements in each row;

[0056] Seventhly, for each infrared image in the training set, the discrete value of each component of the infrared feature vector calculated in the fifth step is queried to obtain the conditional probability distribution of the corresponding co-occurrence matrix, and the mathematical expectation and variance corresponding to the conditional probability distribution are calculated;

[0057] Eighthly, a new target detection model is initialized, and for each infrared image, the feature tensor of each infrared image is obtained by the feature extraction network of the target detection model;

[0058] Step 9, build a distribution prediction model, including a 1x1 convolution layer, a ReLU activation function and a 2x2 pooling layer for feature dimension reduction; then the reduced features are respectively passed through two structurally identical multi-layer perceptrons to predict the mean vector and the variance vector respectively; the multi-layer perceptron includes a fully connected layer, a ReLU activation function, a dropout layer and another fully connected layer;

[0059] Step 10, input the infrared feature tensor of the infrared image calculated in step 8 into the distribution prediction model built in step 9 to obtain the predicted mean vector and variance vector of the conditional probability distribution corresponding to the infrared image, and calculate the mean square error with the real mean vector and variance vector as the distribution prediction loss function;

[0060] Step 11, input the infrared feature tensor of the infrared image calculated in step 8 into the neck network and detection head network of the target detection model to obtain the predicted detection result of the infrared image, including the coordinates of several predicted boxes and the corresponding categories, and compare it with the real label to calculate the detection loss of target detection; sum the detection loss and the distribution prediction loss to obtain the total loss and optimize the target detection model.

[0061] In specific implementation details, as shown in Figures 3-6 The present application scheme includes two modules, a feature co-occurrence matrix construction module and a probability distribution transfer module. For the feature co-occurrence matrix construction module, first, train a feature extraction network for a visible light target detection model and a feature extraction network for an infrared target detection model respectively; second, for each visible light infrared image in the training set, obtain the corresponding visible light feature tensor and infrared feature tensor through the corresponding feature extraction network; third, reduce the visible light feature tensor and the infrared feature tensor to obtain the visible light feature vector and the infrared feature vector respectively; fourth, normalize the feature vectors to change the value to the value range of 0 to 1; fifth, discretize and sample each component of the feature vector to obtain the feature co-occurrence matrix. For the conditional probability distribution transfer module, first, pass each infrared image through the feature extraction network of the target detection model to obtain the corresponding feature tensor; second, pass the feature tensor into the conditional probability prediction network to predict the mean vector and the variance vector to obtain the loss; third, pass the feature tensor into the neck network and the detection head network to obtain the predicted target detection result and calculate the loss; finally, add the two losses to optimize the entire infrared target detection model. The present application scheme can be combined with any target detection algorithm. The following takes the YOLOv3 target detection algorithm as an example to further illustrate the specific implementation of the present application scheme.

[0062] 1. Feature co-occurrence matrix construction module

[0063] This module aims to construct a two-dimensional probability density between visible light features and infrared features, and store it approximately with a certain number of matrices. The specific implementation details of this module are as follows:

[0064] 1) First, use the YOLOv3 target detection algorithm to train a target detection algorithm for visible light images and infrared images respectively; then save the feature extraction network of the trained YOLOv3 model; YOLOv3 target detection model uses Darknet53 as the feature extraction network, Darknet53 has an important feature that uses residual network Residual, the residual convolution in Darknet53 is first a convolution with a kernel size of 3x3 and a step of 2, this convolution will compress the width and height of the input feature layer, at this time a feature layer can be obtained, which is named layer. Then perform a 1x1 convolution and a 3x3 convolution on the feature layer, and add the result to layer, which constitutes a residual structure at this time; by continuously adding 1x1 convolution and 3x3 convolution and residual edges, the network is greatly deepened; the feature of residual network is easy to optimize, and can improve the accuracy by increasing the depth; the residual block inside uses jump connection, which alleviates the gradient vanishing problem caused by increasing the depth in deep neural network.

[0065] Darknet53 includes a convolution layer and five residual blocks (Residual Block 1x64, Residual Block 2x128, Residual Block 8x256, Residual Block 8x512 and Residual Block 4x1024); the input channel number, output channel number, convolution kernel size, convolution operation step and padding parameters of each convolution operation unit of the feature extraction network (Darknet53) are fixed, and the size of each infrared image and visible light image is converted to 640x512x3 into the feature extraction network (Darknet53), and the final feature tensor size is 20x16x1024.

[0066] 2) Secondly, the feature tensor with a size of 20x16x1024 is reduced in dimension, which is divided into 32 tensors with a size of 20x16x32 along the channel direction, and each tensor with a size of 20x16x32 is averaged to get a real number. In this way, the tensor with a size of 20x16x32 is reduced to a feature vector with a length of 32, and the specific calculation formula is:

[0067]

[0068]

[0069] wherein, and are the infrared feature tensor and the visible feature tensor of the i-th infrared visible image, respectively, with the corresponding size of 20x16x1024; and are the infrared feature vector and the visible feature vector of the i-th infrared visible image, respectively, with the length of 32; j represents the j-th component of the feature vector.

[0070] 3) A linear transformation is performed on the feature vector with the size of 32 for normalization, and the value of each component of the feature vector is normalized to the value range of 0 to 1, and the corresponding calculation method is:

[0071]

[0072]

[0073] wherein, and represent the infrared feature vector and the visible feature vector of the j-th component of the i-th image, respectively; and are the calculation maximum and minimum operations;

[0074] 4) Statistics are performed on the feature vectors calculated above and ; Since the value of each component is in the range of 0 to 1, the continuous value is discretely quantized into 40 intervals, and the length of each interval is ; When the value of a component of the feature vector is divided by 0.025 and rounded down, that is, multiplied by 40 and rounded down, the integer obtained by this method represents the interval number of the quantized infrared feature component, and the interval number is 40, which is 0, 1, 2, …, 38, 39 (in addition, when the value of the feature component is exactly 1, it directly corresponds to the 39th interval).

[0075] 5) 32 matrices with the size of 40x40 are constructed, and each feature component corresponds to a matrix; for each infrared image and the corresponding visible image, if the interval numbers of the corresponding components are i and j, respectively, the value of the i-th row and the j-th column of the matrix will be subjected to a "+1" operation; after each component of the feature vector in all infrared images and visible images in the training set is executed, the 32 matrices with the size of 40x40 can be approximately represented as the two-dimensional frequency distribution (the two-dimensional probability distribution can be obtained after normalization) between the infrared feature and the visible feature, and the corresponding calculation method is:

[0076]

[0077]

[0078]

[0079] where represents the element in the jth matrix, the ith row, and the jth column, represents 1 when it is true, otherwise 0; represents the i-th infrared image feature component, represents the i-th visible light image feature component; is a down rounding operation; finally, divide each element in the matrix by the sum of all elements in the row to obtain the conditional probability matrix:

[0080]

[0081] where is a conditional co-occurrence matrix, is the value of the mth row and the nth column of the jth conditional co-occurrence matrix.

[0082] 2. Implementation details of the conditional probability distribution transfer module

[0083] This module aims to train a YOLOv3 target detection network using infrared images, and for each infrared image, predict the probability distribution of the corresponding visible light feature, i.e. the conditional probability distribution. The specific implementation details of this module are as follows:

[0084] 1) First, for each infrared image in the training set, obtain the true conditional distribution of the corresponding reduced dimension feature by querying the conditional co-occurrence matrix; and each infrared image corresponds to 32 conditional distributions. For each distribution, the previous calculation is , then the corresponding length-40 vector represents the corresponding conditional probability distribution.

[0085] 2) Calculate the corresponding mean and variance according to the conditional probability distribution obtained in the previous step; for a row vector with a length of 40, the corresponding index is 0 to 39. The vector approximately represents a discrete random variable, which can take values , , ; in addition, the value of the 0th index of the vector represents the probability of taking the value , the value of the 1st index of the vector represents the probability of taking the value , and so on. The value of the 39th index of the vector represents the probability of taking the value the probability of the condition; according to the calculation method of the mean and variance of the discrete random variable, the calculation method of the mean vector and variance vector of the conditional probability distribution is obtained:

[0086]

[0087]

[0088] wherein, represents the mean vector, and the length is 32; represents the variance vector, and the length is also 32.

[0089] 3) Then, the mean vector and variance vector of the conditional probability distribution calculated in the last step are taken as the real labels of the auxiliary task; next, a conditional probability prediction network is constructed to predict the mean vector and variance vector constructed before; the input of the conditional probability prediction network is the infrared feature tensor output by the feature extraction network, and the size is ; the output of the conditional probability prediction network is two vectors with a length of 32.

[0090] For the feature tensor with a size of , first, a convolution operation is performed through a convolution kernel with a size of , so that the size of the feature becomes ; then, a rectified linear unit (ReLU) activation function is used, at this time, the size of the feature remains unchanged, still ; then, a maximum pooling operation with a size of is performed, so that the size becomes , and the calculated feature is reshaped into a vector with a length of ; then, the vector is passed through two multi-layer perceptrons (MLPs) with the same structure but different parameter values, the structure of the MLP is a fully connected layer that converts a vector with a length of 10240 into a vector with a length of 256; then, a rectified linear unit (ReLU) activation function is used, and a dropout layer (p=0.5) is connected; finally, a fully connected layer is used to convert a vector with a length of 256 into a vector with a length of 32.

[0091] 4) The loss is obtained by comparing the difference between the predicted mean and variance vectors and the real mean and variance vectors, and the loss is weighted and summed with the detection loss:

[0092]

[0093] wherein, is the loss function of the YOLOv3 target detection model, including the loss of the regression frame and the classification loss; and are the predicted and real mean vectors, respectively, and are the predicted and true variance vectors, respectively, is the mean squared error.

[0094] 3. Details of the visible light assisted infrared target detection network combined with YOLOv3

[0095] YOLOv3 is a classic single-stage target detection algorithm that balances the detection accuracy and inference speed of target detection.

[0096] 1) The neck network of YOLOv3 mainly processes three feature layers output by the feature extraction network Darknet53, which are located at the middle layer, the middle-lower layer and the bottom layer of Darknet53, and the sizes of the three feature layers are (80, 64, 256), (40, 32, 512) and (20, 16, 1024) respectively; FPN layers are constructed using the three effective feature layers, and the construction method is as follows: the 20x16x1024 feature layer is processed by 5 times of convolution, and after processing, the YoloHead is used to obtain the prediction result, part of which is used for up-sampling UmSampling2d and combined with the 40x32x512 feature layer, and the size of the combined feature layer is (40, 32, 768); the combined feature layer is processed by 5 times of convolution again, and after processing, the YoloHead is used to obtain the prediction result, part of which is used for up-sampling UmSampling2d and combined with the 80x64x256 feature layer, and the size of the combined feature layer is (80, 64, 384); the combined feature layer is processed by 5 times of convolution again, and after processing, the YoloHead is used to obtain the prediction result.

[0097] 2) The purpose of the detection head YoloHead of the YOLOv3 target detection model is to predict the target corresponding to each image according to the features provided by the neck network, including the coordinate frame and the category of the target; using the FPN feature pyramid, three enhanced features can be obtained, and the shapes of the three enhanced features are (20, 16, 512), (40, 32, 256) and (80, 64, 128) respectively, and then the feature layers with these three shapes are transmitted into the Yolo Head to obtain the prediction result.

[0098] Yolo Head is essentially a 3x3 convolution followed by a 1x1 convolution, and the role of 3x3 convolution is feature integration, and the role of 1x1 convolution is to adjust the number of channels.

[0099] The three feature layers are processed respectively, assuming that the VOC dataset is predicted, and the shape of the output layer of is (20, 16, 75), (40, 32, 75), and (80, 64, 75). The last dimension is 75 because the figure is based on the VOC dataset, which has 20 classes. YOLOv3 has 3 prior boxes for each feature point of each feature layer, so the channel number of the prediction result is 3x25.

[0100] If the coco training set is used, the class is 80, and the last dimension should be 255 = 3x85. The shape of the three feature layers is (20, 16, 255), (40, 32, 255), and (80, 64, 255). In fact, N 640x512 images are input, and after multiple layers of operation, three data with shapes of (N, 20, 16, 255), (N, 40, 32, 255), and (N, 80, 64, 255) are output, corresponding to the positions of 3 prior boxes in 20x16, 40x32, and 80x64 grids of each image.

[0101] 3) The decoding process of the YOLOv3 target detection model is as follows: taking the COCO dataset as an example, for the feature with a size of (20, 16, 255) output by the detection head, the entire image is divided into 20x16 grids; then a plurality of prior boxes are established from the center of each grid. These boxes are pre-set boxes by the network, and the network's prediction result will determine whether these boxes contain objects and the type of the object. For each box, x_offset, y_offset, h, and w, confidence, and classification result are represented. However, this prediction result does not correspond to the position of the final prediction box on the image, and decoding is needed to complete it. Decoding is completed in two steps: first, add the corresponding x_offset and y_offset to each grid point. The result after addition is the center of the prediction box. Then, the width and height of the corresponding prediction box are calculated using the prior box and h and w, so that the position of the entire prediction box can be obtained. After obtaining the final prediction result, score sorting and non-maximum suppression screening are performed.

[0102] 4) The loss function calculation process of the YOLOv3 target detection model is as follows: determine the position of the real box in the image, determine which grid point it belongs to for detection, determine which prior box of the real box and the feature point has the highest overlap, and calculate how the prediction result of the grid point should be to obtain the real box. The prior box with the highest overlap with the real box is used as the positive sample.

[0103] According to the prediction result of the network, a prediction frame is obtained, the coincidence degree of the prediction frame and all real frames is calculated, and if the coincidence degree is greater than a certain threshold, the prior frame corresponding to the prediction frame is ignored; the rest is regarded as a negative sample.

[0104] The final loss is composed of three parts: positive samples, the difference between the encoded length and width and the predicted value of the xy axis offset; positive samples, the value of the confidence in the prediction result compared with 1, negative samples, the value of the confidence in the prediction result compared with 0; the actual existing frame, the comparison between the category prediction result and the actual result.

[0105] 5) The YOLOv3 target detection algorithm used in the embodiment adopts a feature enhancement scheme under the CBAM attention mechanism. CBAM combines the channel attention mechanism and the spatial attention mechanism. The implementation of the channel attention mechanism can be divided into two parts. For a single feature layer input, global average pooling and global maximum pooling are performed respectively. Then the results of average pooling and maximum pooling are processed using a shared fully connected layer, and the two processed results are added together, and then a sigmoid is taken. At this time, the weight (0-1) of each channel of the input feature layer is obtained. After obtaining the weight, multiply the original input feature layer by the weight. For the spatial attention mechanism, the maximum value and the average value are taken on the channel of each feature point of the input feature layer. Then stack the two results, adjust the channel number using a convolution with a channel number of 1, and then take a sigmoid. At this time, the weight (0-1) of each feature point of the input feature layer is obtained. After obtaining the weight, multiply the original input feature layer by the weight.

[0106] It should be noted that within the scope of protection defined in the claims of the present application, the following embodiments can be combined and / or extended, replaced, in any logical manner from the above specific embodiments, such as disclosed technical principles, disclosed technical features or implied disclosed technical features.

[0107] Example 1

[0108] A visible light assisted infrared target detection method, comprising the following steps:

[0109] A feature co-occurrence matrix is constructed to approximately represent the two-dimensional probability distribution between infrared features and visible light features; and a conditional probability distribution migration module is constructed to enable the feature extraction network to have the ability to extract visible light features, and to use visible light images to assist infrared images for target detection;

[0110] The visible light assisted infrared target detection model combined with the target detection algorithm is constructed based on the feature co-occurrence matrix and the conditional probability distribution migration module, and the target corresponding to the image is predicted through the feature extraction network and the target detection algorithm.

[0111] Embodiment 2

[0112] On the basis of embodiment 1, the feature co-occurrence matrix is constructed, and the specific steps include the following sub-steps:

[0113] S11, respectively training a feature extraction network of a visible light target detection model and a feature extraction network of an infrared target detection model;

[0114] S12, for each visible light infrared image in the training set, the corresponding visible light feature tensor and infrared feature tensor are obtained through the corresponding feature extraction network;

[0115] S13, the visible light feature tensor and the infrared feature tensor are reduced in dimension, so as to obtain the visible light feature vector and the infrared feature vector respectively;

[0116] S14, performing normalization operation on the feature vector, so that the value is changed to the value range of 0 to 1;

[0117] S15, respectively performing discrete sampling statistics on each component of the feature vector to obtain the feature co-occurrence matrix.

[0118] Embodiment 3

[0119] On the basis of embodiment 1, the conditional probability distribution migration module is constructed, and the specific steps include the following sub-steps:

[0120] S21, obtaining the corresponding feature tensor through the feature extraction network of the target detection model for each infrared image;

[0121] S22, inputting the feature tensor into the conditional probability prediction network to predict the corresponding mean vector and variance vector to obtain the loss;

[0122] S23, inputting the feature tensor into the neck network and the detection head network to obtain the predicted target detection result, and calculating the loss;

[0123] S24, adding the loss in step S22 and the loss in step S23 to optimize the entire infrared target detection model.

[0124] Embodiment 4

[0125] On the basis of embodiment 1, the visible light assisted infrared target detection model combined with the target detection algorithm is constructed based on the feature co-occurrence matrix and the conditional probability distribution migration module, and the specific steps include the following sub-steps:

[0126] S31, in the model training stage, the parameter update of the feature extraction network is completed through the feature co-occurrence matrix and the conditional probability distribution migration module;

[0127] S32, in the inference stage, the process is consistent with the target detection algorithm, but only the infrared image needs to be input, and the visible light image does not need to be input, and the target corresponding to each image is predicted through the decoding of the feature extraction network and the target detection model.

[0128] Embodiment 5

[0129] On the basis of embodiment 1, the target detection algorithm comprises a YOLOv3 target detection algorithm.

[0130] Embodiment 6

[0131] On the basis of embodiment 1, the target corresponding to the predicted image comprises a coordinate frame and a category of the target.

[0132] Embodiment 7

[0133] On the basis of embodiment 1, in step S15, the number of the constructed feature co-occurrence matrix is the same as the length of the dimension-reduced feature vector in step S13.

[0134] Embodiment 8

[0135] A visible light assisted infrared target detection device comprises:

[0136] A feature co-occurrence matrix construction module is configured to perform the following process: first, a feature extraction network of a visible light target detection model and a feature extraction network of an infrared target detection model are trained respectively; second, for each visible light infrared image in a training set, corresponding visible light feature tensors and infrared feature tensors are obtained through the corresponding feature extraction networks; third, the visible light feature tensors and the infrared feature tensors are dimension-reduced to obtain visible light feature vectors and infrared feature vectors respectively; fourth, the feature vectors are normalized to change the values to a value range of 0 to 1; fifth, each component of the feature vectors is discretely sampled and counted to obtain a feature co-occurrence matrix.

[0137] A conditional probability distribution migration module is configured to perform the following process: first, each infrared image is input into the feature extraction network of the target detection model to obtain a corresponding feature tensor; second, the feature tensor is input into a conditional probability prediction network to obtain a loss by predicting a mean vector and a variance vector; third, the feature tensor is input into a neck network and a detection head network to obtain a predicted target detection result and calculate a loss; and finally, the two losses are added to optimize the entire infrared target detection model.

[0138] Embodiment 9

[0139] On the basis of embodiment 8, a visible light assisted infrared target detection network construction module combined with a target detection algorithm is further included, which is configured to predict a target corresponding to an image through a feature extraction network and a target detection algorithm.

[0140] Embodiment 10

[0141] A computer readable storage medium, in which a computer program is stored, the computer program is loaded and executed by a processor to perform the method of any one of embodiments 1-7.

[0142] The units described in the embodiments of the present application can be implemented by software, or by hardware, or by a combination of software and hardware. The units described can also be located in a single processor. In some cases, the names of the units do not constitute a limitation on the units themselves.

[0143] According to an aspect of the embodiments of the present application, a computer program product or computer program is provided, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device performs the method provided in the various optional implementation manners described above.

[0144] As another aspect, the embodiments of the present application further provide a computer readable medium, which can be included in the electronic device described in the above embodiments, or can exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs, which, when executed by the electronic device, enable the electronic device to implement the method described in the above embodiments.

Claims

1. A visible light-assisted infrared target detection method, characterized in that, Includes the following steps: A feature co-occurrence matrix is ​​constructed to approximate the two-dimensional probability distribution between infrared and visible light features; and a conditional probability distribution transfer module is constructed to enable the feature extraction network to extract visible light features, thereby using visible light images to assist infrared images in target detection. Based on the aforementioned feature co-occurrence matrix and conditional probability distribution transfer module, a visible light-assisted infrared target detection model combining target detection algorithms is constructed. The target corresponding to the image is predicted through a feature extraction network and a target detection algorithm. The construction of the feature co-occurrence matrix specifically includes the following sub-steps: S11, train a feature extraction network for a visible light target detection model and a feature extraction network for infrared target detection respectively; S12, For each visible light infrared image in the training set, the corresponding visible light feature tensor and infrared feature tensor are obtained through the corresponding feature extraction network; S13, reduce the dimensionality of the visible light feature tensor and the infrared feature tensor to obtain the visible light feature vector and the infrared feature vector respectively; S14, normalize the feature vector to change the value to the range of 0 to 1; S15, Discretize and sample each component of the feature vector to obtain the feature co-occurrence matrix; The construction of the conditional probability distribution transition module specifically includes the following sub-steps: S21, obtain the corresponding feature tensor for each infrared image through the feature extraction network of the target detection model; S22, the feature tensor is fed into the conditional probability prediction network to predict the corresponding mean vector and variance vector to obtain the loss; S23, the feature tensor is fed into the neck network and the detection head network to obtain the predicted target detection results, and the loss is calculated; S24. Add the losses from steps S22 and S23 to optimize the entire infrared target detection model.

2. The visible light-assisted infrared target detection method according to claim 1, characterized in that, The construction of a visible light-assisted infrared target detection model based on the feature co-occurrence matrix and conditional probability distribution transfer module, combined with the target detection algorithm, specifically includes the following sub-steps: S31, During the model training phase, the parameters of the feature extraction network are updated through the feature co-occurrence matrix and the conditional probability distribution transfer module. S32 follows the same process as the target detection algorithm in the inference stage, but only requires infrared images as input, not visible light images. It predicts the target corresponding to each image through the feature extraction network and the decoding of the target detection model.

3. The visible light-assisted infrared target detection method according to claim 1, characterized in that, The target detection algorithm includes the YOLOv3 target detection algorithm.

4. The visible light-assisted infrared target detection method according to claim 1, characterized in that, The target corresponding to the predicted image includes a bounding box and the target category.

5. The visible light-assisted infrared target detection method according to claim 1, characterized in that, In step S15, the number of feature co-occurrence matrices constructed is the same as the length of the eigenvectors after dimensionality reduction in step S13.

6. A visible light-assisted infrared target detection device, characterized in that, include: The feature co-occurrence matrix construction module is used to perform the following process: First, train the feature extraction network of a visible light target detection model and the feature extraction network of an infrared target detection model respectively; second, for each visible light and infrared image in the training set, obtain the corresponding visible light feature tensor and infrared feature tensor through the corresponding feature extraction network respectively. Next, the visible light feature tensor and the infrared feature tensor are reduced in dimensionality to obtain the visible light feature vector and the infrared feature vector, respectively. Then, the feature vectors are normalized to reduce the values ​​to the range of 0 to 1. Finally, each component of the feature vector is discretized and sampled statistically to obtain the feature co-occurrence matrix. The conditional probability distribution transfer module is used to perform the following process: First, each infrared image is processed through the feature extraction network of the target detection model to obtain the corresponding feature tensor; second, the feature tensor is fed into the conditional probability prediction network to predict the corresponding mean vector and variance vector to obtain the loss; third, the feature tensor is fed into the neck network and the detection head network to obtain the predicted target detection result and calculate the loss; finally, the two losses are added together to optimize the entire infrared target detection model. A visible light-assisted infrared target detection network construction module, which combines target detection algorithms, is used to predict the target corresponding to the image through feature extraction network and target detection algorithm.

7. A computer-readable storage medium, characterized in that, A computer program is stored in a readable storage medium, the computer program being loaded by a processor and executing the method as described in any one of claims 1 to 5.

Images