A temperature and brightness feature extraction method based on deep fusion neural network

Abstract

A method for extracting temperature and brightness features based on a deep fusion neural network, the specific method is: 1) collecting sample data; 2) generating simulation data; 3) extracting the target shape outline; 4) constructing a deep fusion neural network feature model; 5) extracting the temperature and brightness features of the test sample. The present invention can efficiently construct a brightness and temperature feature extraction model for weak targets in complex natural environments, provide technical reserves for long-term monitoring of the temperature and brightness features of camouflage feature test samples in typical harsh natural environments, and is conducive to the study of the degradation laws of functional indicators such as temperature and brightness of test samples, environmental adaptability evaluation, and service life evaluation in typical environments.

Description

Technical Field

[0001] The present invention relates to the field of image feature extraction, and in particular to a method for extracting temperature and brightness features based on a deep fusion neural network. Background Art

[0002] The extraction of temperature, brightness, backscattering and other features of complex natural scenes is a necessary prerequisite for implementing object classification, segmentation and target detection and recognition, tracking and guidance, and target and natural background fusion analysis. Driven by the application of foreign countries in land and resources development, environmental adaptability research of military facilities, etc., it leads the development of feature extraction algorithms. Foreign feature extraction methods are mainly divided into three categories: spectral feature-oriented, distribution feature-oriented, and multi-source heterogeneous big data learning feature-oriented. The first category of commonly used methods includes spectral derivatives and absorption index, spectral curvature, orthogonal projection divergence method, etc. This type of method mainly extracts spectral absorption or reflection features that can reflect the diagnostic properties of the target by analyzing the radiation characteristics of multiple spectral bands of the target. The absorption index method extracts the radiation characteristics of the target through algebraic operations of several bands. Although this method takes into account the element absorption band of the target, it selects the element absorption band based on artificial experience, and the extracted features are usually not comprehensive. Although the spectral derivative method can extract the radiation characteristics of the target at different scales, it only extracts features of different scales by differential means, which is too simple. The second category is feature extraction methods oriented to feature distribution, such as strong scattering source distribution, strong radiation source distribution, and strong reflection source distribution. This type of method mainly transforms the data in the high-dimensional feature space into the low-dimensional feature space through the extraction and selection of feature space, and at the same time highlights the target structure of interest. Typical methods include principal component analysis PCA, local preservation projection LPP, non-negative matrix decomposition NMF, minimum noise separation MNF, and classic methods such as NPE, LDA, LPP, and NWFE. The third type of method is to learn features for multi-source heterogeneous big data. With the great success of machine learning in the field of image target recognition, many neural network models with excellent performance have been proposed. According to feature images collected in different bands, different distances, different backgrounds, different angles, and different times, neural network deep training can be performed to output feature vectors representing target attributes, which greatly improves the automation and high-precision development of feature extraction algorithms. Typical algorithms include envelope convolutional neural network CNN and residual neural network ResNet.

[0003] Based on foreign theories, China has proposed corresponding correction methods for spectral features, distribution features, and multi-source data learning features, so as to catch up with foreign technologies. In particular, a large number of neural network models have emerged in the field of machine learning, such as the proposed end-to-end feature extraction method based on convolutional neural networks. For aircraft targets of different scales in complex ground scenes, a lightweight feature extraction network is used as the basic network, and a multi-scale feature fusion method is designed. Through jump connections, high-level semantic features and low-level detail features are fused to enhance the structural information of features and support the accurate detection of ground aircraft targets. A remote sensing image feature extraction method combining saliency maps and machine learning is proposed. For ground aircraft targets with different background environments and resolutions, the HC algorithm and Ostu algorithm are first used to roughly locate the target and mark the candidate target, and then multiple features of the candidate target are extracted and fused, which supports the recognition of ground aircraft targets based on infrared features at different resolutions at different airports.

[0004] However, there are prominent problems such as the mutual limitations between the accuracy of feature extraction and the complexity of neural networks and the time complexity of network training, which lead to low automation and informatization levels of feature extraction and low accuracy of feature extraction. Based on the development gap between domestic and foreign feature extraction algorithms for temperature, brightness and other features, as well as the defects of current domestic and foreign feature extraction methods in not comprehensively considering various influencing factors in complex environments and insufficient pertinence in neural network construction, the present invention proposes an automatic extraction method for temperature and brightness camouflage features based on a deep fusion neural network, which realizes self-organization, self-learning and self-optimization of temperature and brightness features of targets of interest in complex natural scenes, greatly improving the accuracy and time efficiency of feature extraction. Summary of the invention

[0005] The purpose of the present invention is to provide a temperature and brightness feature extraction method based on a deep fusion neural network, which can construct a feature extraction model for micro-reflection and weak radiation targets in complex natural environments, and improve the ability to automatically extract weak features in target and background fusion scenarios.

[0006] The object of the present invention is achieved through such a technical solution, and the specific method is as follows:

[0007] 1) Collect sample data: obtain image information of the test sample in the natural environment, as well as the temperature and brightness information of the test sample;

[0008] 2) Generate simulation data: add noise information to the image information of the test sample and convert the posture of the test sample as simulation data;

[0009] 3) Extract the target shape contour: Use the envelope window extraction method to determine the target shape contour information of the test sample in the sample data and simulation data;

[0010] 4) Constructing a deep fusion neural network feature model: The extracted target shape profile information and the temperature and brightness information of the test sample are input into the deep fusion neural network feature model;

[0011] 5) Extracting the temperature and brightness characteristics of the test sample: inputting the image data of the test sample data to be extracted into the deep fusion neural network feature model, and outputting the temperature and brightness characteristics of the test sample to be extracted.

[0012] Further, the specific method of extracting the target outline in step 3) is as follows:

[0013] 3-1) Based on the image spatial resolution and target spatial size information, design an envelope constraint window of the target outline;

[0014] 3-2) In the window, for each pixel, the cumulative distance D from the pixel to other pixels in the window is calculated, and the pixel with the minimum D value is taken as the pixel with the greatest similarity to the target. The formula is as follows:

[0015] D = {∑distance[f(x i ),f(x j |)x i ,x j ∈W}

[0016] Where W is the range of the moving operation window, and distance represents the distance calculation. The specific calculation method is:

[0017] distance(x i ,x j )=(x i T Px j ⊥ x i +x j T Px i ⊥ x j ) 1 / 2

[0018]

[0019] Where T represents the transpose operation of the matrix, N = i, j and I L×L is the L×L dimensional identity matrix;

[0020] 3-3) Move the operation window pixel by pixel until the calculation of the entire image is completed, and obtain the pixel position with the greatest similarity to the target in the entire image.

[0021] Further, the specific method of constructing the deep fusion neural network feature model in step 4) is as follows:

[0022] 4-1) Construct a multi-scale convolutional neural network model;

[0023] 4-2) Construct a deep residual neural network model;

[0024] 4-3) A fusion strategy of taking the mean is adopted for the multi-scale convolutional neural network model and the deep residual neural network model to construct a deep fusion neural network feature model.

[0025] Further, the specific method of constructing the multi-scale convolutional neural network model in step 4-1) is as follows:

[0026] 4-1-1) Set three types of convolution kernels: 1×1, 3×3, and 5×5, and use group convolution;

[0027] 4-1-2) Use Leaky RELU nonlinear activation function;

[0028] 4-1-3) Connect a 3×3 pooling operation with a stride of 2 after the convolutional layer and use Dropout in higher convolutional layers;

[0029] 4-1-4) Construct a multi-scale convolutional neural network structure consisting of 12 convolutional layers and 4 maximum pooling layers.

[0030] Further, the specific method of constructing the deep residual neural network model in step 4-2) is as follows:

[0031] 4-2-1) Setting the residual module unit of the multi-layer convolutional layer;

[0032] 4-2-2) After each convolution process and before the activation function, batch regularization technology is used to accelerate the convergence of the entire function;

[0033] 4-2-3) After the first and second convolutions and batch normalization, insert the activation function Leaky ReLU;

[0034] 4-2-4) After the third convolution and batch normalization, add the input of the identity mapping and activate it again to get the output and build a basic residual unit:

[0035] y=σ(W3σ(W2σ(W1x))+W s x)

[0036] In the formula, σ is the activation function Leaky ReLU, and W is the network weight;

[0037] 4-2-5) Construct a deep residual neural network structure consisting of 1 convolutional layer, 1 maximum pooling layer, 1 averaging layer, and 11 residual learning modules.

[0038] Due to the adoption of the above technical solution, the present invention has the following advantages:

[0039] The present invention can efficiently construct a brightness and temperature feature extraction model for weak targets in complex natural environments, and provide a technical reserve for long-term monitoring of temperature and brightness characteristics of camouflage feature test samples in typical harsh natural environments, which is beneficial to the study of degradation laws of functional indicators such as temperature and brightness of test samples, environmental adaptability evaluation and service life assessment in typical environments.

[0040] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings of the present invention are as follows.

[0042] Figure 1 It is a multi-scale convolutional neural network model diagram;

[0043] Figure 2 It is a deep residual neural network model diagram;

[0044] Figure 3 This is a diagram of the deep fusion neural network feature extraction model. DETAILED DESCRIPTION

[0045] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0046] A temperature and brightness feature extraction method based on a deep fusion neural network, the specific implementation method is as follows:

[0047] 1. Input the temperature and brightness characteristic data of environmental sample pieces collected at different distances, times, angles and backgrounds.

[0048] 2. Use multi-level data fusion and data enhancement techniques such as sample posture transformation and noise addition to obtain more data sets.

[0049] 3. A target position initialization extraction method based on envelope window constraints is used to label more than 90% of the randomly sampled data.

[0050] 4. Network weight initialization: Deep networks with ReLU / Leaky RELU nonlinearity should be initialized with a zero-mean Gaussian distribution with a standard deviation of Where n is the number of inputs to each neuron.

[0051] 5. Minimize the loss function: Use small batch gradient descent and momentum algorithm.

[0052] 6. Learning rate setting: The initial value is set to 0.001. As the number of network iteration training increases, the learning rate decreases.

[0053] 7. Input the data into the deep fusion neural network feature extraction model for training.

[0054] 8. The feature vector output by the network is the temperature and brightness extraction result, which can be used as input for target detection and object classification.

[0055] The target position initialization extraction method based on envelope window constraints is:

[0056] a) Based on the image spatial resolution and target spatial size information, an envelope constraint window of the target outline is designed.

[0057] b) In the window, for each pixel, the cumulative distance D from the pixel to other pixels in the window is calculated, and the pixel with the minimum value of D is taken as the pixel with the greatest similarity to the target.

[0058] D = {∑distance[f(x i ),f(x j )|x i ,x j ∈W} (1)

[0059] Where W is the range of the moving operation window, and distance represents the distance calculation. The specific calculation method is shown in formula (2).

[0060] distance(x i ,x j )=(x i T Px j ⊥ x i +x j T Px i ⊥ x j ) 1 / 2 (2)

[0061]

[0062] Where T represents the transpose operation of the matrix, N = i, j and I L×L is the L×L dimensional identity matrix.

[0063] c) Move the operation window pixel by pixel until the calculation of the entire image is completed, and obtain the pixel position with the greatest similarity to the target in the entire image.

[0064] The specific method of constructing a multi-scale convolutional neural network model is as follows:

[0065] a) Setting three types of convolution kernels (1×1, 3×3, and 5×5) and adopting group convolution can improve the multi-scale feature extraction capability and keep the network lightweight to a certain extent.

[0066] b) Add a 1×1 Bottleneck in the convolution operation to reduce the large number of additional parameters introduced by multi-sized convolution kernels.

[0067] c) Use Leaky RELU nonlinear activation function to prevent gradient diffusion and reduce the time complexity of network training.

[0068] d) Connect a 3×3 pooling operation with a stride of 2 after the convolutional layer and use Dropout in higher convolutional layers to minimize overfitting of the network during parameter optimization.

[0069] e) Construct a multi-scale convolutional neural network structure consisting of 12 convolutional layers and 4 maximum pooling layers, see Appendix Figure 1 .

[0070] The specific method of constructing a deep residual neural network model is as follows:

[0071] a) Setting up residual module units of multi-layer convolutional layers can improve the compression capability of the model in transmitting data and the nonlinear expression capability of the model.

[0072] b) After each convolution process and before the activation function, batch regularization technology is used to accelerate the convergence of the entire function.

[0073] c) After the first and second convolutions and batch normalization, insert the activation function Leaky ReLU;

[0074] d) After the third convolution and batch normalization, the input of the identity mapping is added for activation to finally obtain the output and construct a basic residual unit.

[0075] y=σ(W3σ(W2σ(W1x))+W s x) (4)

[0076] Where:

[0077] σ — activation function Leaky ReLU;

[0078] W—Network weight.

[0079] e) Construct a deep residual neural network structure consisting of 1 convolutional layer, 1 maximum pooling layer, 1 averaging layer, and 11 residual learning modules, see Appendix Figure 2 .

[0080] The specific method of constructing a deep fusion neural network feature model is as follows:

[0081] a) Construct a parallel network structure of multi-size convolutional neural network and deep residual neural network, see the attached Figure 3 .

[0082] b) Perform weight initialization, loss function minimization, and learning rate reduction as the number of network iterations increases.

[0083] c) A mean fusion strategy is adopted for the outputs of the multi-scale convolutional neural network model and the deep residual neural network model to obtain a deep fusion result of the two networks.

[0084] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0085] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0086] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0087] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A temperature and brightness feature extraction method based on deep fusion neural network, characterized in that: The specific method is as follows: 1) Collect sample data: obtain image information of the test sample in the natural environment, as well as the temperature and brightness information of the test sample; 2) Generate simulation data: add noise information to the image information of the test sample and convert the posture of the test sample as simulation data; 3) Extract the target shape contour: Use the envelope window extraction method to determine the target shape contour information of the test sample in the sample data and simulation data; 4) Constructing a deep fusion neural network feature model: The extracted target shape contour information and the temperature and brightness information of the test sample are input into the deep fusion neural network feature model; 5) Extracting the temperature and brightness characteristics of the test sample: inputting the image data of the test sample data to be extracted into the deep fusion neural network feature model, and outputting the temperature and brightness characteristics of the test sample to be extracted.

2. The temperature and brightness feature extraction method based on deep fusion neural network according to claim 1 is characterized in that: The specific method of constructing the deep fusion neural network feature model in step 4) is as follows: 4-1) Construct a multi-scale convolutional neural network model; 4-2) Construct a deep residual neural network model; 4-3) A fusion strategy of taking the mean is adopted for the multi-scale convolutional neural network model and the deep residual neural network model to construct a deep fusion neural network feature model.

3. The temperature and brightness feature extraction method based on deep fusion neural network according to claim 2 is characterized in that: The specific method of constructing the multi-scale convolutional neural network model in step 4-1) is as follows: 4-1-1) Set three types of convolution kernels: 1×1, 3×3, and 5×5, and use group convolution; 4-1-2) Use Leaky RELU nonlinear activation function; 4-1-3) Connect a 3×3 pooling operation with a stride of 2 after the convolutional layer and use Dropout in higher convolutional layers; 4-1-4) Construct a multi-scale convolutional neural network structure consisting of 12 convolutional layers and 4 maximum pooling layers.

Images