Remote sensing information processing method based on deep learning training

Abstract

The invention discloses a remote sensing information processing method based on deep learning training. The remote sensing information processing method comprises the following steps: step 1, rapidly screening wide visible light marine targets based on target saliency analysis and a deep learning model; step 2, fast extraction of a near-shore port ship based on a target knowledge base and small sample machine learning; and step 3, mining massive SAR data features based on the sparse deep belief network. According to the characteristic of wide visible light of a sensor, the method for automatically detecting the visible light marine large ship target is constructed by fully utilizing the newest achievements of computer vision and machine learning, and rapid extraction of the marine ship target is realized. And the ship target is rapidly extracted from the high-resolution image, so that an interpreter can be helped to rapidly interpret and acquire data information.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing survey technology, and in particular to a remote sensing information processing method based on deep learning training. Background Art

[0002] With the rapid development of high-altitude survey technology, the sensor resources available to operating units are becoming increasingly abundant. The massive data provided by the sensor series will greatly enhance the operating unit's ability to survey a wide range of marine ship targets, and will inevitably make high-altitude survey an important means to improve the information-based combat capabilities of operating units. Driven by the visible light, infrared, and hyperspectral data application capabilities of a certain sensor, the results of the integrated construction of the main operating environment sensor series are integrated, and combined with the comprehensive needs of operating units for sensor information applications, the data processing capabilities of ocean observation sensors are improved to solve the problems of insufficient massive remote sensing data processing capabilities and weak data organization and management capabilities of operating units. Modern high-performance computing technology, massive data storage and management technology, and parallel remote sensing image processing algorithms and software are fully utilized to build a scientific, reasonable, and seamlessly scalable integrated system architecture to achieve a remote sensing information processing platform that integrates flexible and effective data storage management and high-performance computing.

[0003] Based on the experience and foundation of high-altitude survey and observation research and development in the main operating environment, according to the characteristics of marine remote sensing data and the requirements of the operating environment, the processing and fusion of optical, SAR, infrared, hyperspectral and other image data and electronic survey data are realized to make full use of domestic, civilian and foreign commercial sensor resources, form a comprehensive guarantee capability for nearshore and offshore targets based on sensor information, and meet the operating units' high-efficiency and high-precision remote sensing data in the new operating environment.

[0004] In recent years, with the gradual maturity of my country's remote sensing sensor network, remote sensing sensors have accumulated massive amounts of remote sensing data in hot spots. At the same time, other remote sensing data sources, such as commercial remote sensing sensors, have also accumulated considerable amounts of data. The existing remote sensing data processing systems are lacking in the ability to process massive amounts of remote sensing data and lack effective data organization and management models. The information mining capabilities of multi-source heterogeneous data still need to be improved, which has become a bottleneck problem currently faced by operating units in remote sensing information processing and cannot meet the needs of combat data extraction in the operating environment under the new situation. Summary of the invention

[0005] The purpose of the present invention is to provide a remote sensing information processing method based on deep learning training to solve the technical problems existing in the prior art.

[0006] To achieve the above object, the method of the present invention comprises the following steps:

[0007] Step 1: Rapid screening of wide-band visible light marine targets based on target saliency analysis and deep learning model: First, the suspected areas of ship targets in the wide-band visible light image are quickly extracted using the saliency determination method in the computer. These areas are all used as candidate targets for the next step of processing; then, a deep convolutional network is constructed for the candidate targets to determine the ship targets, thereby finally realizing the extraction of marine ship targets;

[0008] Step 2: Rapid extraction of nearshore port ships based on target knowledge base and small sample machine learning. Rapid extraction of ship targets from high-resolution images, assisting the reader to quickly interpret and obtain data information, using the port target knowledge base to extract the areas where ship targets often dock, further extracting the suspected target areas, and finally using machine learning methods to classify and screen ship targets;

[0009] Step 3: Feature mining of massive SAR data based on sparse deep belief networks. Restricted Boltzmann machines are used to perform layer-by-layer unsupervised feature learning on a large amount of training data. The learning process uses the contrast divergence algorithm based on Gibbs sampling to solve the parameters. The parameters solved by the multi-layer restricted Boltzmann machine are then used as the initial values ​​of the multi-layer neural network. The model parameters of the multi-layer neural network are then fine-tuned using the labeled training data, and the multinomial logistic regression model is used as the final classification model. The fine-tuning process uses the back-propagation algorithm and the conjugate gradient descent method. The fine-tuned new data is forward propagated through the network to obtain the final feature representation result.

[0010] Furthermore, step 1 further includes: (1) preliminary screening of suspected targets based on target saliency analysis, extracting edge information of ship targets at sea with different scales and different navigation directions to enhance the difference between the target and the background, thereby facilitating the separation of the ship target from the sea background.

[0011] Furthermore, the initial screening of suspected targets based on target saliency analysis further includes: enhancing the ship targets of different sizes on the high-resolution optical image by using a series of scale LoG filter convolutions, and performing initial segmentation using mean-shift to obtain an object set {obj i}, and based on the segmentation, a saliency analysis method is used for analysis; an exponential function is used to build a saliency model, and after the object saliency is calculated, the object saliency is normalized to [0,1], and a threshold th is set, when the object saliency is higher than the threshold th, it is taken as a target object, otherwise it is taken as a background object, thereby achieving a preliminary screening of ship targets in the local area.

[0012] Furthermore, the step 1 further includes: (2) fine screening of ship targets based on a deep convolutional network. After subsequent area screening is performed using the saliency analysis method, the ship targets or the interference background of clouds and waves are further screened as suspected areas, and supervised learning is performed using convolutional neural network training. The back propagation algorithm is used to randomly initialize the network parameters, then calculate the output of the neural network, and then perform a reverse operation based on the difference between the output and the label, and gradually update the weights of the front layers of the neural network until the neural network converges and obtains a set of optimal solutions, thereby obtaining the neural network to be solved, and then inputting the candidate area into the solved neural network to determine the final ship detection result.

[0013] Furthermore, the step 2 further includes: (1) extracting the ship detection area based on the port knowledge base, first using the SIFT algorithm to align the target image with the reference image provided by the port knowledge base, and the process is as follows: first, SIFT feature points are extracted from the reference image and the resampled image of the image to be detected, including detecting the extreme points in the scale space, accurately determining the positions of the key points, and determining the main directions of the key points, and finally generating a SIFT descriptor, describing the SIFT feature points through a 128-dimensional feature vector; then matching the SIFT feature points extracted from the reference image and the resampled image of the image to be detected, and finally deleting the mismatched points using the RANSAC sampling method.

[0014] Furthermore, the step 2 further includes: (2) fast extraction of ship targets based on HOG features and SVM classification, using the OTSU background segmentation method to extract relatively bright suspected ship areas, and when there is a shadow, using a morphological method to project along the main direction of the detection target, and then searching along the main direction. If the main directions of two objects are consistent, the segmented parts are merged along the main direction; and the rotation angle of the ship target berthing terminal is obtained, and the information of the target knowledge base is used to adjust the direction of the ship target, so as to change the multiple possible directions of the ship target berthing into two possibilities, namely, the horizontal direction or the vertical direction.

[0015] Furthermore, step 3 further includes: (1) feature learning based on hybrid structure sparse constraints, a restricted Boltzmann machine unsupervised training method based on hybrid structure sparse constraints, and improving the generalization performance of the system by suppressing useless or noise information in the features through function constraints or regularization.

[0016] Furthermore, step 3 further includes: (2) fine-tuning model parameters, by training multiple restricted Boltzmann machines layer by layer to obtain high-level abstract features to achieve unsupervised feature learning, the parameters of the multiple restricted Boltzmann machines are used as initial values ​​of multi-layer neural network parameters, and the back propagation algorithm and conjugate gradient descent method are used to fine-tune the parameters of the entire neural network for supervised training, and training is performed using data with labeled categories.

[0017] Furthermore, the step 3 further includes: (3) unsupervised feature extraction. In the model training stage, a batch training method is used to randomly and evenly divide the training images into multiple data subsets, and each image belongs to only one of the subsets. The four restricted Boltzmann machines are trained layer by layer using an unsupervised pre-training method. In the layer-by-layer training, the output of the hidden layer of the previous Boltzmann machine is used as the input of the visible layer of the next Boltzmann machine. After the pre-supervised training is completed, a 5-layer neural network is initialized using the weight parameters of the four restricted Boltzmann machines.

[0018] Furthermore, the step 3 further includes: when the unsupervised pre-training and the supervised fine-tuning training of the neural network are completed, the final model is obtained, and when new data is obtained, it is only necessary to input the new data into the visual layer of the network, perform layer-by-layer forward propagation of the neural network, and finally obtain the learned feature vector.

[0019] The method of the present invention has the following advantages:

[0020] Based on the characteristics of wide-band visible light of the sensor, the present invention makes full use of the latest achievements of computer vision and machine learning to construct a visible light large-scale marine ship target automatic detection method, and realizes the rapid extraction of marine ship targets. Rapid extraction of ship targets from high-resolution images helps to assist the interpreter in quickly interpreting and obtaining data information.

[0021] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the following specifically cites a preferred embodiment and describes it in detail with the accompanying drawings as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a flow chart of a remote sensing information processing method based on deep learning training of the present invention;

[0023] Figure 2 Schematic diagram of the convolution kernel of the LoG filter, (a) a plane diagram of the convolution kernel; (b) a three-dimensional diagram of the convolution kernel; (c) a diagram of the relationship between the width between zero points of the convolution kernel and the scale parameter;

[0024] Figure 3Schematic diagram of the convolution and subsampling process;

[0025] Figure 4 is the convolutional neural network structure of the present invention;

[0026] Figure 5 Extract algorithm flow for ship targets;

[0027] Figure 6 This is a schematic diagram of the SIFT feature point matching process;

[0028] Figure 7 This is a diagram of the target feature learning system based on deep belief network;

[0029] Figure 8 Schematic diagram for pre-training and model parameter initialization. DETAILED DESCRIPTION

[0030] The technical scheme of the present invention will be clearly and completely described below in conjunction with specific embodiments, but those skilled in the art should understand that the embodiments described below are only used to illustrate the present invention and should not be regarded as limiting the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] A remote sensing information processing method based on deep learning training, such as Figure 1 As shown, including:

[0032] (I) Rapid screening of wide-band visible light marine targets based on target saliency analysis and deep learning model

[0033] Screening marine ship targets from visible light images based on manual interpretation is a very time-consuming and labor-intensive process. The size of a standard sensor image exceeds 80,000 pixels * 80,000 pixels, and the data volume exceeds 16G. If visual interpretation is relied upon, the interpretation time for a single scene will exceed 1.5 hours, and the work is difficult to process in parallel.

[0034] During the system construction process, based on the characteristics of the sensor's wide visible light range, the latest achievements in computer vision and machine learning were fully utilized to build a visible light automatic detection module for large marine ship targets, realizing the rapid extraction of marine ship targets.

[0035] The present invention includes two aspects: first, using the computer saliency determination method, the suspected areas of ship targets in the wide visible light image are quickly extracted, and these areas are all used as candidate targets for the next step of processing; then, for the candidate targets, a deep convolutional network is constructed to determine the ship targets, thereby finally realizing the extraction of marine ship targets. The specific method includes:

[0036] (1) Preliminary screening of suspected targets based on target significance analysis

[0037] Most of the previous image target detection algorithms are based on sliding windows and their extension methods. Their principle is to use a fixed-size detection window in the image and scan the image sequentially or according to certain rules with a certain step length, with the aim of enclosing possible targets in a certain detection window. This type of sliding window method does not rely on any prior knowledge or parameter learning, so it has many disadvantages: first, the sliding window scanning in the image will generate a large number of windows to be detected, resulting in low detection efficiency; in addition, since it does not rely on prior knowledge and searches with a certain step length, the generated window often cannot accurately locate the target, resulting in many false alarms and missed alarms, reducing the detection accuracy and recall rate.

[0038] In view of the problem that the scales and directions of ships at sea are different, it is considered to extract the edge information of the ship target to enhance the difference between the target and the background, so as to facilitate the separation of the ship target and the sea background. Taking into account the scale of the target, the Laplacian of Gaussian (LoG) filter is selected as the target enhancement filter for edge extraction.

[0039] Given the Laplacian operator and the two-dimensional Gaussian function as shown in equation (5-1) and equation (5-2), y and x correspond to the row direction (row number) and column direction (column number) of the two-dimensional image (matrix), σ is the scale parameter, and by combining equation (5-1) and equation (5-2) and expanding them, the mathematical expression of the LoG filter can be obtained as shown in (5-3).

[0040]

[0041] like Figure 1 As shown in the figure, a schematic diagram of the LoG filter convolution kernel is given, in which the scale parameter σ=2 and the filter window size L=13.

[0042] The selection of the scale parameter σ is determined by the size of the target on the image, and its value should be determined by referring to the pixel size of the target on the image. Considering that the sizes of marine ship targets vary, it is necessary to establish a LoG filter for possible target sizes. In the present invention, a multi-scale LoG filter is established to enhance the target. According to the properties of the LoG filter, the relationship between the zero-point width ω of the filter convolution kernel and σ is shown in Figure (c) above, and the zero-point width ω of the filter is consistent with the target size. Therefore, the expression of the multi-scale interval width ω is shown in (5-4), where as the difference parameter between adjacent scales.

[0043]

[0044] From the figure (c) above, we can see that the relationship between the scale parameter σ and ω can be expressed as formula (5-5), so the column value of σ is In this way, a series of scale LoG filter convolutions are used to enhance the processing of ship targets of different sizes on high-resolution optical images.

[0045] After multi-scale enhancement of the target, mean-shift is directly used for initial segmentation to obtain the object set {obj i}, and based on the segmentation, consider using the saliency analysis method for analysis. In the present invention, the uniqueness index of a target object is mainly considered. and spatial distribution indicators

[0046] The uniqueness index As shown in formula (5-6). obj is the number of objects obtained by initial segmentation in the protection area, represents the mean of the object, is the position of the object in the local area. In the actual calculation, the mean of the relative coordinates of all pixels in the local area of ​​the object is taken.

[0047]

[0048] and is the weight between two objects. In the specific implementation, the weight The Gaussian model is adopted, as shown in formula (5-7).

[0049]

[0050] where σ p As a control parameter, through experimental analysis, the uniform value is 0.3; Z i is the normalization factor, satisfying Expanding formula (5-6) yields the uniqueness index shown in formula (5-8).

[0051]

[0052] Spatial distribution characteristics of objects As shown in formula (5-9). is the position of the object in the local area. In actual calculation, it is the mean of the relative coordinates of all pixels of the object in the local area. i The definition is shown in formula (5-10), which defines all pixels belonging to category C; Defines the object and objects The similarity of the mean values.

[0053]

[0054] Weight The Gaussian model is still used, as shown in formula (5-11), where σ m As the control parameter, select σ based on data testing m =15. And the constraints between weight factors satisfy

[0055]

[0056] By expanding formula (5-9), we can obtain the result shown in formula (5-12).

[0057]

[0058] In this way, after calculating the uniqueness index and spatial aggregation index in the local area of ​​the object, the object's significance index can be calculated using formula (5-13).

[0059]

[0060] In the formula and are the uniqueness index and spatial distribution index of the object respectively; k is the control parameter. In the experiment, it was found that the spatial distribution index It can highlight the spatial distribution of the object well, so we consider using the exponential function to build the saliency model, and set k = 4 through experiments in this paper. After calculating the object saliency, it is normalized to [0,1], and an appropriate threshold th is set. When the saliency of the object is higher than the threshold, it is considered as the target object, otherwise it is a background object, thereby achieving the preliminary screening of ship targets in the local area.

[0061] (2) Detailed screening of ship targets based on deep convolutional networks

[0062] After subsequent area screening through saliency analysis, a series of suspected areas will be obtained. These areas are mainly ship targets or interference backgrounds such as clouds and waves. In order to improve the extraction accuracy of the target, these suspected areas need to be further screened.

[0063] As the forefront of machine learning, deep neural networks have been widely used in the field of automatic target detection and recognition, and have achieved very good results. Convolutional neural networks are multi-layer perceptrons that make full use of spatial relationships to reduce the number of parameters, thereby improving the performance of general backpropagation algorithms.

[0064] like Figure 2As shown, the convolutional neural network in the present invention is a multi-layer neural network composed of an input layer, a convolutional layer, a pooling layer, a fully connected layer and an output layer. Each layer in the network is composed of multiple two-dimensional planes, and each two-dimensional plane contains multiple independent neurons. Wherein the C layer is usually a convolutional layer, and each neuron in this layer is connected to the local area of ​​the previous layer. The original signal characteristics are enhanced and the noise is reduced through the convolution operation, thereby extracting local features, which can be used for classification, but using all the features to train the classifier will bring a huge amount of calculation and easily cause overfitting. In order to solve this problem, the features of different regions in the image can be aggregated and counted, and the maximum value or average value of the features of the region is used to represent the features of the region. These aggregated statistical features not only have lower dimensions, but also prevent overfitting. In addition, this operation also has translation invariance, and this operation is pooling.

[0065] Specifically, the convolution process can be described as: using a filter f with trainable parameters x After convolution of the input image of the previous layer, add bias b x , get the feature map C of the convolution layer x ; The downsampling process can be described as: After the N pixels of each neighborhood of the image input by the convolution layer are summed into one pixel, the scalar W x+1 Weighted, plus bias b x+1 b x+1 , the feature map S of the downsampling layer is generated through the activation function x+1 The weights of each neuron on a mapping surface in a convolutional neural network are shared, which significantly reduces the number of free parameters in the network, thereby greatly reducing the complexity of network parameter selection. Figure 2 shown.

[0066] The convolutional neural network used in the present invention is as follows: Figure 3 As shown in the figure, the network structure contains a total of 5 layers, consisting of 2 convolutional layers, 2 pooling layers and 1 fully connected layer.

[0067] The training process of the convolutional network is supervised learning, so the training sample set of the convolutional neural network is composed of vector pairs of the form (input vector, ideal output vector). For the present invention, we selected 2500+ ship samples by manual screening and randomly selected background samples. By performing data enhancement on the 2500+ ship samples (including X-axis and Y-axis mirroring, and +-90 degree rotation and mirroring), the data samples were expanded by 8 times to 2w+. Some positive and negative samples such as Figure 4 shown.

[0068] The back propagation algorithm is used for network training. This method first randomly initializes the network parameters, then calculates the network output, and then performs reverse operations based on the difference between the output and the label, gradually updating the weights of the previous layers of the network until the network converges. Suppose there is a sample set {(x (1) ,y (1) ),...,(x (m) ,y (m) )}, a total of m samples are included to train the neural network. The back propagation algorithm uses the gradient descent method to solve the neural network. For the sample (x, y), the cost function is defined as shown in formula (5-14):

[0069]

[0070] This is a network output value h W,b The variance cost function between (x) and the target value y. For the above data set containing m samples, its overall cost function is defined as shown in formula (5-15):

[0071]

[0072] The first term in the above formula is used to represent the mean square error, and the second term is a weight decay term. By changing the size of λ, the relative importance of the two terms in the above formula can be adjusted to prevent overfitting. Solving the neural network is to solve the parameters W and b when the cost function J(W,b) takes the minimum value. Normal distribution is usually used to randomly initialize the parameters. and Then the gradient descent method is used to solve it.

[0073] The update rules for parameters W and b in the gradient descent method are shown in equations (5-16) and (5-17) respectively:

[0074]

[0075] Here, α is the learning rate.

[0076] The back propagation algorithm is used to calculate the partial derivatives in the above formula. The specific steps are as follows:

[0077] ①. Perform forward propagation operation to obtain the activation value of each layer output.

[0078] ②. For the nth l The layer is the output layer calculation, and the expression is shown in formula (5-18):

[0079]

[0080] ③. For the first l=n l -1,nl -2,n l -3,...,2, each layer is calculated separately, as shown in formula (5-19): (l) =((W (l) ) T δ (l+1) )·f′(z (l) ) (5-19)

[0081] ④. Calculate the final partial derivatives, as shown in equations (5-20) and (5-21):

[0082]

[0083] The steps to use the back propagation algorithm to solve the partial derivatives and then update the weights using the gradient descent method are:

[0084] ①. For all layers, let ΔW (l) :=0,Δb (l) :=0

[0085] ②. For layers i = 1 to m, first use the back propagation algorithm to calculate as well as Then calculate ΔW (l) and Δb (l) , the calculation formula is shown in formula (5-22) and (5-23):

[0086]

[0087] ③. Update the weight parameters, as shown in equations (5-24) and (5-25):

[0088]

[0089] The gradient descent method reduces the cost function through continuous iteration until convergence, and then a set of optimal solutions will be obtained, and the neural network to be solved will be obtained. In this way, the candidate area can be input for judgment to obtain the final ship detection result.

[0090] This method has been tested many times. Taking a cloudy sensor image of a scene as an example, 3000 candidate suspected large ship target slices were screened out by using saliency, and 300 suspected large ship targets were further screened out by using a deep convolutional network, and an easy-to-use ship target confirmation tool was built. With this tool, all ship targets can be easily confirmed within 5 minutes. Practice shows that the detection rate of large ship targets using the above method is better than 95%, and the interpretation time of a single-scene sensor image is reduced from the original 90 minutes to 5 minutes, which improves the interpretation efficiency and greatly reduces the workload of the interpreter.

[0091] (II) Rapid extraction of nearshore port ships based on target knowledge base and small sample machine learning

[0092] The data that can be accessed by the present invention include high-resolution optical payloads of sensors, sensors, sensors and other platforms. The amount of image data accessed every day is large. Interpreters need to interpret ship target information from fixed ports. Therefore, rapid extraction of ship targets from high-resolution images will help assist interpreters in quickly interpreting and acquiring data information.

[0093] The extraction of ship targets in fixed ports can make full use of the target knowledge base. The port target knowledge base can be used to extract the areas where ship targets often dock, further extract the suspected target areas, and finally use machine learning methods to classify and screen ship targets.

[0094] (1) Ship detection area extraction based on port knowledge base

[0095] First, the SIFT algorithm is used to realize the registration processing of the target image and the reference image provided by the port knowledge base. The basic process is as follows: first, SIFT feature points are extracted from the reference image and the resampled image of the image to be detected, including detecting the extreme points in the scale space, accurately determining the position of the key points, and determining the main direction of the key points. Finally, a SIFT descriptor is generated, and the SIFT feature points are described by a 128-dimensional feature vector; then, the SIFT feature points extracted from the resampled image of the reference image and the image to be detected are matched, and finally the RANSAC sampling method is used to delete the mismatched points. The entire SIFT feature point matching process is as follows: Figure 5 shown.

[0096] In order to ensure the stability of the feature points extracted by the algorithm, the SIFT algorithm first establishes a multi-scale space and extracts scale-invariant feature points in the scale space. The construction of the scale space is to achieve the scale transformation of the original image by constantly changing parameters to obtain multi-resolution images of the same image. It is generally achieved through an image pyramid, and the resolution of each layer of the image in the pyramid is gradually reduced. In order to further find stable feature points in the image that can achieve high-precision alignment, a Gaussian difference scale space (DOG scale space) can be established based on the image Gaussian pyramid space. Each layer of the image in the DOG scale space is the difference between adjacent layers in each group of images in the Gaussian pyramid, and is calculated by convolution with Gaussian difference kernels of different scales. The Gaussian difference scale space is represented by the DOG operator, as shown in formula (5-26):

[0097] D(x,y,σ)=(G(x,y,kσ)-G(x,y,σ))*I(x,y)=L(x,y,kσ)-L(x,y,σ) (5-26)

[0098] Among them, G(x, y, σ) is the only possible linear convolution kernel that can achieve scale space transformation - Gaussian linear convolution. The specific expression is shown in formula (5-27):

[0099]

[0100] Among them, x and y are the coordinate positions of the pixel point, σ is the scale factor, and the larger σ is, the blurrier the image is.

[0101] In this way, the steps of establishing a multi-scale space can be described as follows: first, the image is convolved using a Gaussian function to complete the image preprocessing. Then, the image is downsampled using different sampling distances to generate a pyramid image grouping structure, and then each group of images is convolved with a Gaussian kernel function of different scales to generate a hierarchical structure for each group of images. In this way, the Gaussian pyramid structure of an image is established, and then the difference image of the adjacent layer images in each group of the Gaussian pyramid is used to construct a Gaussian difference image. The images between each group of adjacent layers are subtracted to generate a Gaussian difference image, and then each group of images in the Gaussian pyramid is operated in turn to finally obtain a Gaussian difference scale space.

[0102] After the Gaussian difference scale space is established, extreme point detection is performed in the difference scale space by comparing the grayscale values ​​of the sampling point with the 26 pixels in its field, including the 8 neighboring pixels in the same image and the 9 adjacent pixels in the upper and lower layers of the image. It should be noted that when the sampling point is an extreme point, the point is extracted as a candidate feature point, otherwise continue to traverse other pixels in the image to find the extreme point.

[0103] Since the position coordinates of the extreme points detected by DoG extreme value detection are all integers, the actual feature point positions are not necessarily located at integer coordinate positions. In order to locate feature points at the sub-pixel level, a three-dimensional quadratic function is fitted to the local image sampling points to determine the position of the extreme points after interpolation. The extreme points at the edge are extremely sensitive to noise. The Gaussian difference function has a strong edge response and is easily affected by noise. In order to ensure the stability of the feature points, some low-contrast points and edge response points need to be eliminated. First, the 2×2 Hessian matrix is ​​used to calculate the principal curvature, as shown in formula (5-28):

[0104]

[0105] The partial derivative in formula (5-28) can be approximated by the neighborhood difference of the current sampling point. The principal curvature of D is proportional to the eigenvalue of H. Among the eigenvalues ​​of H, the eigenvector corresponding to the largest eigenvalue represents the direction with the largest curvature; similarly, the eigenvector corresponding to the smallest eigenvalue represents the direction with the smallest curvature. Define the trace Tr(H) and determinant Det(H) of H, and their expressions are shown in formula (5-29):

[0106]

[0107] The value of Det(H) is calculated at the key point and compared with the set threshold. If it is less than the threshold, it is eliminated. This achieves the positioning of SIFT key points.

[0108] After determining the key point, the key main direction is further calculated. At this time, the gradient distribution characteristics of the key point neighborhood pixels are used to calculate the key point main direction. By using the neighborhood pixel gradient histogram and taking the neighborhood pixels into account through weighting, the main direction can be estimated more accurately. Select the Gaussian image L closest to the key point scale, and calculate the gradient modulus m(x,y) and gradient direction θ(x,y) of each point (x,y) on L, as shown in equations (5-30) and (5-31):

[0109]

[0110] In actual calculation, sampling is performed in the neighborhood window centered on the key point, and the gradient direction of the neighborhood pixels is counted using a histogram. The statistical radius is 3*1.5*σ. When counting the histogram, Gaussian weighting is used for every three adjacent pixels (the template is [0.25, 0.5, 0.25]), and the weighting is repeated twice in a row. The horizontal axis of the histogram represents the gradient direction, with 36 segments for every 10 degrees, and the vertical axis represents the weighted gradient modulus. In the above gradient histogram, the direction corresponding to the maximum value is found and used as the main direction of the feature point. When there is another peak with 80% of the energy of the main peak, this direction is considered to be the auxiliary direction of the feature point. Therefore, a key point may be assigned multiple directions (one main direction and more than one auxiliary direction), which can improve the stability of feature matching.

[0111] At this point, the key points of the image have been detected. Each key point has three pieces of information: position, scale and direction. The next step is to quantify the SIFT features. To ensure that the SIFT feature vector is rotationally invariant, the coordinate axis is first rotated to the main direction of the key point to ensure rotational invariance. Then, a Gaussian image is selected based on the scale of the key point, and the gradient modulus and gradient direction of all pixels in a 16×16 window centered on the key point (the row and column where the key point is located are not taken) are calculated according to equations (5-30) and (5-31). Then the neighborhood is divided into 4×4 sub-areas, and the gradient direction histogram of each 4×4 image block in the 8 directions of {0°, 45°, 90°, 135°, 180°, 225°, 270°, 315°} is calculated, thereby forming an 8-dimensional vector to represent the seed point, so that a feature point is represented by a 16×8=128-dimensional vector.

[0112] After obtaining the SIFT feature points of the target image and the reference image, the feature points can be matched. First, the Kd tree is constructed for the feature points on the resampled images of the reference image and the image to be detected to complete the search faster. Then the Euclidean distance between the feature points in the two images is calculated and used as the similarity judgment of the feature point vectors. Based on the feature points of the target image, the feature points that are closest and second closest to the feature points of the target image are searched in the image to be registered, and finally the target feature vector is quickly matched.

[0113] Assume that the feature point descriptor of the original image is represented by R: i =(r i1 ,r i2 ,...,r i128 ), and the feature point descriptor of the image to be registered is represented by S: S i =(s i1 ,s i2 ,...,s i128 ), then the similarity measure of any two descriptors is expressed as shown in formula (5-32):

[0114]

[0115] Assume that d(R i ,S i ) and d(R i ,S p ) are R i With the closest point S i and the next closest point to T S p The distance between them is the distance between them. To obtain the paired feature point descriptors, the following conditions must be met: d(R i ,S p )<T, T is the threshold value, which is generally set to a value between [0.4, 0.65]. This will initially achieve the matching between SIFT feature points.

[0116] Although the 128-dimensional feature descriptor of the SIFT algorithm has good fault tolerance, some mismatched point pairs may appear in the registration. Through experimental analysis, the reason is often caused by the presence of multiple similar objects in the image itself, so it is necessary to further eliminate the mismatched point pairs. Although it is found in the experiment that the purpose of eliminating mismatched point pairs can be achieved by reducing the threshold T set in the registration process, it is also easy to lose many correct registration point pairs. In actual engineering, random sampling consistency, that is, RANSAC (Random Sample Consensus) method is used to obtain the optimal registration point pairs, and then eliminate mismatches. First, extract several pairs of registration points from the obtained registration point pairs, calculate the transformation matrix, and record these pairs of points as "inliers". Continue to find non-inliers in the registration point pairs. If these registration point pairs conform to the matrix, they are added to the inliers. When the number of point pairs in the inliers is greater than the set threshold, the matrix is ​​determined to be an accurate transformation matrix. According to the above method, random sampling is performed N times, the maximum set of inliers is selected, and mismatched point pairs such as non-inliers are eliminated.

[0117] After matching the same-name points, the geometric transformation between the target image and the reference image can be obtained. Taking into account the inherent geometric distortion of the image, the quadratic polynomial model is considered to be used to fit the geometric model transformation between the two images. The basic formula is shown in formula (5-33):

[0118]

[0119] Where (x, y) is the coordinate of the point on the image to be detected, (x ref ,y ref ) are the coordinates of the points on the reference image. In this way, the geometric transformation relationship between the image to be detected and the reference image is established. Further, the range corresponding to the ROI on the reference image on the image to be detected can be obtained. The actual detection range on the image to be detected can be extracted through mask operation.

[0120] (2) Rapid extraction of ship targets based on HOG features and SVM classification

[0121] The background of the port target is relatively simple and dark, so when extracting the suspected area, background segmentation methods such as OTSU can be used to extract the relatively bright suspected ship area. When there is a shadow, the ship segmentation result may be incomplete. Consider using morphological methods for processing. The basic idea is to project along the main direction of the detection target and then search along the main direction. If the main directions of two objects are the same, merge the segmented parts along the main direction, such as Figure 6 shown.

[0122] And with the help of the target knowledge base information, the rotation angle of the ship target berthing terminal can be obtained. In this way, with the help of the information in the knowledge base, the direction of the ship target can be corrected. This innovatively changes the multiple possible directions of the ship target berthing into two possibilities, namely horizontal or vertical.

[0123] The purpose of aligning the ship target berthing area is to use the Histogram of Oriented Gradients (HOG) as the feature expression of the sample. This method can make good use of the edge gradient information of the target. Obviously, this is very suitable for the ship target docked at the port terminal. However, the HOG feature does not have the direction rotation invariance. The present invention realizes the alignment of the ship target direction through the knowledge base data, thereby facilitating the use of HOG features to describe the target.

[0124] The calculation steps of HOG are as follows: first calculate the first-order gradient of the image, and then perform horizontal and vertical convolution on the image using the [-1, 0, 1] kernel and its transpose. The specific calculation formula is shown in formula (5-34):

[0125]

[0126] Among them, I(x,y), G x (x,y),G y (x, y) represents the pixel value, horizontal gradient value and vertical gradient value of image I at the pixel point (x, y), and further the gradient G(x, y) and direction α(x, y) at the pixel point (x, y) can be obtained, as shown in formula (5-35):

[0127]

[0128] Then, the gradient image of the image is obtained by calculating each point, and then the local image area is encoded. The image is divided into several cells, and the gradient distribution is independently counted in each cell. The gradient histogram is counted with the gradient direction as the horizontal axis. In actual processing, in order to improve the adaptability of the feature to factors such as local illumination and shadow, several cells are combined into large blocks, and the histogram within the block is normalized to obtain the final HOG feature.

[0129] Furthermore, we used SVM with soft margin to classify HOG features, as shown in formula (5-36):

[0130]

[0131] Where w and b are the classification model parameters, ζ iis a slack variable. To solve the sample inseparability problem, the problem can be transformed into an optimization function as shown in formula (5-37):

[0132]

[0133] where α i is the Lagrange multiplier,<x,z> is the feature inner product, and can be mapped using a kernel function. Generally, a Gaussian kernel function is used, which can map the original feature to infinite dimensions, as shown in formula (5-38):

[0134]

[0135] In this way, we can realize the determination of ship targets by using HOG features and SVM classifier. After extracting the ship targets in the straightening area, we further use the dock direction information in the knowledge base to realize the rotation recovery of the detection frame, thereby marking the specific detection position in the original image.

[0136] When the user loads the image for target interpretation, the automatic detection results will be loaded and prompted to the user to assist the user in the interpretation work.

[0137] 3. Feature mining of massive SAR data based on sparse deep belief networks

[0138] The invention is constructed by accessing sensors and sensor SAR payload data, and it is of great significance to interpret and interpret the ship target information from the SAR image data.

[0139] Aiming at the characteristics of massive SAR data, the restricted Boltzmann machine is used to perform layer-by-layer unsupervised feature learning on a large amount of training data. The learning process uses the contrast divergence algorithm based on Gibbs sampling to solve the parameters. Then the parameters solved by the multi-layer restricted Boltzmann machine are used as the initial values ​​of the multi-layer neural network. The model parameters of the multi-layer neural network are fine-tuned using the labeled training data, and the multi-layer logistic regression model is used as the final classification model. The fine-tuning process uses the back propagation algorithm and the conjugate gradient descent method. Finally, the new data is forward propagated through the network to obtain the final feature representation result.

[0140] (1) Feature learning based on hybrid structure sparse constraints

[0141] In high-resolution SAR image recognition, an image contains not only the target of interest to the interpreter, but also a large amount of complex background. This interference information will affect the performance of the recognition method. A common method is to suppress useless or noise information in the features through function constraints or regularization to improve the generalization performance of the system. In this regard, this technology proposes an unsupervised training method of a restricted Boltzmann machine based on hybrid structure sparse constraints. Specifically, the optimization loss function is defined as shown in formula (5-60):

[0142] L=-logP(v)+μ||P(h=1|v)||1+λ||P(h=1|v)|| 1,2 +η||W||2 (5-60)

[0143] Among them, μ represents the hidden layer node activation probability, L1 norm constraint partition coefficient, λ represents the hidden layer node activation probability L 1,2 The partition coefficient of the mixed norm constraint, η represents the partition coefficient of the L2 norm constraint of the weight matrix W.

[0144] The second term on the right side of formula (5-60) is the hidden layer node sparse constraint based on the L1 norm, which is defined as follows:

[0145]

[0146] According to sparse coding theory, the optimization problem with norm constraints is equivalent to the optimization problem with L0 norm constraints, and the latter can make most of the optimized target parameters 0, achieving the purpose of parameter sparseness. Here, adding sparse constraints can make the activation probability of most hidden layer nodes 0, thereby achieving the purpose of encoding the input data features with a small number of hidden layer nodes.

[0147] The third term on the right side of formula (5-60) is based on L 1,2 Sparse constraint of the mixed norm. Divide the hidden layer nodes of the restricted Boltzmann machine into m sets, where the L of the hidden nodes is defined 1,2 The mixed norm is as shown in formula (5-62):

[0148]

[0149] Among them G m Represents the mth hidden layer node set. By optimizing and adding L 1,2The objective function of the mixed norm is that the sum of the squares of the node activation values ​​of most hidden layer node sets is close to zero, which makes the activation values ​​of most nodes in the set also close to zero, thus achieving an effect similar to the L1 norm constraint. Therefore, when data is input to the visible layer nodes, only a small number of hidden layer nodes in the set are activated, thus achieving the purpose of sparse constraint. By adding the L1 norm of the hidden layer node activation value and the L 1,2 Mixed norm constraints can give different types of constraints to the activation probabilities of hidden layer nodes.

[0150] The fourth term on the right side of formula (5-60) is the L2 norm constraint that limits the Boltzmann machine network weight matrix. Its specific form is shown in formula (5-63):

[0151]

[0152] Adding constraints on the weight matrix during optimization can make the model smoother and reduce the possibility of overfitting by suppressing excessive weights. At the same time, this constraint can suppress redundant weights in the weight matrix and enhance the generalization performance of the model.

[0153] Optimizing energy models usually relies on stochastic gradient descent, which requires the use of algorithms such as Gibbs sampling to estimate the parameters of the model and thus calculate the gradient. However, repeatedly performing Gibbs sampling on undirected graph models such as Boltzmann machines is not only computationally intensive but also very time-consuming. This technology uses the contrast divergence algorithm to perform approximate calculations. This approximate calculation can achieve similar optimization results and greatly reduces the amount of calculation. Therefore, this method is widely used in the actual gradient solution process.

[0154] (2) Model parameter fine-tuning

[0155] By training multiple restricted Boltzmann machines layer by layer, we can obtain high-level abstract features, that is, achieve the purpose of unsupervised feature learning. The parameters of these restricted Boltzmann machines will be used as the initial values ​​of the multi-layer neural network parameters. Then we use the back propagation algorithm and conjugate gradient descent method to fine-tune the parameters of the entire network. This process is a supervised training process and will be trained with labeled data.

[0156] In the optimization problem of high-dimensional feature space, in order to avoid the possible local optimum, the model has more optimization paths to choose from, which makes it easier to find the global optimal solution for the optimization problem in high-dimensional space than in low-dimensional space. In other words, the more hidden layer nodes in the network, the lower the possibility of the optimization process falling into the local extreme value, and the more parameters, the stronger the feature expression ability of the model. However, the increase in model parameters will bring another more serious problem, that is, the possibility of overfitting is greatly increased, which greatly reduces the generalization performance of the model. In the relevant theory of machine learning, the most commonly used method to avoid overfitting is regularization, that is, adding regularization terms to the optimization objective function to suppress over-training of parameters. In this technology, we will use sparsity constraints to improve the back propagation algorithm used for multi-layer neural network optimization to improve the generalization performance of the model.

[0157] When the activation probability of a hidden layer node is close to 1, the node is considered to be activated, and when it is close to 0, the node is suppressed. When the average activation probability of a node in the hidden layer for all input data is small, it can be considered that the node is not activated most of the time, and the hidden layer node can be considered to be in a sparse state. The average activation value of the jth hidden node in the hidden layer is defined as formula (5-64):

[0158]

[0159] where a j (x (i) ) represents the activation probability of the jth hidden layer node when the i-th data is input to the visual layer, and M represents the total number of input data. When the value of is a small value, it can be considered that the hidden layer nodes have reached a sparse state. Close to a smaller value, the following sparsity constraint function is defined here as shown in formula (5-65):

[0160]

[0161] Add this constraint function to the loss function of the back propagation algorithm, and define it as formula (5-66):

[0162]

[0163] Where β represents the partition coefficient of the sparse constraint term, and J(W,b;x,y) represents the loss function of the back propagation algorithm, which is in the form of formula (5-67):

[0164]

[0165] where h w,b(x) is the predicted value of the input data x through the multi-layer neural network, and y is the manually labeled value of the data. The loss function is also optimized using the stochastic gradient descent method. The specific method is shown in equations (5-68) and (5-69):

[0166]

[0167]

[0168] The partial derivative calculation method of the model is as shown in equations (5-70) and (5-71):

[0169]

[0170] in represents the back-propagated residual of the i-th node in the k+1-th hidden layer of the neural network. In this model, the residual function can be derived:

[0171]

[0172] In formula (5-72), f'(x i ) represents the derivative of the activation function. The ρ value is set to a value close to 0. Through the above iterative optimization, a multi-layer neural network model with sparse hidden layer nodes is finally output.

[0173] (3) Unsupervised feature extraction

[0174] In the model training stage, a batch training method is used to randomly and evenly divide the training images into multiple data subsets, and each image belongs to only one of the subsets. The unsupervised pre-training method introduced above is used to train four restricted Boltzmann machines layer by layer, where the output of the hidden layer of the previous Boltzmann machine is used as the input of the visible layer of the next Boltzmann machine. After the pre-training is completed, a 5-layer neural network is initialized with the weights and other parameters of the four restricted Boltzmann machines. During the parameter initialization setting process, the target feature learning system pre-training and model parameter initialization based on the deep belief network are as follows Figure 7 shown.

[0175] After the parameters are initialized, a supervised layer is added to the top layer of the neural network. The parameters of this layer are initialized by a Gaussian distribution random number matrix with a mean of 0 and a standard deviation of 0.01. Then we use the supervised fine-tuning algorithm introduced earlier and the labeled data to perform supervised training of the neural network. The classification model of the supervised layer is defined as follows:

[0176]

[0177] After the network's unsupervised pre-training and supervised fine-tuning training are completed, a final model is obtained. When new data is obtained, the new data only needs to be simply processed and input into the network's visual layer, and the neural network is forward propagated layer by layer to finally learn the feature vector.

[0178] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A remote sensing information processing method based on deep learning training, comprising: Step 1: Rapid screening of wide-band visible light marine targets based on target saliency analysis and deep learning model: First, the suspected area of ​​the ship target in the wide-band visible light image is quickly extracted using the saliency determination method in the computer, and the suspected area is used as a candidate target for the next step of processing; then a deep convolutional network is constructed for the candidate target to determine the ship target, thereby finally realizing the extraction of the marine ship target; Step 2: Rapid extraction of nearshore port ships based on target knowledge base and small sample machine learning. Rapid extraction of ship targets from high-resolution images, assisting the reader to quickly interpret and obtain data information, using the port target knowledge base to extract the areas where ship targets often dock, further extracting the suspected target areas, and finally using machine learning methods to classify and screen ship targets; Step 3: Feature mining of massive SAR data based on sparse deep belief networks. Restricted Boltzmann machines are used to perform layer-by-layer unsupervised feature learning on a large amount of training data. The learning process uses the contrast divergence algorithm based on Gibbs sampling to solve the parameters. The parameters solved by the multi-layer restricted Boltzmann machine are then used as the initial values ​​of the multi-layer neural network. The model parameters of the multi-layer neural network are then fine-tuned using the labeled training data, and the multinomial logistic regression model is used as the final classification model. The fine-tuning process uses the back-propagation algorithm and the conjugate gradient descent method. The fine-tuned new data is forward propagated through the network to obtain the final feature representation result.

2. The remote sensing information processing method based on deep learning training according to claim 1, characterized in that: The step 1 further includes: (1) preliminary screening of suspected targets based on target saliency analysis, extracting edge information of ship targets at sea with different scales and different navigation directions to enhance the difference between the target and the background, thereby facilitating the separation of the ship target from the sea background.

3. The remote sensing information processing method based on deep learning training according to claim 2 is characterized in that: The initial screening of suspected targets based on target saliency analysis further includes: enhancing the ship targets of different sizes on the high-resolution optical image by convolution of a series of scale LoG filters, and performing initial segmentation by mean-shift to obtain an object set {obj i }, and based on the segmentation, a saliency analysis method is used for analysis; an exponential function is used to build a saliency model, and after the object saliency is calculated, the object saliency is normalized to [0,1], and a threshold th is set, when the object saliency is higher than the threshold th, it is taken as a target object, otherwise it is taken as a background object, thereby achieving a preliminary screening of ship targets in the local area.

4. The remote sensing information processing method based on deep learning training according to claim 2 is characterized in that: The step 1 further includes: (2) fine screening of ship targets based on a deep convolutional network. After subsequent area screening is performed using the saliency analysis method, ship targets or interference backgrounds of clouds and waves are further screened as suspected areas. Convolutional neural network training is used for supervised learning. A back propagation algorithm is used to randomly initialize network parameters, then calculate the output of the neural network, and then perform a reverse operation based on the difference between the output and the label, gradually updating the weights of the front layers of the neural network until the neural network converges and obtains a set of optimal solutions, thereby obtaining the neural network to be solved, and then inputting the candidate area into the solved neural network to determine the final ship detection result.

5. The remote sensing information processing method based on deep learning training according to claim 1, characterized in that: The step 2 further includes: (1) extracting the ship detection area based on the port knowledge base, first using the SIFT algorithm to align the target image with the reference image provided by the port knowledge base, and the process is as follows: first, SIFT feature points are extracted from the reference image and the resampled image of the image to be detected, including detecting the extreme points in the scale space, accurately determining the positions of the key points, and determining the main directions of the key points. Finally, a SIFT descriptor is generated, and the SIFT feature points are described by a 128-dimensional feature vector; then, the SIFT feature points extracted from the reference image and the resampled image of the image to be detected are matched, and finally the RANSAC sampling method is used to delete the mismatched points.

6. The remote sensing information processing method based on deep learning training according to claim 5, characterized in that: The step 2 further includes: (2) fast extraction of ship targets based on HOG features and SVM classification, using the OTSU background segmentation method to extract relatively bright suspected ship areas, and when there is a shadow, using a morphological method to project along the main direction of the detection target, and then searching along the main direction. If the main directions of two objects are consistent, the segmented parts are merged along the main direction; and the rotation angle of the ship target berthing terminal is obtained, and the direction of the ship target is adjusted using the information of the target knowledge base, so as to change the multiple possible directions of the ship target berthing into two possibilities, namely the horizontal direction or the vertical direction.

7. The remote sensing information processing method based on deep learning training according to claim 1, characterized in that: The step 3 further includes: (1) feature learning based on hybrid structure sparse constraints, a restricted Boltzmann machine unsupervised training method based on hybrid structure sparse constraints, and improving the generalization performance of the system by suppressing useless or noise information in the features through function constraints or regularization.

8. The remote sensing information processing method based on deep learning training according to claim 7, characterized in that: The step 3 further includes: (2) fine-tuning the model parameters, by training multiple restricted Boltzmann machines layer by layer to obtain high-level abstract features to achieve unsupervised feature learning, the parameters of the multiple restricted Boltzmann machines are used as the initial values ​​of the multi-layer neural network parameters, and the back propagation algorithm and conjugate gradient descent method are used to fine-tune the parameters of the entire neural network for supervised training, and training is performed using data with labeled categories.

9. The remote sensing information processing method based on deep learning training according to claim 8, characterized in that: The step 3 further includes: (3) unsupervised feature extraction. In the model training stage, a batch training method is used to randomly and evenly divide the training images into multiple data subsets, and each image belongs to only one of the subsets. The four restricted Boltzmann machines are trained layer by layer using an unsupervised pre-training method. In the layer-by-layer training, the output of the hidden layer of the previous Boltzmann machine is used as the input of the visible layer of the next Boltzmann machine. After the pre-supervised training is completed, a 5-layer neural network is initialized using the weight parameters of the four restricted Boltzmann machines.

10. The remote sensing information processing method based on deep learning training according to claim 9, characterized in that: The step 3 further includes: when the unsupervised pre-training and the supervised fine-tuning training of the neural network are completed, the final model is obtained, and when new data is obtained, it is only necessary to input the new data into the visual layer of the network, perform layer-by-layer forward propagation of the neural network, and finally obtain the learned feature vector.

Images