A remote sensing image target detection method based on visual saliency and multi-scale feature fusion

Abstract

The present invention discloses a remote sensing image target detection method based on visual saliency and multi-scale feature fusion, comprising the following steps: step 1, saliency map generation; step 2, target candidate region extraction: roughly positioning the target candidate frame according to the closed contour characteristic of the object target; step 3, multi-scale feature fusion and detection. The present invention adopts the above remote sensing image target detection method based on visual saliency and multi-scale feature fusion to solve the problem of remote sensing image target detection under conditions of complex background and inconsistent scale.

Description

Technical Field

[0001] The present invention relates to the technical field of high-resolution remote sensing image detection, and in particular to a remote sensing image target detection method based on the fusion of visual saliency and multi-scale features. Background Art

[0002] The integrated earth observation platform of the sky, air and ground established by the high-resolution earth observation system provides a very rich data source for the processing of remote sensing images. With the development of big data and artificial intelligence technology, remote sensing information perception and fusion are increasingly valued in the civil and military fields.

[0003] High-resolution remote sensing images are intuitive and easy to understand, and their spatial resolution is generally high. They can provide rich details of the target, which also provides strong support for accurate detection of the target. At the same time, with the development of earth observation systems, how to effectively process remote sensing images and obtain more valuable information from them has gradually become a hot topic and difficulty in current research at home and abroad.

[0004] Target detection is an important task in remote sensing image processing. In the civilian field, target recognition can be used for land resource survey and planning, urban construction, traffic monitoring and emergency search and rescue; in the military field, it can achieve comprehensive monitoring of a certain area, detect military facilities, discover military targets of great strategic significance, and understand military deployment and combat intentions from a tactical perspective. However, due to the complex background, unclear target features and different scales of high-resolution remote sensing images, it is difficult to detect them. Summary of the invention

[0005] The purpose of the present invention is to provide a remote sensing image target detection method based on the fusion of visual saliency and multi-scale features, which solves the problem of remote sensing image target detection under conditions of complex background and inconsistent scales.

[0006] To achieve the above object, the present invention provides a remote sensing image target detection method based on visual saliency and multi-scale feature fusion, comprising the following steps:

[0007] Step 1: Generate saliency map;

[0008] Step 2: Extraction of target candidate regions: Coarse positioning of the target candidate frame based on the closed contour characteristics of the target object;

[0009] Step 3: Multi-scale feature fusion and detection.

[0010] Preferably, the generation of the saliency map in step 1 is specifically as follows:

[0011] The visual saliency of the remote sensing image is calculated, and the saliency map is generated using global color contrast. If the color of a pixel in the remote sensing image is different from the colors of the surrounding pixels, the pixel is considered to have a certain degree of saliency. If the color of the pixel is similar to that of the surrounding pixels, the pixel is considered to be non-saliency. If the color of a part or target in the remote sensing image is the same or similar, the colors of the surrounding pixels are similar. At this time, the boundary around the pixel and its distance from the center of the remote sensing image are combined to comprehensively consider whether the pixel value is saliency;

[0012] Define a pixel I in the remote sensing image I k The significance value of:

[0013]

[0014] Among them, D(I k ,I i ) is pixel I k and I i The color distance metric of the remote sensing image pixel is expressed as:

[0015]

[0016] Among them, C l is the remote sensing image pixel I k The color value of the remote sensing image, n represents the number of colors, f j Indicates color C j The probability of appearing in the remote sensing image is calculated, and then the color channel is quantized into m values. The quantized remote sensing image color is a dimensionality reduction processing of the remote sensing image. At the same time, the remote sensing image further ignores the colors with low occurrence frequency, reduces the sensitivity to low-frequency color signals, enhances the robust performance of remote sensing image processing, selects n colors with the highest occurrence frequency, and then calculates the weight value of each color.

[0017] Preferably, the color weight value is calculated as follows:

[0018] First, define F j As shown below:

[0019]

[0020] A smaller F j The value indicates that the pixels of the same color in the remote sensing image are close to each other and show a high degree of aggregation. Represents pixel I k The position in the remote sensing image, A j Indicates that there is a certain same color C in the remote sensing image j The center position of all pixels, N jIndicates that the remote sensing image has a certain same color C j The number of all pixels;

[0021] Secondly, the weight value of the selected color is expressed as:

[0022] w j =exp(-F j )(4)

[0023] The final pixel P k The saliency value in remote sensing images is expressed as

[0024]

[0025] Preferably, the step 2 of roughly positioning the target candidate frame according to the closed contour feature of the object target specifically comprises the following steps:

[0026] S1, calculate the gradient map corresponding to the remote sensing image, and calculate and solve its gradient amplitude;

[0027] S2, scale the gradient amplitude map initially obtained in the remote sensing image, normalize it to a scale of 10*10 and use it as a gradient feature;

[0028] S3, extracting sample features of the target and background in the remote sensing image, inputting them into the SVM to train the classifier, so as to distinguish the features of the target and non-target, so as to determine whether there is a possible target in the remote sensing image;

[0029] S4. Use the classifier trained in S3 to search the remote sensing image to obtain a series of target areas. Use non-maximum suppression to select the candidate box where the target is most likely to exist. After the target candidate box is selected, the deep reinforcement learning method is used to gradually learn how to accurately locate the target from feedback and incentives to address the problem that the candidate box is not accurate enough in locating the target.

[0030] Preferably, the steps of using deep reinforcement learning method to gradually learn how to accurately locate the target from feedback and incentives are as follows:

[0031] A regular quadrilateral with a side length smaller than the target candidate box is used to search inside the candidate box. An action set A1-A5 is defined to represent the four extreme positions and one center position for searching inside the candidate box. Then, IoU(r,g) is introduced, where r represents the current area and g represents the ground truth. At this time, a reward function is defined as follows:

[0032] R r (s t ,s t+1 )=IoU(r t+1,g)-IoU(r t ,g) (6)

[0033] Among them, at time t, the area r t The state is s t Then, after an action A in the action set, region r t becomes r t+1 , the state is determined by s t becomes s t+1 If after an action, the IoU decreases instead of increases, a -1 reward value is fed back to the agent, otherwise, a +1 reward value is given to the agent. By controlling the reward value, the agent gradually learns how to obtain the maximum reward, that is, gradually approaches and finally determines the exact position of the target. For the above reinforcement learning process, a deep Q network is used for optimization and solution. Finally, the target area obtained by reinforcement learning is extracted as the final candidate area.

[0034] Preferably, the four extreme directions are the upper left corner, the upper right corner, the lower left corner and the lower right corner.

[0035] Preferably, the multi-scale feature fusion and detection in step 3 specifically includes the following steps:

[0036] S1. Perform block sampling on the candidate area obtained in step 2, and the dimension of each block is x×x;

[0037] S2, performing unsupervised training on the collected blocks using an autoencoder, wherein the parameters obtained from the training are used to initialize the neural network;

[0038] S3, scale the candidate area to form images of three scales, and construct a convolutional neural network of three channels using the method in S2;

[0039] S4. After the unsupervised training of the three-channel convolutional neural network is completed, the unsupervised trained model is used as a feature extractor;

[0040] S5, using the feature extractor in S4 to extract and represent the features of the candidate region, construct a fused feature vector, and enhance the expression performance of the model;

[0041] S6, training a classifier using the fused feature vector and label information;

[0042] S7, extracting features of the candidate region through the trained feature extractor, and fusing the features of the three channels;

[0043] S8. Classify the fused features to complete target detection.

[0044] Preferably, the unsupervised training using the autoencoder in step S2 is specifically as follows: n blocks are extracted from the candidate area, the dimension of each block is x×x, where x represents the size of the convolution kernel, and each block x×x can be transformed into an N-dimensional vector, where N=x×x×3, thus obtaining a matrix N×n, where the column vector represents a block v input ;

[0045] An input vector v input Perform the following processing:

[0046] p=f(W1v input +b1) (7)

[0047] Among them, W1 is the weight of the autoencoder network, b1 is its bias, and f() is the activation function. The weight internalizes the distribution law and deep mapping relationship of the target feature, transforms it into a feature map and outputs the corresponding feature vector. Then, the vector is decoded by the following formula:

[0048] q=W2 T ·p+b2 (8)

[0049] Among them, W2 is the decoding network weight matrix, b2 is the decoding bias, and reconstruction is performed through encoding and decoding. At this time, the loss function of the network model is defined as the reconstruction error between the original target and the reconstructed target. After that, the feature extractor is trained by minimizing the following loss function:

[0050]

[0051] Therefore, the present invention adopts the above-mentioned remote sensing image target detection method based on visual saliency and multi-scale feature fusion to solve the problem of remote sensing image target detection under conditions of complex background and inconsistent scale.

[0052] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 The present invention is a flowchart of an embodiment of a remote sensing image target detection method based on the fusion of visual saliency and multi-scale features. DETAILED DESCRIPTION

[0054] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0055] Example

[0056] The present invention provides a remote sensing image target detection method based on visual saliency and multi-scale feature fusion, comprising the following steps:

[0057] Step 1, saliency map generation: perform visual saliency calculation on the remote sensing image, and use global color contrast to generate a saliency map. If the color of a pixel in the remote sensing image is different from the colors of the surrounding pixels, the pixel is considered to have a certain degree of saliency. If the color of the pixel is similar to that of the surrounding pixels, the pixel is considered to be non-saliency. If the color of a part or target in the remote sensing image is the same or similar, the colors of the surrounding pixels are similar. At this time, the boundary around the pixel and its distance from the center of the remote sensing image are combined to comprehensively consider whether the pixel value is significant.

[0058] Define a pixel I in the remote sensing image I k The significance value of:

[0059]

[0060] Among them, D(I k ,I i ) is pixel I k and I i The color distance metric of the remote sensing image pixel is expressed as:

[0061]

[0062] Among them, C l is the remote sensing image pixel I k The color value of the remote sensing image, n represents the number of colors, f j Indicates color C j The probability of appearing in the remote sensing image is calculated, and then the color channel is quantized into m values. The color of the quantized remote sensing image is a dimensionality reduction processing of the remote sensing image. At the same time, the remote sensing image further ignores the color with low frequency, reduces the sensitivity to the low-frequency color signal, enhances the robust performance of the remote sensing image processing, selects the n colors with the highest frequency, and then calculates the weight value of each color, which is the parameter of the visual saliency of the remote sensing image. Its physical meaning is that for different remote sensing image scenes or targets, the color with the highest frequency always has a certain combination form. In order to reduce the number of colors of the remote sensing image, the color channel is quantized into m values. The quantized remote sensing image color can not only effectively represent the information of the original image, but also reduce the complexity of subsequent remote sensing image processing, which can be considered as a dimensionality reduction processing of the remote sensing image. At the same time, the remote sensing image further ignores the color with low frequency, reduces the sensitivity to the low-frequency color signal, enhances the robust performance of the remote sensing image processing, and selects the n colors with the highest frequency. Then, the weight value of each color is calculated.

[0063] The color weight value is calculated as follows:

[0064] First, define Fj As shown below:

[0065]

[0066] A smaller F j The value indicates that the pixels of the same color in the remote sensing image are close to each other and show a high degree of aggregation. Represents pixel I k The position in the remote sensing image, A j Indicates that there is a certain same color C in the remote sensing image j The center position of all pixels, N j Indicates that the remote sensing image has a certain same color C j The number of all pixels in the remote sensing image; Formula (3) reflects the degree of color aggregation and the characteristics of the distribution area in the image, so as to better describe the remote sensing image and help to establish a reasonable calculation model.

[0067] Secondly, the weight value of the selected color is expressed as:

[0068] w j =exp(-F j ) (4)

[0069] The final pixel P k The saliency value in remote sensing images is expressed as

[0070]

[0071] Step 2: Extracting target candidate regions: Based on the closed contour feature of the target object, the target candidate frame is roughly located, which specifically includes the following steps:

[0072] S1. Calculate the gradient map corresponding to the remote sensing image and calculate and solve its gradient amplitude.

[0073] S2. The gradient amplitude map initially obtained in the remote sensing image is scaled, normalized to a scale of 10*10 and used as a gradient feature.

[0074] S3. Extract sample features of the target and background in the remote sensing image, input them into the SVM to train the classifier, so as to distinguish the features of the target and non-target, so as to determine whether there is a possible target in the remote sensing image.

[0075] S4. Use the classifier trained in S3 to search the remote sensing image to obtain a series of target areas. Use non-maximum suppression to select the candidate box that is most likely to contain the target. After the target candidate box is selected, the deep reinforcement learning method is used to gradually learn how to accurately locate the target from feedback and incentives to address the problem that the candidate box is not accurate enough in locating the target. The steps to gradually learn how to accurately locate the target from feedback and incentives using the deep reinforcement learning method are as follows:

[0076] A regular quadrilateral with a side length smaller than the target candidate box is used to search inside the candidate box. An action set A1-A5 is defined to represent the four extreme positions (upper left corner, upper right corner, lower left corner and lower right corner) and a center position for searching inside the candidate box. Then, IoU(r,g) is introduced, where r represents the current area and g represents the ground truth. At this time, a reward function is defined as follows:

[0077] R r (s t ,s t+1 )=IoU(r t+1 ,g)-IoU(r t ,g) (6)

[0078] Among them, at time t, the area r t The state is s t Then, after an action A in the action set, region r t becomes r t+1 , the state is determined by s t becomes s t+1 If after an action, the IoU decreases instead of increases, a -1 reward value is fed back to the agent, otherwise, a +1 reward value is given to the agent. By controlling the reward value, the agent gradually learns how to obtain the maximum reward, that is, gradually approaches and finally determines the exact position of the target. For the above reinforcement learning process, a deep Q network is used for optimization and solution. Finally, the target area obtained by reinforcement learning is extracted as the final candidate area.

[0079] Step 3: Multi-scale feature fusion and detection includes the following steps:

[0080] S1. Perform block sampling on the candidate area obtained in step 2, and the dimension of each block is x×x.

[0081] S2. Perform unsupervised training on the collected blocks using an autoencoder, where the parameters obtained from the training are used to initialize the neural network; due to the inconsistent scale of remote sensing images, the target scale is not uniform, which makes it difficult to detect. Therefore, a multi-scale and multi-channel feature fusion model is constructed to further enhance the feature representation ability of the target. For the target candidate area obtained in step 2, unsupervised learning is used to learn the initial value of the feature extractor. Input it into an encoder encoder. The encoding can be understood as a mapping and reconstruction of the remote sensing image input, which is another effective representation of the original data information. Then, the decoder decoder is used to decode the previous encoding and output a message. The goal is to output this information consistent with the initial remote sensing image input signal. Therefore, the encoder and decoder parameters in the autoencoder can be effectively learned, trained and adjusted to minimize the reconstruction error between the input data of the remote sensing image and the output data features after mapping, that is, to reproduce the input information to the maximum extent.

[0082] Then, training is performed layer by layer, and the second expression of the original input remote sensing image information can be obtained. At this point, the input signal of the remote sensing image is reconstructed through learning and training, and a feature that can well represent the input remote sensing image is obtained. This feature can represent the original input signal to the greatest extent. The specific steps of unsupervised training using the autoencoder are as follows: n blocks are extracted from the candidate area, and the dimension of each block is x×x, where x represents the size of the convolution kernel. Each block x×x can be transformed into an N-dimensional vector, where N=x×x×3. Therefore, a matrix N×n is obtained, in which the column vector represents a block v input ;

[0083] An input vector v input Perform the following processing:

[0084] p=f(W1v input +b1) (7)

[0085] Among them, W1 is the weight of the autoencoder network, b1 is its bias, and f() is the activation function. The weight internalizes the distribution law and deep mapping relationship of the target feature, transforms it into a feature map and outputs the corresponding feature vector. Then, the vector is decoded by the following formula:

[0086] q=W2 T ·p+b2 (8)

[0087] Among them, W2 is the decoding network weight matrix, b2 is the decoding bias, and reconstruction is performed through encoding and decoding. At this time, the loss function of the network model is defined as the reconstruction error between the original target and the reconstructed target. After that, the feature extractor is trained by minimizing the following loss function:

[0088]

[0089] S3. Perform scale transformation on the candidate area to form images of three scales in total, and construct a three-channel convolutional neural network using the method in S2.

[0090] S4. After the unsupervised training of the three-channel convolutional neural network is completed, the unsupervised trained model is used as a feature extractor.

[0091] S5. Use the feature extractor in S4 to extract and represent features of the candidate region, construct a fused feature vector, and enhance the expression performance of the model.

[0092] S6. Use the fused feature vector and label information to train the classifier.

[0093] S7. Extract features of the candidate area through the trained feature extractor, and fuse the features of the three channels.

[0094] S8. Classify the fused features to complete target detection.

[0095] Therefore, the present invention adopts the above-mentioned remote sensing image target detection method based on visual saliency and multi-scale feature fusion to solve the problem of remote sensing image target detection under conditions of complex background and inconsistent scale.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A remote sensing image target detection method based on visual saliency and multi-scale feature fusion, characterized in that: The following steps are involved: Step 1: Generate saliency map; Step 2: Extraction of target candidate regions: Coarse positioning of the target candidate frame based on the closed contour characteristics of the target object; The step 2 of roughly positioning the target candidate frame according to the closed contour feature of the object target specifically includes the following steps: S1, calculate the gradient map corresponding to the remote sensing image, and calculate and solve its gradient amplitude; S2, scale the gradient amplitude map initially obtained in the remote sensing image, normalize it to a scale of 10*10 and use it as a gradient feature; S3, extracting sample features of the target and background in the remote sensing image, inputting them into the SVM to train the classifier, so as to distinguish the features of the target and non-target, so as to determine whether there is a possible target in the remote sensing image; S4, use the classifier trained in S3 to search the remote sensing image to obtain a series of target areas, use non-maximum suppression to select the candidate box that is most likely to contain the target. After the target candidate box is selected, the deep reinforcement learning method is used to gradually learn how to accurately locate the target from feedback and incentives to solve the problem that the candidate box is not accurate enough in locating the target. Step 3: Multi-scale feature fusion and detection.

2. The method for remote sensing image target detection based on visual saliency and multi-scale feature fusion according to claim 1, characterized in that: The generation of the saliency map in step 1 is specifically as follows: The visual saliency of the remote sensing image is calculated, and the saliency map is generated using global color contrast. If the color of a pixel in the remote sensing image is different from the colors of the surrounding pixels, the pixel is considered to have a certain degree of saliency. If the color of the pixel is similar to that of the surrounding pixels, the pixel is considered to be non-saliency. If the color of a part or target in the remote sensing image is the same or similar, the colors of the surrounding pixels are similar. At this time, the boundary around the pixel and its distance from the center of the remote sensing image are combined to comprehensively consider whether the pixel value is saliency; Define a pixel I in the remote sensing image I k The significance value of: Among them, D(I k ,I i ) is pixel I k and I i The color distance metric of the remote sensing image pixel is expressed as: Among them, C l is the remote sensing image pixel I k The color value of the remote sensing image, n represents the number of colors, f j Indicates color C j The probability of appearing in the remote sensing image is calculated, and then the color channel is quantized into m values. The quantized remote sensing image color is a dimensionality reduction processing of the remote sensing image. At the same time, the remote sensing image further ignores the colors with low occurrence frequency, reduces the sensitivity to low-frequency color signals, enhances the robust performance of remote sensing image processing, selects n colors with the highest occurrence frequency, and then calculates the weight value of each color.

3. The method for remote sensing image target detection based on visual saliency and multi-scale feature fusion according to claim 2, characterized in that: The color weight value is calculated as follows: First, define F j As shown below: A smaller F j The value indicates that the pixels of the same color in the remote sensing image are close to each other and show a high degree of aggregation. Ik Represents pixel I k The position in the remote sensing image, A j Indicates that there is a certain same color C in the remote sensing image j The center position of all pixels, N j Indicates that the remote sensing image has a certain same color C j The number of all pixels; Secondly, the weight value of the selected color is expressed as: w j =exp(-F j ) (4) The final pixel P k The saliency value in remote sensing images is expressed as 4. The method for remote sensing image target detection based on visual saliency and multi-scale feature fusion according to claim 3, characterized in that: Using deep reinforcement learning methods to gradually learn how to accurately locate the target from feedback and incentives, the steps are as follows: A regular quadrilateral with a side length smaller than the target candidate box is used to search inside the candidate box. An action set A1-A5 is defined to represent the four extreme positions and one center position for searching inside the candidate box. Then, IoU(r,g) is introduced, where r represents the current area and g represents the groundtruth. At this time, a reward function is defined as follows: R r (s t , s t+1 ) = IoU(r t+1 , g) -IoU(r t , g) (6) Among them, at time t, the area r t The state is s t Then, after an action A in the action set, region r t becomes r t+1 , the state is determined by s t becomes s t+1 If after an action, the IoU decreases instead of increases, a -1 reward value is fed back to the agent, otherwise, a +1 reward value is given to the agent. By controlling the reward value, the agent gradually learns how to obtain the maximum reward, that is, gradually approaches and finally determines the exact position of the target. For the above reinforcement learning process, a deep Q network is used for optimization and solution. Finally, the target area obtained by reinforcement learning is extracted as the final candidate area.

5. The method for remote sensing image target detection based on visual saliency and multi-scale feature fusion according to claim 4, characterized in that: The four extreme directions are the upper left corner, the upper right corner, the lower left corner and the lower right corner.

6. The method for remote sensing image target detection based on visual saliency and multi-scale feature fusion according to claim 1, characterized in that: The multi-scale feature fusion and detection in step 3 specifically includes the following steps: S1. Perform block sampling on the candidate area obtained in step 2, and the dimension of each block is x×x; S2, performing unsupervised training on the collected blocks using an autoencoder, wherein the parameters obtained from the training are used to initialize the neural network; S3, scale the candidate area to form images of three scales, and construct a convolutional neural network of three channels using the method in S2; S4. After the unsupervised training of the three-channel convolutional neural network is completed, the unsupervised trained model is used as a feature extractor; S5, using the feature extractor in S4 to extract and represent the features of the candidate region, construct a fused feature vector, and enhance the expression performance of the model; S6, training a classifier using the fused feature vector and label information; S7, extracting features of the candidate region through the trained feature extractor, and fusing the features of the three channels; S8. Classify the fused features to complete target detection.

7. The method for remote sensing image target detection based on visual saliency and multi-scale feature fusion according to claim 6, characterized in that: The unsupervised training using the autoencoder in step S2 is specifically as follows: n blocks are extracted from the candidate area, and the dimension of each block is x×x, where x represents the size of the convolution kernel. Each block x×x can be transformed into an N-dimensional vector, where N=x×x×3. Therefore, a matrix N×n is obtained, in which the column vector represents a block v input ; An input vector v input Perform the following processing: p=f(W1v input +b1) (7) Among them, W1 is the weight of the autoencoder network, b1 is its bias, and f() is the activation function. The weight internalizes the distribution law and deep mapping relationship of the target feature, transforms it into a feature map and outputs the corresponding feature vector. Then, the vector is decoded by the following formula: q=W2 T ·p+b2 (8) Among them, W2 is the decoding network weight matrix, b2 is the decoding bias, and reconstruction is performed through encoding and decoding. At this time, the loss function of the network model is defined as the reconstruction error between the original target and the reconstructed target. After that, the feature extractor is trained by minimizing the following loss function:

Images