Remote sensing image authenticity identification method and system based on feature joint learning

Abstract

The application provides a remote sensing image authenticity identification method and system based on feature joint learning, comprising the following steps: S1, inputting remote sensing image data and setting a training period; S2, extracting manual features from each remote sensing image data, including color richness, definition and image texture; S3, constructing a CFANet, unifying the feature map sizes of different convolution layers through pooling, and then extracting deep learning multi-scale features through a series connection operation; S4, fusing the manual features and the deep learning multi-scale features to generate a new full connection layer and train a model; and S5, extracting first-order statistical features and second-order statistical features, making a multi-order information fusion decision, and outputting a remote sensing image authenticity detection result. The application solves the problem of deep counterfeit identification of satellite remote sensing images, achieves accurate identification results in various remote sensing scenes, improves the identification accuracy, and effectively improves the data security and application value of satellite remote sensing images.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, and more specifically, to a method and system for identifying the authenticity of remote sensing images based on joint feature learning. Background Technology

[0002] In recent years, significant progress has been made in image forgery tasks, mainly divided into forgery localization and forgery detection. Forgery localization primarily determines the precise location where the original image has been altered, while forgery detection aims to verify the overall authenticity of the image. By integrating deep learning-based forgery localization and detection techniques, excellent performance in distinguishing genuine from fake images has been achieved in many forgery detection tasks. Currently, convolutional neural networks (CNNs), generative adversarial networks (GANs), and diffusion models are widely used in deep forgery image detection. CNN-based models mainly utilize convolutional layers for effective feature extraction and integrate attention modules into the neural network architecture to enhance global feature extraction capabilities. GAN-based models primarily utilize generator and discriminator training strategies to enhance their ability to generate and detect forged images.

[0003] In the field of computer vision, some general-purpose models have been developed to identify complex forgery operations. For example, models that utilize tampered boundary artifacts and noisy views to learn semantically independent features can adapt to various types of operations. However, due to differences in image acquisition methods and perception technologies, models designed for natural images may not fully meet the needs of remote sensing image forgery localization. Furthermore, the performance of existing methods for identifying the authenticity of natural images can vary depending on the content and manipulation of the forged image, limiting their ability to comprehensively address all types of images and forgery detection. Therefore, in the field of remote sensing, there is currently a lack of specific methods to identify high-quality forged images generated by generative models. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for identifying the authenticity of remote sensing images based on joint feature learning.

[0005] A remote sensing image authenticity identification method based on joint feature learning, provided by the present invention, includes: Step S1: Input remote sensing image data and set the training period; The remote sensing image data includes real remote sensing image data and fake remote sensing image data; Step S2: Extract handcrafted features from each remote sensing image data, including color richness, sharpness, and image texture; Step S3: Construct CFANet, unify the feature map size of different convolutional layers through pooling, and extract multi-scale features of deep learning through concatenation operations; Step S4: Fuse handcrafted features and deep learning multi-scale features to generate a new fully connected layer and train the model; Step S5: Use the trained model to extract first-order and second-order statistical features, perform multi-order information fusion decision-making, and output the authenticity detection results of the remote sensing image.

[0006] Preferably, step S1 includes: A dataset of N input data Divided into training set and verification set ; in, , , These represent the input dataset, training set, and validation set, respectively. This represents the image features of the r-th data point input to the initialization module; This represents the image label of the r-th data input to the initialization module; This represents the image feature of the j-th data point in the training set; This represents the image label of the j-th data point in the training set; This represents the image feature of the p-th data point in the validation set; This represents the image label of the p-th data in the validation set; Indicates the number of training samples. This indicates the number of samples in the validation set.

[0007] Preferably, step S2 includes the following sub-steps: Step S2.1: Extract spatial domain features from image data ; Spatial domain features ; = ; = ; = ; = ; = ; in, This represents the color richness index of the i-th image; This represents the number of different colors in the i-th image; This represents the total number of pixels in the i-th image; This represents the second angular moment of the i-th image based on the gray-level co-occurrence matrix (GLCM). Represents the gray-level co-occurrence matrix; It is the number of gray levels in the i-th image; This represents the contrast of the i-th image based on the gray-level co-occurrence matrix (GLCM). This represents the entropy of the i-th image based on the gray-level co-occurrence matrix (GLCM). This represents the inverse difference moment of the i-th image based on the gray-level co-occurrence matrix (GLCM); Step S2.2: Extract color histogram features from image data ; Step S2.3: Extract image frequency domain features ; Step S2.4: Concatenate to obtain the hand-crafted feature vector ; ; in, The spatial domain features are obtained for the i-th image S1; The color histogram features are obtained for the i-th image S2; The spatial domain features of the i-th image S1 are obtained.

[0008] Preferably, the color histogram feature include: ; Let 'c' represent the color channel (R, G, B). = ; = ; = ; = ; = ; = ; = ; = ; ; ; ; ; ; ; in, This represents the mean of the color histogram of the i-th image, c. This represents the standard deviation of the color histogram of the i-th image; Represents the color histogram of the i-th image. Figure 3 The central moment of the first order; This represents the total number of pixels in the i-th image; This represents the value of color channel c corresponding to the histogram of the k-th color channel; This represents the probability of gray level k appearing; This represents the number of gray levels in the i-th image; This represents the mean of the grayscale histogram of the i-th image; This represents the standard deviation of the grayscale histogram of the i-th image; This represents the skewness of the grayscale histogram of the i-th image; Represents the kurtosis of the grayscale histogram of the i-th image; This represents the entropy of the grayscale histogram of the i-th image; This represents the block effect index of the i-th image; Represents the i-th image; Indicates the height of the i-th image; Indicates the width of the i-th image; This represents the i-th image texture index; This represents the Sobel gradient in the x-direction of the i-th image; This represents the Sobel gradient in the y-direction of the i-th image; Represents the Laplacian index of the i-th image; This represents the result of the Laplacian operator for the i-th image.

[0009] Preferably, the image frequency domain features include: ; = ; = ; = ; = ; Among them, F Represents the second-order angular moment in the frequency domain of the i-th image; Represents the co-occurrence matrix based on the frequency domain; F is the number of gray levels in the i-th image; This represents the frequency domain contrast of the i-th image based on the frequency domain co-occurrence matrix; This represents the frequency domain entropy of the i-th image based on the frequency domain co-occurrence matrix; It represents the frequency domain inverse difference moment of the i-th image based on the frequency domain co-occurrence matrix.

[0010] Preferably, step S3 includes the following sub-steps: Step S3.1: Construct a new CFANet model based on A-ConvNet, use pooling operations to unify the size of different feature maps, and use concatenation operations to aggregate convolutional feature maps in feature spaces of different scales; Step S3.2: Expand the feature map tensors of different convolutional layers into vectors to represent the deep learning features of the i-th image.

[0011] Preferably, step S4 includes the following sub-steps: Step S4.1: Stack handcrafted features and deep features to obtain jointly learned features. ; ; in, This represents the handcrafted feature vector of the i-th image obtained in step S2; This represents the depth feature vector obtained by unfolding the FM4 tensor of the i-th image obtained in step S3; Step S4.2: Perform model training; Construct a classification task with the following loss function: ; in, Represents the cross-entropy loss function; This represents the probability distribution of the model output; Indicates real label One-hot encoded vector; This represents the parameters that the model needs to learn during training; Indicates the number of training set data; This represents the j-th image in the training set; This represents the label of the j-th image in the training set.

[0012] Preferably, step S5 includes: Deep learning features , and and the resulting joint learning features This is called a first-order statistical characteristic; Input samples from the training set, validator, and test set. , We obtain the first-order statistical feature vectors of these features in FM1, FM2, FM3, FM4 and the jointly learned features. We then use the obtained training set data to learn five SVM classifiers, and obtain five soft classification results for the validation set and test set through these classifiers.

[0013] Preferred, assuming This represents the soft classification result obtained by using SVM on the validation set samples under four convolutional feature maps; a target function is constructed to learn the weights, namely, minimizing the mean squared error between the fused result and the ground truth on the validation set. ; ,

[0014] Solve this constrained optimization problem to find the optimal value. ; After obtaining the weights, the soft classification results are fused using the weighted arithmetic mean rule. : ; in, , These represent the input training set and validation set, respectively. , and Let represent the vectors obtained by expanding the feature maps FM1, FM2, FM3, and FM4 of the i-th image, respectively. Indicates real label One-hot encoded vector; Denotes the Euclidean norm; This represents the membership degree of the i-th image as a forgery in the p-th SVM classifier of the first-order statistical features; This represents the degree to which the i-th image belongs to the true membership in the p-th SVM classifier of the first-order statistical features; Indicates the number of validation set data; This represents the weight of the k-th classification result of the first-order statistical feature.

[0015] The covariance matrix of a tensor tensor (FM) is commonly used to reflect second-order statistical characteristics. In practice, it can also be used to construct classification models with good results. For an FM... ∈ ,in W , H and D These are the width, height, and depth. We can obtain... There are vectors, which are pairs along the third dimension. The vectors are used to construct a feature matrix. Therefore, its covariance matrix C ∈ It can be calculated as follows: ; Matrix logarithm operations are used to transform the covariance matrix from manifold space to Euclidean space, thereby obtaining second-order statistical features for downstream classification tasks. Let... U And Σ are respectively C Given the eigenvector matrix and eigenvalue matrix, then C = U Σ At this point, the logarithm of the covariance matrix can be obtained through... The calculation yielded the result. It can be seen that... Since it is a symmetric matrix, the elements of the upper triangular matrix are vectorized to obtain a vector s∈ .

[0016] Assume the covariance matrices of the feature maps of different convolutional layers are expressed as follows: C1∈ C2∈ C3∈ and C4∈ ; For the data input into the CFANet model, second-order statistical feature vectors of FM1, FM2, FM3, and FM4 will be obtained: s1∈ ,s2∈ ,s3∈ and s4∈ ; Four SVM classifiers are learned using the training set data represented by second-order statistical features. These classifiers yield four soft classification results for the validation set. The second-order statistical features of different convolutional feature maps also have a certain degree of complementarity. Fusing them at the decision level will enhance the reliability of the soft classification results.

[0017] assumed These represent the soft classification results of the validation set obtained by these classifiers. Their weights... This can also be obtained by minimizing the mean squared error between the fusion result and the true value on the validation set, i.e. ,

[0018] After obtaining the weights, the soft classification results are fused using the weighted arithmetic mean rule. : ; in, Indicates real label One-hot encoded vector; Denotes the Euclidean norm; This represents the membership degree of the i-th image in the p-th SVM classifier of the second-order statistical features, belonging to category 0 (forgery); This represents the membership degree of the i-th image in the p-th SVM classifier of the second-order statistical features, belonging to class 1 (true). Indicates the number of validation set data; This represents the weight of the k-th classification result of the second-order statistical feature; S3, soft classification result fusion; The soft classification results obtained using first-order and second-order statistical features and the soft classification results obtained using the CFANet model are complementary to some extent. Effectively fusing these soft classification results is expected to further improve classification accuracy.

[0019] assumed This represents the soft classification result of the validation set obtained by the softmax layer of the joint learning model of handmade features and deep features.

[0020] For validation samples It can be combined , and Classification decision is made. The soft classification results are generated using an SVM classifier based on first-order and second-order statistical features. and And the soft classification results obtained by the softmax layer of the joint learning model of handcrafted features and deep features. Due to varying reliability, weighting is necessary for fusion. Here, we still use the weighted arithmetic mean rule for fusion, where the weights... We can learn it using the following formula: ,

[0021] Solve this constrained optimization problem to find the optimal value. The weighted arithmetic mean fusion result can be obtained as follows:

[0022] For the target to be identified ,according to The element with the highest probability value is the identification result of that sample.

[0023] A remote sensing image authenticity identification system based on joint feature learning, according to the present invention, includes: The initialization module is used to input real and fake remote sensing image data and set the training period; The manual feature extraction module extracts manual features of color richness, sharpness, and image texture from various remote sensing image data. The deep feature extraction module constructs CFANet, unifies the feature map size of different convolutional layers through pooling, and extracts multi-scale features of deep learning through concatenation operations to obtain soft classification results of first-order and second-order statistical features of different convolutional feature layers. The evidence fusion module estimates the relative weights of each convolutional feature map and softmax layer classification result based on accuracy, estimates the final weights of each classification result, performs a weighted arithmetic average of the results, conducts multi-order information fusion decision-making, and finally obtains the sample category based on the fusion result.

[0024] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention adopts an innovative feature processing strategy, which integrates human-made features with features extracted by deep networks and performs joint learning. This retains the accurate characterization of local details by human-made features and also leverages the ability of deep features to capture global semantics. This expands the receptive field of the model, obtains richer image information, and enables more thorough parameter updates and more efficient gradient transfer during backpropagation, thereby significantly improving the optimization effect.

[0025] 2. This invention innovatively optimizes the deep feature extraction module and makes targeted improvements to the A-CovNet structure. When processing feature maps output by different convolutional layers, pooling operations are introduced to unify the feature map sizes of different convolutional layers. Furthermore, the features of multiple convolutional layers are organically integrated through concatenation operations, which fully extracts multi-scale features from deep learning, enhances feature discriminativeness, and improves robustness.

[0026] 3. This invention employs multi-level information decision-making fusion. First, it systematically extracts first- and second-order statistical feature vectors to construct a rich feature representation system. Based on these features, a new classifier is trained using Support Vector Machine (SVM) and used to make category decisions. The probability output of the jointly learned model of handcrafted features and deep features is arithmetically weighted and fused with the probability output of these new classifiers, further improving the model's performance.

[0027] 4. Strong domain-specificity: For the identification of forgery in optical remote sensing images, manual features directly reflect traces of tampering in remote sensing images, improving the accuracy of identification.

[0028] 5. Dynamic weight optimization mechanism: In multi-source information fusion, the weights are dynamically learned by minimizing the mean square error of the validation set, rather than being preset to a fixed value.

[0029] 6. Scalability and flexibility: CFANet's modular design allows for the replacement or addition / removal of convolutional layers to adapt to remote sensing images of different resolutions or complexities; the multi-stage fusion framework can easily integrate new features or classifiers, facilitating subsequent improvements.

[0030] Other beneficial effects of the present invention will be explained in detail through the introduction of specific technical features and technical solutions in specific embodiments. Those skilled in the art should be able to understand the beneficial technical effects brought about by these technical features and technical solutions through the introduction of these technical features and technical solutions. Attached Figure Description

[0031] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of the remote sensing image authenticity identification method and system flow based on feature joint learning according to the present invention.

[0032] Figure 2 This is a flowchart of the remote sensing image authenticity identification method and system model based on feature joint learning according to the present invention.

[0033] Figure 3 This is a schematic diagram of a remote sensing image dataset sample for this invention. Detailed Implementation

[0034] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0035] Reference Figure 1 As shown, a remote sensing image authenticity identification method based on joint feature learning includes: Step 1: Initialize the module by inputting real and fake remote sensing image data and setting the training period; The initialization module input data includes images and their corresponding labels (i.e., true or false); If the initialization module inputs N data points to form a dataset It can be divided into training sets. and verification set ; , , These represent the input dataset, training set, and validation set, respectively. This represents the image features of the r-th data point input to the initialization module; This represents the image label of the r-th data input to the initialization module; This represents the image feature of the j-th data point in the training set; This represents the image label of the j-th data point in the training set; This represents the image feature of the p-th data point in the validation set; This represents the image label of the p-th data in the validation set; Indicates the number of training samples. This indicates the number of samples in the validation set.

[0036] To obtain the best identification results from optical remote sensing images, the most direct strategy is to improve quality by learning more features and more types of classifiers. Training classifiers with richer feature vectors and combining the soft outputs of multiple classifiers yields more reliable outputs, and the learned features become more robust.

[0037] Step 2: Extract manual features from remote sensing image data, such as color richness, sharpness, and image texture; We select unique and common indicators from remote sensing images for manual feature extraction (block artifacts BIQ, texture TIQ, Laplacian index LIQ, etc.), directly targeting common traces of remote sensing image tampering (such as compression artifacts, edge discontinuities, etc.). Specifically, this includes: S1. Extract Spatial Domain Feature (SDF) from image data. ; The spatial domain The calculation method is as follows:

[0038]

[0039]

[0040]

[0041]

[0042]

[0043] In the formula, This represents the color richness index of the i-th image; This represents the second angular moment of the i-th image based on the gray-level co-occurrence matrix (GLCM). This represents the contrast of the i-th image based on the gray-level co-occurrence matrix (GLCM). This represents the entropy of the i-th image based on the gray-level co-occurrence matrix (GLCM). This represents the inverse difference moment of the i-th image based on the gray-level co-occurrence matrix (GLCM); This represents the number of different colors in the i-th image; This represents the total number of pixels in the i-th image; It is the number of gray levels in the i-th image; Represents the gray-level co-occurrence matrix; S2. Extract the color histogram feature (CHF) from the image data. ; Color histogram features The calculation method is as follows:

[0044] Let 'c' represent the color channel (R, G, B).

[0045]

[0046]

[0047]

[0048]

[0049]

[0050]

[0051]

[0052]

[0053]

[0054]

[0055]

[0056]

[0057]

[0058] in, This represents the mean of the color histogram of the i-th image, c. This represents the standard deviation of the color histogram of the i-th image; Represents the color histogram of the i-th image. Figure 3 The central moment of the first order; This represents the mean of the grayscale histogram of the i-th image; This represents the standard deviation of the grayscale histogram of the i-th image; This represents the skewness of the grayscale histogram of the i-th image; Represents the kurtosis of the grayscale histogram of the i-th image; This represents the entropy of the grayscale histogram of the i-th image; This represents the block effect index of the i-th image; This represents the i-th image texture index; Represents the Laplacian index of the i-th image; This represents the total number of pixels in the i-th image; This represents the number of gray levels in the i-th image; This represents the value of color channel c corresponding to the histogram of the k-th color channel; This represents the probability of gray level k appearing; Represents the i-th image; Indicates the height of the i-th image; Indicates the width of the i-th image; This represents the Sobel gradient in the x-direction of the i-th image; This represents the Sobel gradient in the y-direction of the i-th image; This represents the result of the Laplacian operator for the i-th image; S3. Extract image frequency domain features (FDF) ; Frequency domain characteristics The calculation method is as follows:

[0059]

[0060]

[0061]

[0062]

[0063] Among them, F F represents the second-order angular moment in the frequency domain of the i-th image; This represents the frequency domain contrast of the i-th image based on the frequency domain co-occurrence matrix; This represents the frequency domain entropy of the i-th image based on the frequency domain co-occurrence matrix; This represents the frequency-domain inverse difference moment of the i-th image based on the frequency-domain co-occurrence matrix; Represents the co-occurrence matrix based on the frequency domain; It is the number of gray levels in the i-th image; S4. Concatenate the data to obtain the Handcrafted Feature (HF) vector. ; Spatial domain features obtained based on S1, S2, and S3 Color histogram features Frequency domain characteristics To obtain more comprehensive handcrafted features for training and judgment, handcrafted feature vectors are constructed. It is pieced together from the above three handcrafted features, allowing you to experience a wider field of wilderness.

[0064] Handmade feature vectors The calculation method is as follows:

[0065] in, The spatial domain features are obtained for the i-th image S1; The color histogram features are obtained for the i-th image S2; The spatial domain features are obtained for the i-th image S1; Step 3: Construct a convolutional feature aggregation network (CFANet), use pooling operations to unify the feature map size of different convolutional layers, and then extract multi-scale features through concatenation operations; Discriminative information in different convolutional layers is complementary. To effectively utilize the complementary information in shallow and deep convolutional feature maps, a convolutional feature aggregation module is proposed, optimizing the A-ConvNet model and performing deep feature extraction to obtain global image information, thereby targeting image tampering globally. Specifically, this includes: S1 and CFANet setup; The A-ConvNet model focuses on extracting high-level semantic information of the target. However, its utilization of shallow convolutional features is relatively limited. A novel Convolutional Feature Aggregation Network (CFANet) model is built upon A-ConvNet. It employs pooling operations to unify the size of different feature maps and uses concatenation operations to aggregate convolutional feature maps from different scale feature spaces. The proposed CFANet model structure is as follows: Figure 2 The Deep Feature Extraction Module is shown below.

[0066] The CFANet model input was set to 88×88, and five convolutional layers were added. Since pooling does not increase the number of parameters to be learned and has a large receptive field, average pooling and max pooling operations were used to unify the width and height of different convolutional feature maps (FM).

[0067] To effectively aggregate feature maps of different sizes generated by multiple convolutional layers, the width and height of FM are uniformly set to 3×3.

[0068] For CFANet, FM1 is 42×42×16 in size. Using max pooling and average pooling operations without padding, the width and height of the new feature map will become 3×3.

[0069] The size of FM2 is 19×19×32. Using max pooling and average pooling operations with padding, the width and height of the new feature map will become 3×3.

[0070] The size of FM3 is 7×7×64. Using average pooling with padding, the width and height of the new feature map will become 3×3.

[0071] This resulted in three new feature maps with sizes of 3×3×16, 3×3×32, and 3×3×64, respectively.

[0072] Then, the convolutional feature maps of different lengths are concatenated using a stacking operation. At this point, the size of FM4 will become 3×3×16 + 3×3×32 + 3×3×64 + 3×3×112 = 3×3×240.

[0073] S2, Deep Feature Extraction; The feature maps of different convolutional layers can often be regarded as features of downstream classification tasks. The feature maps generated by the CFANet model can be used to prepare for training new classifiers and making category decisions.

[0074] Let FM1, FM2, FM3, and FM4 represent the feature maps of different convolutional layers, and the corresponding feature map tensors are represented as follows: ∈ , ∈ , ∈ and ∈ For the i-th image, these tensors can be expanded into vectors, represented as follows: , and Their dimensions are respectively 42×42×16 = 28224, 19×19×32 = 11552, 7×7×64 = 3136 and 3×3×240 = 2160. These vectors can represent the deep learning features of the i-th image.

[0075] Step 4: Integrate manual features and deep features to obtain new discriminative features; The handcrafted feature vectors obtained in step two and the deep feature vectors obtained in step three are combined to form a new fully connected layer. This enriches the information carried by the original deep feature vectors, broadens the receptive field, and allows for better optimization of parameters during model training.

[0076] S1. Stack handcrafted features and deep features to obtain jointly learned features. ; ; in, This represents the handcrafted feature vector of the i-th image obtained in step two; This represents the depth feature vector obtained by unfolding the FM4 tensor of the i-th image obtained in step three. S2, Model Training; For this classification task, the jointly learned features obtained from S1 are used. As a fully connected layer, the handcrafted features, due to their localized specificity, enable... Obtaining expert-specific information beyond the global information is more beneficial for learning this classification task, and is expected to improve the model's generalization ability and solve the overfitting problem.

[0077] Construct a classification task with the following loss function: ; Represents the cross-entropy loss function; This represents the probability distribution of the model output; Indicates real label One-hot encoded vector; This represents the j-th image in the training set; This represents the label of the j-th image in the training set; This represents the parameters that the model needs to learn during training; Indicates the number of training set data; Step 5: Extract the first-order and second-order statistical features from the deep convolutional feature map to achieve the fusion of multi-order information and decision-making; A single feature vector can only reflect the basic distribution of data and cannot capture higher-order correlations between features, leading to overfitting. Therefore, employing multi-order information fusion decision-making, combining first- and second-order statistical features to complementarily reflect comprehensive image information, can improve the ability to identify complex tampering patterns and enhance robustness. Figure 2 The model shown extracts first-order and second-order statistical features, and then performs multi-order information fusion decision-making. Specifically, this includes: S1, Mining of First-Order Statistical Feature Information; Step 3: Deep Learning Features , and and the joint learning features obtained in step four These are called first-order statistical features, which can be directly used to train classifiers (such as SVM). First-order statistical features obtained using different convolutional feature maps are usually complementary, so the soft classification results of the target to be identified obtained using these features are also complementary. In this case, they can be combined from the perspective of decision-level fusion.

[0078] Input samples from the training set, validator, and test set. , This allows us to obtain the first-order statistical feature vectors of these features in FM1, FM2, FM3, FM4, and the jointly learned features. Then, we use the obtained training set data to learn five SVM classifiers, and obtain five soft classification results for the validation and test sets through these classifiers.

[0079] assumed This represents the soft classification result obtained by using SVM on the validation set samples under four convolutional feature maps. Since these first-order statistical features are extracted from the same object and are highly correlated, the resulting soft classification results are also highly correlated. Furthermore, the feature maps of different convolutional layers and the jointly learned features typically have different importance for downstream classification tasks. Therefore, these soft classification results should not be treated equally during fusion; their weights must be considered to obtain high-quality fusion results. Here, an objective function is constructed to learn the weights by minimizing the mean squared error between the fusion result and the ground truth on the validation set. ;

[0080]

[0081] Solve this constrained optimization problem to find the optimal value. After obtaining the weights, the soft classification results are fused using a weighted arithmetic mean rule. : : ; in, , These represent the input training set and validation set, respectively. , and These represent the vectors obtained by expanding the feature maps FM1, FM2, FM3, and FM4 of the i-th image in step three, respectively. Indicates real label One-hot encoded vector; Denotes the Euclidean norm; This represents the membership degree of the i-th image in the p-th SVM classifier of the first-order statistical features, belonging to category 0 (forgery). This represents the membership degree of the i-th image in the p-th SVM classifier of the first-order statistical features, belonging to class 1 (true). Indicates the number of validation set data; This represents the weight of the k-th classification result of the first-order statistical feature; S2, Second-order statistical feature mining The covariance matrix of a tensor tensor (FM) is commonly used to reflect second-order statistical characteristics. In practice, it can also be used to construct classification models with good results. For an FM... ∈ ,in W , H and D These are the width, height, and depth. We can obtain... There are vectors, which are pairs along the third dimension. The vectors are used to construct a feature matrix. Therefore, its covariance matrix C ∈ It can be calculated as follows: ; Matrix logarithm operations are used to transform the covariance matrix from manifold space to Euclidean space, thereby obtaining second-order statistical features for downstream classification tasks. Let... U And Σ are respectively C Given the eigenvector matrix and eigenvalue matrix, then C = U Σ At this point, the logarithm of the covariance matrix can be obtained through... The calculation yielded the result. It can be seen that... Since it is a symmetric matrix, the elements of the upper triangular matrix are vectorized to obtain a vector s∈ .

[0082] Assume the covariance matrices of the feature maps of different convolutional layers are expressed as follows: C1∈ C2∈ C3∈ and C4∈ ; For the data input into the CFANet model, second-order statistical feature vectors of FM1, FM2, FM3, and FM4 will be obtained: s1∈ ,s2∈ ,s3∈ and s4∈ ; Four SVM classifiers are learned using the training set data represented by second-order statistical features. These classifiers yield four soft classification results for the validation set. The second-order statistical features of different convolutional feature maps also have a certain degree of complementarity. Fusing them at the decision level will enhance the reliability of the soft classification results.

[0083] assumed These represent the soft classification results of the validation set obtained by these classifiers. Their weights... This can also be obtained by minimizing the mean squared error between the fusion result and the true value on the validation set, i.e. ,

[0084] After obtaining the weights, the soft classification results are fused using the weighted arithmetic mean rule. : ; in, Indicates real label One-hot encoded vector; Denotes the Euclidean norm; This represents the membership degree of the i-th image in the p-th SVM classifier of the second-order statistical features, belonging to category 0 (forgery); This represents the membership degree of the i-th image in the p-th SVM classifier of the second-order statistical features, belonging to class 1 (true). Indicates the number of validation set data; This represents the weight of the k-th classification result of the second-order statistical feature; S3, soft classification result fusion; The soft classification results obtained using first-order and second-order statistical features and the soft classification results obtained using the CFANet model are complementary to some extent. Effectively fusing these soft classification results is expected to further improve classification accuracy.

[0085] assumed This represents the soft classification result of the validation set obtained by the softmax layer of the joint learning model of handmade features and deep features.

[0086] For validation samples It can be combined , and Classification decision is made. The soft classification results are generated using an SVM classifier based on first-order and second-order statistical features. and And the soft classification results obtained by the softmax layer of the joint learning model of handcrafted features and deep features. Due to varying reliability, weighting is necessary for fusion. Here, we still use the weighted arithmetic mean rule for fusion, where the weights... We can learn it using the following formula: ,

[0087] Solve this constrained optimization problem to find the optimal value. The weighted arithmetic mean fusion result can be obtained as follows:

[0088] For the target to be identified ,according to The element with the highest probability value is the identification result of that sample.

[0089] like Figure 3 As a preferred example, a remote sensing image dataset from Beijing and Seattle is provided, containing both real and fake images, with each sample being an 88×88 pixel slice image. Basic sample information is as follows: Figure 3 As shown.

[0090] A Support Vector Machine (SVM) with a linear kernel was used as the base classifier to obtain the soft classification results and pseudo-labels for the samples. The training period was set to 10. This example uses classification accuracy, recall, and F1 score as algorithm evaluation metrics. The test results are shown in Tables 1 and 2.

[0091] For optical remote sensing image data, this method simultaneously extracts handcrafted and depth features and employs multi-level statistical feature fusion decision-making, improving the performance and stability of target recognition. Compared to other single feature extraction and decision-making algorithms, it achieves higher accuracy, reaching over 95% recognition accuracy. The results demonstrate that the remote sensing image authenticity identification method and system based on feature joint learning is an effective means for accurate identification of optical remote sensing images and has significant practical application value.

[0092] Table 1

[0093] Table 2

[0094] The present invention also provides a remote sensing image authenticity identification system based on feature joint learning. The remote sensing image authenticity identification system based on feature joint learning can be implemented by executing the process steps of the remote sensing image authenticity identification method based on feature joint learning. That is, those skilled in the art can understand the remote sensing image authenticity identification method based on feature joint learning as a preferred embodiment of the remote sensing image authenticity identification system based on feature joint learning.

[0095] According to the present invention, a remote sensing image authenticity identification system based on joint feature learning is provided, which... Figure 2 For example, including: The initialization module is used to input real and fake remote sensing image data and set the training period.

[0096] The manual feature extraction module extracts manual features such as color richness, sharpness, and image texture from various remote sensing image data. The deep feature extraction module constructs CFANet, unifies the feature map size of different convolutional layers through pooling, and extracts multi-scale features of deep learning through concatenation operations to obtain soft classification results of first-order and second-order statistical features of different convolutional feature layers. The evidence fusion module estimates the relative weights of each convolutional feature map and softmax layer classification result based on accuracy, estimates the final weights of each classification result, calculates the weighted arithmetic average of the results, performs multi-order information fusion decision, and then obtains the category of the sample based on the fusion result.

[0097] The input real and fake remote sensing image data includes images and their corresponding labels; The manual feature extraction module includes: Spatial domain feature extraction unit, used to extract spatial domain features of image; The color histogram feature extraction unit is used to extract the color histogram features of the image. The frequency domain feature extraction unit is used to extract frequency domain features from the image. The deep feature extraction module includes: Construct loss units to find the optimal solution for the loss term, and obtain convolutional feature maps and fully connected layers; The statistical feature extraction unit is used to extract first- and second-order statistical features of FM1-FM4; The evidence fusion module includes: The weighted fusion unit is used to classify images based on various types of classifiers trained with first-order and second-order statistical features, obtain soft classification results, and fuse these soft classification results based on the weighted arithmetic mean rule to obtain a fused soft classification result.

[0098] In more preferred examples, the initialization module is input with real and fake remote sensing image data, and the training period is set. The manual feature extraction module extracts spatial and feature data, color histogram features, and frequency domain features from remote sensing image data. The deep feature extraction module constructs CFANet, unifies the feature map size of the convolutional layer through pooling, and extracts multi-scale features of deep learning through concatenation operations to obtain soft classification results of first-order and second-order statistical features of various convolutional feature layers. The evidence fusion module estimates the relative weights of the convolutional feature maps and softmax layer classification results based on accuracy, estimates the final weights of each classification result, performs a weighted arithmetic average of the results, performs multi-order information fusion decision-making, and then obtains the category of the sample based on the fusion result.

[0099] Iterate through all remote sensing image data to obtain the authenticity identification results of all remote sensing images.

[0100] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0101] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for identifying the authenticity of remote sensing images based on joint feature learning, characterized in that, include: Step S1: Input remote sensing image data and set the training period; The remote sensing image data includes real remote sensing image data and fake remote sensing image data; Step S2: Extract handcrafted features from each remote sensing image data, including color richness, sharpness, and image texture; Step S3: Construct CFANet, unify the feature map size of different convolutional layers through pooling, and extract multi-scale features of deep learning through concatenation operations; Step S4: Fuse handcrafted features and deep learning multi-scale features to generate a new fully connected layer and train the model; Step S5: Use the trained model to extract first-order and second-order statistical features, perform multi-order information fusion decision-making, and output the authenticity detection results of the remote sensing image. Step S3 includes the following sub-steps: Step S3.1: Construct a new CFANet model based on A-ConvNet, use pooling operations to unify the size of different feature maps, and use concatenation operations to aggregate convolutional feature maps in feature spaces of different scales; Step S3.2: Expand the feature map tensors of different convolutional layers into vectors to represent the deep learning features of the i-th image; Step S5 includes: Deep learning features , and and the resulting joint learning features This is called a first-order statistical characteristic; Input samples from the training set, validator, and test set. , We obtain the first-order statistical feature vectors of these features in FM1, FM2, FM3, FM4 and the jointly learned features. We then use the obtained training set data to learn five SVM classifiers, and obtain five soft classification results for the validation set and test set through these classifiers. assumed This represents the soft classification result obtained by using SVM on the validation set samples under four convolutional feature maps; a target function is constructed to learn the weights, namely, minimizing the mean squared error between the fused result and the ground truth on the validation set. ; , Solve the constrained optimization problem to find the optimal value. ; After obtaining the weights, the soft classification results are fused using the weighted arithmetic mean rule. ： ； in, , These represent the input training set and validation set, respectively. , and Let represent the vectors obtained by expanding the feature maps FM1, FM2, FM3, and FM4 of the i-th image, respectively. Indicates real label One-hot encoded vector; Denotes the Euclidean norm; This represents the membership degree of the i-th image as a forgery in the p-th SVM classifier of the first-order statistical features; This represents the degree to which the i-th image belongs to the true membership in the p-th SVM classifier of the first-order statistical features; Indicates the number of validation set data; This represents the weight of the k-th classification result of the first-order statistical feature.

2. The remote sensing image authenticity identification method based on joint feature learning according to claim 1, characterized in that, Step S1 includes: A dataset of N input data Divided into training set and verification set ; in, , , These represent the input dataset, training set, and validation set, respectively. This represents the image features of the r-th data point input to the initialization module; This represents the image label of the r-th data input to the initialization module; This represents the image feature of the j-th data point in the training set; This represents the image label of the j-th data point in the training set; This represents the image feature of the p-th data point in the validation set; This represents the image label of the p-th data in the validation set; Indicates the number of training samples. This indicates the number of samples in the validation set.

3. The remote sensing image authenticity identification method based on joint feature learning according to claim 2, characterized in that, Step S2 includes the following sub-steps: Step S2.1: Extract spatial domain features from image data ; Spatial domain features ; = ； = ； = ； = ； = ； in, This represents the color richness index of the i-th image; This represents the number of different colors in the i-th image; This represents the total number of pixels in the i-th image; This represents the second angular moment of the i-th image based on the gray-level co-occurrence matrix (GLCM). Represents the gray-level co-occurrence matrix; It is the number of gray levels in the i-th image; This represents the contrast of the i-th image based on the gray-level co-occurrence matrix (GLCM). This represents the entropy of the i-th image based on the gray-level co-occurrence matrix (GLCM). This represents the inverse difference moment of the i-th image based on the gray-level co-occurrence matrix (GLCM); Step S2.2: Extract color histogram features from image data ; Step S2.3: Extract image frequency domain features ; Step S2.4: Concatenate to obtain the hand-crafted feature vector ; ； in, The spatial domain features are obtained for the i-th image S1; The color histogram features are obtained for the i-th image S2; The spatial domain features of the i-th image S1 are obtained.

4. The remote sensing image authenticity identification method based on feature joint learning according to claim 3, characterized in that, The color histogram feature include: ； Let 'c' represent the color channel (R, G, B). = ； = ； = ； = ； = ； = ； = ； = ； ; ； ； ； ； ； in, This represents the mean of the color histogram of the i-th image, c. This represents the standard deviation of the color histogram of the i-th image; This represents the third central moment of the color histogram of the i-th image; This represents the total number of pixels in the i-th image; This represents the value of color channel c corresponding to the histogram of the k-th color channel; This represents the probability of gray level k appearing; This represents the number of gray levels in the i-th image; This represents the mean of the grayscale histogram of the i-th image; This represents the standard deviation of the grayscale histogram of the i-th image; This represents the skewness of the grayscale histogram of the i-th image; Represents the kurtosis of the grayscale histogram of the i-th image; This represents the entropy of the grayscale histogram of the i-th image; This represents the block effect index of the i-th image; Represents the i-th image; Indicates the height of the i-th image; Indicates the width of the i-th image; This represents the i-th image texture index; This represents the Sobel gradient in the x-direction of the i-th image; This represents the Sobel gradient in the y-direction of the i-th image; Represents the Laplacian index of the i-th image; This represents the result of the Laplacian operator for the i-th image.

5. The remote sensing image authenticity identification method based on joint feature learning according to claim 4, characterized in that, The image frequency domain features include: ； = ； = ； = ； = ； Among them, F Represents the second-order angular moment in the frequency domain of the i-th image; Represents the co-occurrence matrix based on the frequency domain; F is the number of gray levels in the i-th image; This represents the frequency domain contrast of the i-th image based on the frequency domain co-occurrence matrix; This represents the frequency domain entropy of the i-th image based on the frequency domain co-occurrence matrix; It represents the frequency domain inverse difference moment of the i-th image based on the frequency domain co-occurrence matrix.

6. The remote sensing image authenticity identification method based on joint feature learning according to claim 1, characterized in that, Step S4 includes the following sub-steps: Step S4.1: Stack handcrafted features and deep features to obtain jointly learned features. ; ; in, This represents the handcrafted feature vector of the i-th image obtained in step S2; This represents the depth feature vector obtained by unfolding the FM4 tensor of the i-th image obtained in step S3; Step S4.2: Perform model training; Construct a classification task with the following loss function: ; in, Represents the cross-entropy loss function; This represents the probability distribution of the model output; Indicates real label One-hot encoded vector; This represents the parameters that the model needs to learn during training; Indicates the number of training set data; This represents the j-th image in the training set; This represents the label of the j-th image in the training set.

7. A remote sensing image authenticity identification system based on joint feature learning, characterized in that, include: The initialization module is used to input real and fake remote sensing image data and set the training period; The manual feature extraction module extracts manual features of color richness, sharpness, and image texture from various remote sensing image data. The deep feature extraction module constructs CFANet, unifies the feature map size of different convolutional layers through pooling, and extracts multi-scale features of deep learning through concatenation operations to obtain soft classification results of first-order and second-order statistical features of different convolutional feature layers. The evidence fusion module estimates the relative weights of each convolutional feature map and softmax layer classification result based on accuracy, estimates the final weights of each classification result, calculates the weighted arithmetic average of the results, performs multi-order information fusion decision, and finally obtains the category of the sample based on the fusion result. The deep feature extraction module includes: A new CFANet model is built on top of A-ConvNet. Pooling operations are used to unify the size of different feature maps, and concatenation operations are used to aggregate convolutional feature maps in feature spaces of different scales. The feature map tensors of different convolutional layers are unfolded into vectors to represent the deep learning features of the i-th image; The evidence fusion module includes: Deep learning features , and and the resulting joint learning features This is called a first-order statistical characteristic; Input samples from the training set, validator, and test set. , We obtain the first-order statistical feature vectors of these features in FM1, FM2, FM3, FM4 and the jointly learned features. We then use the obtained training set data to learn five SVM classifiers, and obtain five soft classification results for the validation set and test set through these classifiers. assumed This represents the soft classification result obtained by using SVM on the validation set samples under four convolutional feature maps; a target function is constructed to learn the weights, namely, minimizing the mean squared error between the fused result and the ground truth on the validation set. ; , Solve the constrained optimization problem to find the optimal value. ; After obtaining the weights, the soft classification results are fused using the weighted arithmetic mean rule. ： ； in, , These represent the input training set and validation set, respectively. , and Let represent the vectors obtained by expanding the feature maps FM1, FM2, FM3, and FM4 of the i-th image, respectively. Indicates real label One-hot encoded vector; Denotes the Euclidean norm; This represents the membership degree of the i-th image as a forgery in the p-th SVM classifier of the first-order statistical features; This represents the degree to which the i-th image belongs to the true membership in the p-th SVM classifier of the first-order statistical features; Indicates the number of validation set data; This represents the weight of the k-th classification result of the first-order statistical feature.

Images