A land surveying and mapping data classification method based on self-growing convolutional neural network

Abstract

The application discloses a land surveying and mapping data classification method based on a self-growth convolutional neural network, and comprises the following steps: constructing a convolutional neural network model, wherein the model comprises a spectral feature extraction module and a spatial feature extraction module for extracting corresponding feature information from the spectrum and space, a feature fusion module for fusing the spectral and spatial feature information, and a network output module for classifying the fused feature information; training the model by using a labeled data set; after each training, selecting part of unlabeled data to add to the labeled data set, and simultaneously judging whether the model meets the self-growth; if yes, increasing the spectral feature extraction module and the spatial feature extraction module to update the model, continuing to train the updated model by using a new labeled data set until a stop condition is met, and outputting the trained model; and classifying the to-be-classified data by using the trained model. The application can obtain better hyperspectral image classification performance.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, specifically relating to a land surveying data classification method based on a self-growing convolutional neural network. Background Technology

[0002] With the development of image imaging, sensor technology, and aerospace technology, hyperspectral images can be obtained in the field of land surveying. Furthermore, the spatial and spectral information of ground features can be acquired using the characteristics of hyperspectral images. Hyperspectral images have a large number of spectral bands and high spectral resolution, capable of capturing dozens to hundreds of narrow-band spectral information for each ground feature, thus organically combining the spatial and spectral information of the hyperspectral image.

[0003] While the vast amount of hyperspectral data brings tremendous opportunities, it also presents numerous challenges. With the continuous development of remote sensing technology in recent years, its spatial and spectral resolution has increased significantly, leading to a continuous increase in data volume. Compared to traditional remote sensing images, the rich spectral information in hyperspectral images gives their data structure a high-dimensional characteristic. Furthermore, the inherent characteristics of hyperspectral remote sensing images, such as map-spectral integration, high dimensionality, strong band correlation, and severe data redundancy, pose new challenges to hyperspectral remote sensing image classification methods. How to efficiently utilize the rich spectral and spatial information of hyperspectral remote sensing images while reducing the computational complexity caused by high-dimensional data has become a primary issue in the field of hyperspectral remote sensing. In recent years, convolutional neural networks, due to their powerful feature extraction and classification capabilities, have been widely applied in hyperspectral image land cover classification research and have achieved good classification performance.

[0004] However, existing hyperspectral image classification methods using convolutional neural networks share a common problem: information loss occurs during spectral and spatial feature extraction due to insufficient utilization of extracted features, or information redundancy is caused by retaining too much irrelevant information. These methods fail to fully utilize key information in hyperspectral image bands and obtain more distinguishable spectral and spatial features. Furthermore, they require a large number of hyperspectral samples to train the neural network, resulting in poor classification performance for hyperspectral images with limited training data. Summary of the Invention

[0005] To address the aforementioned problems in existing technologies, this invention provides a land surveying data classification method based on a self-growing convolutional neural network. The technical problem to be solved by this invention is achieved through the following technical solution:

[0006] This invention provides a land surveying data classification method based on a self-growing convolutional neural network, comprising:

[0007] The land surveying image data is captured using a drone equipped with a camera; the land surveying image data includes a small amount of labeled datasets and a large amount of unlabeled datasets;

[0008] A convolutional neural network model is constructed. The convolutional neural network model includes a basic feature extraction module consisting of a spectral feature extraction module and a spatial feature extraction module, as well as a feature fusion module and a network output module. The spectral feature extraction module and the spatial feature extraction module are used to extract corresponding feature information from the spectral and spatial aspects, respectively. The feature fusion module is used to fuse the spectral feature information and the spatial feature information. The network output module is used to classify the fused feature information.

[0009] The constructed convolutional neural network model is iteratively trained using the labeled dataset. After each iteration, a portion of unlabeled data is selected from the unlabeled dataset and added to the labeled dataset. Simultaneously, it is determined whether the convolutional neural network model meets the self-growth condition. If it does, a spectral feature extraction module and a spatial feature extraction module are added to the basic feature extraction module to update the convolutional neural network model. The updated convolutional neural network model is then trained again using the new labeled dataset. If the condition is not met, the convolutional neural network model is maintained, and the maintained convolutional neural network model is trained again using the new labeled dataset until the iteration stopping condition is met, and the trained convolutional neural network model is output.

[0010] The trained convolutional neural network model is used to classify the captured land survey image data to be classified.

[0011] In one embodiment of the present invention, the spectral feature extraction module includes a plurality of 3D variable convolution modules connected in sequence, wherein a max pooling layer is connected between the 3D variable convolution modules.

[0012] Each of the 3D variable convolutional modules includes a plurality of 3D variable convolutional layers connected in sequence. Each of the 3D variable convolutional layers is followed by an activation layer, and the activation layer connected to the first 3D variable convolutional layer and the last 3D variable convolutional layer are connected to form a residual structure.

[0013] In one embodiment of the present invention, the spatial feature extraction module includes several parallel multi-scale feature extraction branches, all of which are sequentially connected to a concatenation layer and a max pooling layer, wherein...

[0014] Each of the multi-scale feature extraction branches includes a scale operation layer, a convolutional layer, a normalization layer, and an activation layer connected in sequence.

[0015] In one embodiment of the present invention, the feature fusion module includes a plurality of 2D convolutional layers connected in sequence.

[0016] In one embodiment of the present invention, the network output module includes a splicing layer, an average pooling layer, a fully connected layer, and a softmax classifier.

[0017] In one embodiment of the present invention, selecting a portion of unlabeled data from the unlabeled dataset and adding it to the labeled dataset includes:

[0018] The consistency of the unlabeled dataset is measured by a high-confidence sample selection strategy, and a portion of the unlabeled data is selected from the unlabeled dataset and added to the labeled dataset.

[0019] In one embodiment of the present invention, a consistency measure is performed on the unlabeled dataset using a high-confidence sample selection strategy, and a portion of unlabeled data is selected from the unlabeled dataset and added to the labeled dataset, including:

[0020] Obtain the classification result corresponding to the current convolutional neural network model, and construct the mapping graph and probability matrix corresponding to the unlabeled dataset based on the classification result;

[0021] Design a window of a preset size, and use the preset size window to extract the mapping matrix corresponding to each unlabeled data in the unlabeled dataset from the mapping graph;

[0022] The neighborhood consistency criterion is used to select high-confidence unlabeled data from the mapping matrix to form the dataset to be added;

[0023] Based on the probability matrix, select high-probability data from the dataset to be added and add it to the labeled dataset.

[0024] In one embodiment of the present invention, determining whether a convolutional neural network model satisfies the self-growth condition includes:

[0025] Construct the overall loss function corresponding to the convolutional neural network model;

[0026] The loss value of the current convolutional neural network model is calculated based on the overall loss function. It is then determined whether the loss value is greater than a preset loss value. If not, the convolutional neural network model satisfies the self-growth condition; if so, the convolutional neural network model does not satisfy the self-growth condition.

[0027] In one embodiment of the present invention, during the training process, the overall loss function corresponding to the constructed convolutional neural network model includes two parts, expressed by the formula:

[0028] L = L1 + λL2;

[0029] Where L represents the overall loss function, L1 represents the cross-entropy loss function, L2 represents the local feature preservation function, and λ represents the tradeoff parameter.

[0030] In one embodiment of the present invention, after selecting a portion of unlabeled data from the unlabeled dataset and adding it to the labeled dataset during the training process, the method further includes:

[0031] For unlabeled data that are difficult to classify in the updated unlabeled dataset, a weighted mean filter is applied using the unlabeled data in its nearest neighbor space.

[0032] The beneficial effects of this invention are:

[0033] The proposed method for classifying land surveying data based on a self-growing convolutional neural network is a comprehensive data classification approach. Specifically, it utilizes a drone equipped with a camera to capture land surveying image data, which includes a small amount of labeled data and a large amount of unlabeled data. A convolutional neural network model is constructed, comprising a basic feature extraction module consisting of a spectral feature extraction module and a spatial feature extraction module, as well as a feature fusion module and a network output module. The spectral and spatial feature extraction modules extract corresponding feature information from spectral and spatial perspectives, respectively. The feature fusion module fuses the spectral and spatial feature information, and the network output module classifies the fused feature information. The method then utilizes labeled data... The system iteratively trains the constructed convolutional neural network (CNN) model. After each iteration, a portion of unlabeled data is selected from the unlabeled dataset and added to the labeled dataset. Simultaneously, it checks whether the CNN model meets the self-growth condition. If it does, a spectral feature extraction module and a spatial feature extraction module are added to the basic feature extraction module to update the CNN model. The updated CNN model is then trained again using the new labeled dataset. If the condition is not met, the CNN model is maintained, and the maintained CNN model is trained again using the new labeled dataset until the iteration stopping condition is met. The trained CNN model is then output. The trained CNN model is then used to classify the captured land survey imagery data to be classified. As can be seen, the convolutional neural network model constructed in this invention extracts features from both spectral and spatial perspectives, which can improve classification accuracy and extract spatial information contained in hyperspectral images at a deeper level. Furthermore, during the training process of the convolutional neural network model, after each iteration, a portion of unlabeled data is selected from the unlabeled dataset and added to the labeled dataset to form a new labeled dataset. More labeled data is used for training, which avoids overfitting due to insufficient labeled data, resulting in a better convolutional neural network model after training and thus improving the classification performance of the convolutional neural network model. Moreover, during the training process, this invention no longer uses a fixed convolutional neural network model but adaptively grows the convolutional neural network model. The resulting convolutional neural network model can be used to extract shallow features, while the gradually growing convolutional neural network model can be used to extract higher-level features. This convolutional neural network model structure can achieve better hyperspectral image classification performance.

[0034] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0035] Figure 1This is a flowchart illustrating a land surveying data classification method based on a self-growing convolutional neural network provided in an embodiment of the present invention.

[0036] Figure 2 This is a schematic diagram of the structure of the convolutional neural network model provided in an embodiment of the present invention;

[0037] Figure 3 This is a schematic diagram of the spectral feature extraction module provided in an embodiment of the present invention;

[0038] Figure 4 This is a schematic diagram of the spatial feature extraction module provided in an embodiment of the present invention;

[0039] Figure 5 This is a schematic diagram of the feature fusion module and network output module provided in an embodiment of the present invention;

[0040] Figure 6 This is a schematic diagram of an example process for updating a labeled dataset provided in an embodiment of the present invention;

[0041] Figure 7 This is a schematic diagram of the structure of the convolutional neural network model after self-growth provided in an embodiment of the present invention;

[0042] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0043] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0044] To achieve high-precision classification of land surveying imagery data with limited labeled samples, please refer to [link to relevant documentation]. Figure 1 This invention provides a land surveying data classification method based on a self-growing convolutional neural network, specifically including the following steps:

[0045] S10. Use drones equipped with cameras to capture land survey image data; the land survey image data includes a small amount of labeled datasets and a large amount of unlabeled datasets.

[0046] This invention utilizes a drone equipped with a camera, but is not limited to this method, to capture land survey image data. The captured land survey image data is a hyperspectral image with high dimensionality. Due to the large amount of captured data and the redundancy and correlation of the captured data, most of the captured land survey image data is unlabeled data, with only a small amount of labeled data. This results in a limited amount of labeled data for subsequent neural network model training, which makes the classification of the captured data difficult.

[0047] S20. Construct a convolutional neural network model. The convolutional neural network model includes a basic feature extraction module consisting of a spectral feature extraction module and a spatial feature extraction module, as well as a feature fusion module and a network output module. The spectral feature extraction module and the spatial feature extraction module are used to extract corresponding feature information from the spectral and spatial aspects, respectively. The feature fusion module is used to fuse the spectral feature information and the spatial feature information. The network output module is used to classify the fused feature information.

[0048] Current hyperspectral image classification methods based on neural network models all have shortcomings. The common problem with these methods is that they lose information due to insufficient utilization of extracted features during spectral and spatial feature extraction, or retain too much irrelevant information, resulting in information redundancy. They cannot fully utilize the key information in the hyperspectral image bands and obtain more distinguishable spectral and spatial features. Furthermore, they require a large number of hyperspectral samples to train the neural network, which leads to poor performance in hyperspectral image classification when there are insufficient existing labeled samples. They also do not pay more attention to the differences in information between different spectra.

[0049] To address the shortcomings of existing neural network models, this invention provides an alternative solution, constructing a method such as... Figure 2 The novel convolutional neural network model shown includes a basic feature extraction module consisting of a spectral feature extraction module and a spatial feature extraction module, as well as a feature fusion module and a network output module. Each module is described in detail below.

[0050] A spectral feature extraction module is used to extract spectral feature information from the spectral aspect. This invention provides an optional solution, such as... Figure 3 The spectral feature extraction module shown includes several 3D variable convolution modules connected in sequence, with a max pooling layer connecting the 3D variable convolution modules.

[0051] Each of the 3D variable convolutional modules includes a plurality of 3D variable convolutional layers connected in sequence. Each 3D variable convolutional layer is followed by an activation layer, and the activation layer connected to the first 3D variable convolutional layer and the last 3D variable convolutional layer are connected to form a residual structure. The spectral feature extraction module constructed in this embodiment of the invention can extract rich inter-spectral features through its 3D deformable convolutional blocks. By focusing on and filtering these inter-spectral features through the 3D deformable convolutional blocks, more discriminative inter-spectral features can be extracted. This overcomes the limitations of existing technologies in inter-spectral feature extraction, which, due to fixed convolutional kernels, cannot extract more useful information or retain too much irrelevant information, resulting in information redundancy. This improves the classification accuracy of ground objects in hyperspectral images.

[0052] Taking three 3D variable convolutional modules as an example, the kernel size of the max-pooling layer between the first and second 3D variable convolutional modules is set to 2*2*4, and the number of kernels is set to 8; the kernel size of the max-pooling layer between the second and third 3D variable convolutional modules is set to 2*2*4, and the number of kernels is set to 16. The kernel size of the 3D variable convolutional layers in each 3D variable convolutional module is set to 3*3*3, and the activation function of each activation layer is set to the ReLU activation function.

[0053] Furthermore, the spatial feature extraction module is used to extract spectral feature information from a spatial perspective. This embodiment of the invention provides an optional solution, such as... Figure 4 The spatial feature extraction module shown includes several parallel multi-scale feature extraction branches, all of which are sequentially connected to a concatenation layer and a max pooling layer.

[0054] Each multi-scale feature extraction branch includes a scale operation layer, a convolutional layer, a normalization layer, and an activation layer connected in sequence. The spatial feature extraction module constructed in this embodiment enables the convolutional neural network model to focus on spatial features at different scales, overcoming the shortcomings of existing technologies that use a single scale to extract spatial features from hyperspectral images. Through the multi-scale feature extraction branches, these multi-scale spatial features can be focused on and filtered, extracting more discriminative spatial features. This overcomes the information loss caused by insufficient utilization of extracted features or information redundancy caused by retaining too much irrelevant information in existing technologies during spatial feature extraction, thus improving the classification ability of the convolutional neural network model during sample training.

[0055] Taking three multi-scale feature extraction branches as an example, in the first multi-scale feature extraction branch, the scale operation layer reduces the selected image's edges by one pixel, and the convolutional kernel size is set to 5*5*4. In the second multi-scale feature extraction branch, the scale operation layer reduces the selected image's edges by two pixels, and the convolutional kernel size is set to 3*3*4. In the third multi-scale feature extraction branch, the convolutional kernel size is set to 1*1*4, and the number of kernels is set to 16. The activation function of the activation layer in each multi-scale feature extraction branch is set to the ReLU activation function, and the output of the three activation layers is 16 features of size 5*5*25. After concatenation by the concatenation layer, 16 features of size 5*5*75 are obtained. Then, max pooling is performed by the max pooling layer, with the convolutional kernel set to 2*2*8 and the number of kernels set to 16.

[0056] Furthermore, the feature fusion module is used to fuse spectral feature information and spatially corresponding feature information. This embodiment of the invention provides an optional solution, such as... Figure 5The feature fusion module shown includes several 2D convolutional layers connected in sequence. This invention primarily addresses the problem of excessive redundant information between spectral bands in hyperspectral images due to their numerous spectral bands. By using a feature fusion module to extract useful inter-spectral and spatial features, the convolutional neural network model focuses more on the useful information within the feature data, thereby improving the classification accuracy of hyperspectral images.

[0057] The feature fusion module fuses spectral and spatially corresponding feature information. Therefore, among several 2D convolutional layers, some are used to process spectral feature information, and some are used to process spatial feature information. Finally, a weighted method is used to achieve feature information fusion. For example, in this embodiment of the invention, one 2D convolutional layer is set to process spectral feature information, two 2D convolutional layers are set to process spatial feature information, and three 2D convolutional layers are connected in series. The kernel size of each of the three 2D convolutional layers is set to 9*9, and the number of kernels is set to 1.

[0058] Furthermore, the network output module is used to classify the fused feature information. This embodiment of the invention provides an optional solution, such as... Figure 5 The network output module shown includes a stitching layer, an average pooling layer, a fully connected layer, and a softmax classifier. Specifically, the stitching layer stitches together the features output by the feature fusion module. The stitching result is then deduplicated by the average pooling layer, and then passed through the fully connected layer to obtain more discriminative spectral spatial features. Finally, the softmax classifier is used to obtain the classification result of the hyperspectral image.

[0059] S30. Iteratively train the constructed convolutional neural network model using the labeled dataset. After each iteration, select some unlabeled data from the unlabeled dataset and add it to the labeled dataset. At the same time, determine whether the convolutional neural network model meets the self-growth condition. If it does, add a spectral feature extraction module and a spatial feature extraction module to the basic feature extraction module to update the convolutional neural network model. Then, continue training the updated convolutional neural network model using the new labeled dataset. If it does not meet the condition, keep the convolutional neural network model and continue training the kept convolutional neural network model using the new labeled dataset until the iteration stopping condition is met, and output the trained convolutional neural network model.

[0060] To address the problem of difficulty in obtaining optimal neural network models based on limited labeled datasets, this invention provides an optional solution: selecting a portion of unlabeled data from an unlabeled dataset and adding it to a labeled dataset, including:

[0061] A high-confidence sample selection strategy is used to measure consistency in the unlabeled dataset, and a portion of the unlabeled data is selected and added to the labeled dataset. Specifically:

[0062] Obtain the classification result corresponding to the current convolutional neural network model, and construct the mapping graph and probability matrix corresponding to the unlabeled dataset based on the classification result; design a preset-size window, and use the preset-size window to extract the mapping matrix corresponding to each unlabeled data in the unlabeled dataset from the mapping graph; use the neighborhood consistency criterion to select high-confidence unlabeled data from the mapping matrix to form the dataset to be added; select high-probability data to be added from the dataset to be added based on the probability matrix and add it to the labeled dataset.

[0063] First, a softmax classifier is connected after the current convolutional neural network model. Based on the classification results of the current convolutional neural network model, a mapping of the unlabeled hyperspectral training data is constructed as follows: Figure 6 As shown in the figure, different symbols represent different classification results obtained through the current convolutional neural network model. For each sample in the unlabeled dataset, the mapping matrix corresponding to each sample in the unlabeled dataset is extracted from the mapping map using a window of preset size. Spatial information is considered when selecting high-quality samples to add. A neighborhood consistency criterion is used to determine whether a sample in the unlabeled dataset can be used as data to be added. The neighborhood consistency criterion is specifically: using a pre-set highest threshold T... v =N ne The voting (highest consensus) mechanism is used to evaluate the confidence of samples corresponding to the unlabeled dataset, and the number of neighboring samples with the same label is counted from the mapping matrix, N. ne This represents the number of neighboring samples with the same label as the corresponding sample in the unlabeled dataset. The number of neighboring samples selected from these samples is not less than N. ne If a sample in the unlabeled dataset is identified as having a high confidence level and can be used as data to be added, then that sample in the unlabeled dataset will be added to the dataset to be added.

[0064] Next, the classification results of the softmax classifier are converted into corresponding probability values. The probability values ​​corresponding to the classification results of each sample in the labeled dataset are used to form a probability matrix. The probability matrix is ​​used to measure the classification results of each sample x in the training data. i The probability value of belonging to a certain label category. For the data to be added in the dataset of the same category, all the data to be added from the dataset are sorted according to the probability matrix, as expressed by the formula:

[0065] T i =f rank (p(M(x i )=yi )) (1)

[0066] Among them, T i Represents a sample x in an unlabeled dataset i Belongs to tag category y i The sorting index of the probability values. Sort value T i It can be designed using a category matrix and a probability matrix, and its calculation process is as follows: y i It is a sample x in the unlabeled dataset i The label value of the corresponding label category, M(·) represents a category matrix used to determine the category of a sample x in the unlabeled dataset. i Does it belong to tag category y? i p(·) represents a sample x in the unlabeled dataset. i Belongs to tag category y i The probability, f rank The '·' sign represents a ranking function that sorts all unlabeled samples in the dataset according to their class probability values ​​in descending order. Specifically, samples in the unlabeled dataset are assigned progressively lower rank values ​​based on their probability. For example, the highest probability value is assigned a rank of 1, the second highest probability value is assigned a rank of 2, and so on. A higher rank value indicates a lower class probability. Based on this rank ranking, a specific percentage of data with low rank values ​​can be selected for each class as the final dataset to be added, ensuring high-quality data selection. The specific percentage is set according to the actual design. Finally, all the selected data to be added is added to the labeled dataset to complete the update of the labeled dataset.

[0067] As can be seen, in the training process of this embodiment of the invention, the convolutional neural network model is directly trained using a limited labeled dataset. Then, data to be added is generated based on the currently trained convolutional neural network model. Specifically, the labeled dataset is fed into the currently trained convolutional neural network model to complete feature extraction and classification, and the probability matrix corresponding to the labeled dataset is obtained. Then, using the above-mentioned high-confidence sample selection strategy, unlabeled data with high confidence is selected as data to be added, thereby providing more usable recognition information for image classification, which can be used in the training of high-precision convolutional neural network models.

[0068] The method for updating labeled datasets proposed in this invention avoids manual annotation, improves the accuracy of newly added labeled data, and enables high-confidence labeled datasets to provide more prior information for training convolutional neural network models. Deep learning networks based on more prior information can avoid the overfitting problem that is prone to occur with limited labeled datasets, thereby enabling the training of convolutional neural network models corresponding to the optimal network parameters.

[0069] During the entire training process of the convolutional neural network model, the overall loss function corresponding to the constructed convolutional neural network model consists of two parts, expressed by the formula:

[0070] L=L1+λL2 (2)

[0071] Where L represents the overall loss function, L1 represents the cross-entropy loss function, which is the main loss function term of the convolutional neural network model, L2 represents the local feature preservation function, and λ represents the tradeoff parameter, for example, the tradeoff parameter λ takes a value of 0.001. Specifically:

[0072] The cross-entropy loss optimization function is expressed as follows:

[0073]

[0074]

[0075] Where L1 represents the loss between the predicted label vector and the true label vector, d(·) represents the Euclidean distance, and F ω (·) represents the feature extraction function for parameter ω, f l Let x represent the feature of the l-th class in the labeled dataset, C represent the number of classes, and x represent the feature of the l-th class. j Let y represent a sample in a labeled dataset. j Indicates sample x j The corresponding label, Q, indicates a labeled dataset.

[0076] Through research and analysis by the inventors, it has been found that data structures can provide rich information for obtaining the inherent attributes of data and the relationships between samples. However, in the feature mapping of traditional neural networks, the spatial structure information that helps obtain discriminative features cannot be preserved. Local linear embedding is a basic nonlinear unsupervised manifold learning algorithm that projects data from a high-dimensional space to a low-dimensional space, transforming the global nonlinear structure of the high-dimensional data into a local linear structure. By reconstructing data points using the neighborhood of samples in the mapped low-dimensional space and the local weight matrix in the high-dimensional space, the local geometric structure of the original high-dimensional data is well preserved, thereby reducing data redundancy. The local feature preservation function designed in the final embodiment of this invention is expressed by the following formula:

[0077]

[0078] Where N represents the number of samples in the labeled dataset, θ represents the network parameters of the convolutional neural network, k represents the number of nearest neighbor samples, and m(x) i ,θ) represents a sample x in a labeled dataset. i In the representation in the low-dimensional feature space, m(x) ij ,θ) represents the neighborhood sample x ij In the representation in the low-dimensional feature space, ω ij Represents neighborhood sample x ij Sample x in the labeled hyperspectral training data i Contribution to reconstruction Let F represent the square of the F-norm. By introducing Equation (5), the local reconstruction relationship between each sample in the labeled dataset and its neighboring samples can be maintained in the mapping space.

[0079] The overall loss function constructed in this embodiment of the invention can constrain the relationship between the input feature space and the mapped feature space of the network. It can optimize the network parameters of the convolutional neural network model by enriching labeled data information and mining unlabeled data information, thereby providing guidance for better interpretation of hyperspectral images and thus better achieving the classification of hyperspectral images.

[0080] Furthermore, since this embodiment of the invention provides a limited labeled dataset update strategy, the convolutional neural network (CNN) model training has more labeled datasets as training samples. Traditional CNN model training is limited by a limited labeled dataset; now, with more training samples, it means that more complex CNN models can be designed to extract richer feature information for classification. This embodiment of the invention provides an optional solution: after updating the labeled dataset, it first determines whether the CNN model meets the self-growth condition. If it does, a spectral feature extraction module and a spatial feature extraction module are added to the basic feature extraction module to update the CNN model. Then, the updated CNN model is trained again using the new labeled dataset. If it does not meet the condition, the CNN model is maintained, and the maintained CNN model is trained again using the new labeled dataset until the iteration stopping condition is met, at which point the trained CNN model is output.

[0081] Regarding how to determine the self-growth condition, this embodiment of the invention provides an optional solution for determining whether a convolutional neural network model satisfies the self-growth condition, including:

[0082] Construct the overall loss function corresponding to the convolutional neural network model; calculate the loss value of the current convolutional neural network model based on the overall loss function, and determine whether the loss value is greater than the preset loss value. If not, the convolutional neural network model satisfies the self-growth condition; if so, the convolutional neural network model does not satisfy the self-growth condition. The preset loss value is designed based on the actual situation.

[0083] For cases where the self-growth condition is met, embodiments of the present invention provide an optional scheme for updating the convolutional neural network model, such as... Figure 7 As shown, a spectral feature extraction module and a spatial feature extraction module are added to the basic feature extraction module, while the network structure of other modules remains unchanged. The added spectral and spatial feature extraction modules have the same network parameters as the original spectral and spatial feature extraction modules. M represents the number of times self-growth is satisfied in multiple training iterations. During the growth of the convolutional neural network model, as the number of basic feature extraction modules in the convolutional neural network model increases, its network parameters will also increase accordingly. If the traditional method of directly and randomly initializing a large number of parameters in the network is used, it will affect the feature extraction capability of the network to a certain extent. In order to ensure the feature extraction capability, in the training of this embodiment, the initial parameters of the convolutional neural network model after training and growth are taken from the network parameters of the convolutional neural network model after the previous training. The network parameters of the newly added basic feature extraction modules are taken from the network parameters of the last basic feature extraction module in the convolutional neural network model after the previous training.

[0084] As can be seen, the convolutional neural network model in this embodiment of the invention can be used to extract shallow features, while the gradually increasing convolutional neural network model can be used to extract higher-level features. Such a convolutional neural network model structure can achieve better hyperspectral image classification performance.

[0085] It should be noted that the sampling method is not limited during the entire training process of the convolutional neural network model. For example, the Adam model optimizer can be used. For different training datasets, the learning rate of the Adam model optimizer can be set to a value in the range of 0.00001 to 0.001. The maximum number of iterations during the training iteration process depends on the actual design needs. For example, the maximum number of iterations can be 20.

[0086] Furthermore, this embodiment of the invention provides an optional solution: during the training process, after selecting a portion of unlabeled data from the unlabeled dataset and adding it to the labeled dataset, the method further includes: performing weighted mean filtering on the unlabeled data in the updated unlabeled dataset that is difficult to classify, using the unlabeled data in its nearest neighbor space.

[0087] After each training iteration, the labeled dataset is expanded. The unlabeled dataset added to the labeled dataset is removed. In this embodiment of the invention, the samples in the remaining unlabeled dataset are considered difficult to classify. Therefore, a weighted mean filter is applied to these samples to incorporate spatial location information and eliminate noise and background interference. This embodiment of the invention provides an optional solution: for the difficult-to-classify unlabeled data in the updated unlabeled dataset, a weighted mean filter is applied using the unlabeled data in its nearest neighbor space. The specific implementation is as follows:

[0088] In hyperspectral images, each pixel exhibits spectral and spatial correlation with its spatially adjacent pixels, suggesting that they are likely composed of the same type of land cover. Based on this, embodiments of the present invention provide a method for handling pixel x... i Example of weighted mean filtering operation:

[0089] Assume that sample i corresponds to pixel x in the entire hyperspectral image. i Its coordinates in the land survey image to be classified are (h i ,w i ), then with x i For the center pixel, a square nearest neighbor space Ω(x) with side length ω. i It can be defined as:

[0090]

[0091] Among them, the nearest neighbor space Ω(x) i It contains ω×ω pixels, excluding the center pixel x. i The rest of ω 2 -1 pixels can be represented as: For pixels located at the edge of a hyperspectral image, the pixel itself is used for pixel filling.

[0092] center pixel x i By using a weighted summation method, ω in the nearest neighbor space is utilized. 2 Reconstructing the pixel x by taking -1 neighboring pixels and the pixel itself. i ', then x i 'Can be represented as:

[0093]

[0094] Where, x j Represents the nearest neighbor space Ω(x) i The neighboring pixels in ) v k Represents the nearest neighbor space Ω(x) i The k-th pixel x in ) ik The weights in the weighted summation can be solved using the heat kernel function:

[0095]

[0096] Where, d i Represents the nearest neighbor space Ω(x) i All pixels in ) and the center pixel x i The average distance can be expressed as:

[0097]

[0098] The weighted mean filtering method adjusts the filtering window by setting the value of the parameter ω. Essentially, it measures the nearest neighbor space Ω(x). i The nearest neighbor pixel and the center pixel x) i The similarity of the pixels is used to reconstruct the center pixel using a weighted mean. The higher the similarity, the greater the weight; the greater the difference, the smaller the weight. Therefore, the weighted mean filtering method can effectively eliminate the interference of background points and noise, obtaining images with smoother edges, thus providing guidance for better interpretation of hyperspectral images and ultimately achieving better classification of hyperspectral images.

[0099] S40. Use a trained convolutional neural network model to classify the captured land survey image data to be classified.

[0100] In summary, the land surveying data classification method based on a self-growing convolutional neural network proposed in this invention is a comprehensive data classification method. Specifically, it utilizes a drone equipped with a camera to capture land surveying image data; wherein, the land surveying image data includes a small amount of labeled datasets and a large amount of unlabeled datasets; a convolutional neural network model is constructed; the convolutional neural network model includes a basic feature extraction module consisting of a spectral feature extraction module and a spatial feature extraction module, as well as a feature fusion module and a network output module. The spectral feature extraction module and the spatial feature extraction module are used to extract corresponding feature information from spectral and spatial aspects, respectively; the feature fusion module is used to fuse the spectral and spatial feature information; and the network output module is used to classify the fused feature information. The labeled dataset is used to iteratively train the constructed convolutional neural network model. After each iteration, a portion of unlabeled data is selected from the unlabeled dataset and added to the labeled dataset. Simultaneously, it is determined whether the convolutional neural network model meets the self-growth condition. If it does, a spectral feature extraction module and a spatial feature extraction module are added to the basic feature extraction module to update the convolutional neural network model. The updated convolutional neural network model is then trained again using the new labeled dataset. If the condition is not met, the convolutional neural network model is maintained, and the maintained model is trained again using the new labeled dataset until the iteration stopping condition is met. The trained convolutional neural network model is then output. The trained convolutional neural network model is then used to classify the captured land survey imagery data to be classified. As can be seen, the convolutional neural network model constructed in this embodiment of the invention extracts features from both spectral and spatial aspects, which can improve classification accuracy and extract spatial information contained in hyperspectral images at a deeper level. Furthermore, in the training process of the convolutional neural network model, after each iteration of training, a portion of unlabeled data is selected from the unlabeled dataset and added to the labeled dataset to form a new labeled dataset. More labeled data is used for training, which can avoid overfitting due to insufficient labeled data, resulting in a better convolutional neural network model after training, thereby improving the classification performance of the convolutional neural network model. Moreover, in the training process of the convolutional neural network model, this embodiment of the invention no longer uses a fixed convolutional neural network model, but rather adaptively grows the convolutional neural network model during training. The resulting convolutional neural network model can be used to extract shallow features, while the gradually growing convolutional neural network model can be used to extract higher-level features. This convolutional neural network model structure can achieve better hyperspectral image classification performance.

[0101] Please see Figure 8This invention provides an electronic device, including a processor 801, a communication interface 802, a memory 803, and a communication bus 804, wherein the processor 801, the communication interface 802, and the memory 803 communicate with each other through the communication bus 804.

[0102] Memory 803 is used to store computer programs;

[0103] When the processor 801 executes the program stored in the memory 803, it implements the steps of the above-mentioned land surveying data classification method based on a self-growing convolutional neural network.

[0104] This invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described land surveying data classification method based on a self-growing convolutional neural network.

[0105] For the electronic device / storage medium embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be referred to in the description of the method embodiment.

[0106] In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0107] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the specification and accompanying drawings, will understand and implement other variations of the disclosed embodiments in carrying out the claimed invention. In the specification, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. While certain measures are described in different embodiments, this does not mean that these measures cannot be combined to produce good results.

[0108] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A land surveying data classification method based on a self-growing convolutional neural network, characterized in that, include: The land surveying image data is captured using a drone equipped with a camera; the land surveying image data includes a small amount of labeled datasets and a large amount of unlabeled datasets; A convolutional neural network model is constructed. The convolutional neural network model includes a basic feature extraction module consisting of a spectral feature extraction module and a spatial feature extraction module, as well as a feature fusion module and a network output module. The spectral feature extraction module and the spatial feature extraction module are used to extract corresponding feature information from the spectral and spatial aspects, respectively. The feature fusion module is used to fuse the spectral feature information and the spatial feature information. The network output module is used to classify the fused feature information. The constructed convolutional neural network (CNN) model is iteratively trained using the labeled dataset. After each iteration, a portion of unlabeled data is selected from the unlabeled dataset and added to the labeled dataset. Simultaneously, it is determined whether the CNN model satisfies the self-growth condition. If it does, a spectral feature extraction module and a spatial feature extraction module are added to the basic feature extraction module to update the CNN model. The updated CNN model is then trained again using the new labeled dataset. If it does not satisfy the self-growth condition, the CNN model is maintained, and the maintained CNN model is trained again using the new labeled dataset until the iteration stopping condition is met, at which point the trained CNN model is output. Determining whether the CNN model satisfies the self-growth condition includes: constructing the overall loss function corresponding to the CNN model; calculating the loss value of the current CNN model based on the overall loss function; and determining whether the loss value is greater than a preset loss value. If not, the CNN model satisfies the self-growth condition; otherwise, it does not. During training, the overall loss function corresponding to the constructed CNN model consists of two parts, expressed by the formula: ； in, Represents the overall loss function. Represents the cross-entropy loss function. This represents a local feature preservation function. The parameters represent the trade-offs; the cross-entropy loss optimization function is expressed as: ； ； in, Represents Euclidean distance. Indicates parameters Feature extraction function, This indicates the first element in the labeled dataset. Features of each category Represented as the number of categories. This represents a sample in a labeled dataset. Indicates sample The corresponding tags This indicates a labeled dataset; The local feature preservation function is expressed as follows: ； in, This indicates the number of samples in the labeled dataset. This represents the network parameters of a convolutional neural network. k This represents the number of samples in the nearest neighbor's neighborhood. This represents samples in a labeled dataset. Representation in low-dimensional feature space, Representing neighborhood samples Representation in low-dimensional feature space, Representing neighborhood samples Samples in the label hyperspectral training data Contribution to reconstruction express The square of the norm; The trained convolutional neural network model is used to classify the captured land survey image data to be classified.

2. The land surveying data classification method based on a self-growing convolutional neural network according to claim 1, characterized in that, The spectral feature extraction module includes several 3D variable convolution modules connected in sequence. Each 3D variable convolution module is connected to a max-pooling layer. Each of the 3D variable convolutional modules includes a plurality of 3D variable convolutional layers connected in sequence. Each of the 3D variable convolutional layers is followed by an activation layer, and the activation layer connected to the first 3D variable convolutional layer and the last 3D variable convolutional layer are connected to form a residual structure.

3. The land surveying data classification method based on a self-growing convolutional neural network according to claim 1, characterized in that, The spatial feature extraction module includes several parallel multi-scale feature extraction branches, all of which are sequentially connected to a concatenation layer and a max pooling layer. Each of the multi-scale feature extraction branches includes a scale operation layer, a convolutional layer, a normalization layer, and an activation layer connected in sequence.

4. The land surveying data classification method based on a self-growing convolutional neural network according to claim 1, characterized in that, The feature fusion module includes several 2D convolutional layers connected in sequence.

5. The land surveying data classification method based on a self-growing convolutional neural network according to claim 1, characterized in that, The network output module includes a splicing layer, an average pooling layer, a fully connected layer, and a softmax classifier.

6. The land surveying data classification method based on a self-growing convolutional neural network according to claim 1, characterized in that, Select a portion of unlabeled data from the unlabeled dataset and add it to the labeled dataset, including: The consistency of the unlabeled dataset is measured by a high-confidence sample selection strategy, and a portion of the unlabeled data is selected from the unlabeled dataset and added to the labeled dataset.

7. The land surveying data classification method based on a self-growing convolutional neural network according to claim 6, characterized in that, A consistency measure is performed on the unlabeled dataset using a high-confidence sample selection strategy. A portion of the unlabeled data is selected from the unlabeled dataset and added to the labeled dataset, including: Obtain the classification result corresponding to the current convolutional neural network model, and construct the mapping graph and probability matrix corresponding to the unlabeled dataset based on the classification result; Design a window of a preset size, and use the preset size window to extract the mapping matrix corresponding to each unlabeled data in the unlabeled dataset from the mapping graph; The neighborhood consistency criterion is used to select high-confidence unlabeled data from the mapping matrix to form the dataset to be added; Based on the probability matrix, select high-probability data from the dataset to be added and add it to the labeled dataset.

8. The land surveying data classification method based on a self-growing convolutional neural network according to claim 1, characterized in that, During training, after selecting a portion of unlabeled data from the unlabeled dataset and adding it to the labeled dataset, the process also includes: For unlabeled data that are difficult to classify in the updated unlabeled dataset, a weighted mean filter is applied using the unlabeled data in its nearest neighbor space.

Images