A remote sensing image segmentation method and device based on a convolutional neural network

Abstract

The application discloses a kind of based on convolutional neural network's remote sensing image segmentation method and device, wherein method includes: using ground station server according to remote sensing image dataset Training convolutional neural network obtains trained convolutional neural network;During training, for each layer, seed convolution kernel generated according to random seed is convolved to input data to obtain seed feature map, according to preset rule, seed feature map is nonlinearly transformed to obtain multiple mapping feature maps, seed feature map and mapping feature map are used as the input data of next layer;Using ground station server, the seed convolution kernel corresponding to each layer in trained convolutional neural network and the random number for saving nonlinear transformation hyperparameter are sent to near-earth orbit satellite;Using near-earth orbit satellite, trained convolutional neural network is deployed according to seed convolution kernel and random number, and the segmentation task of remote sensing image is completed.The application reduces floating point calculation amount and parameter quantity, simplifies satellite model deployment and operation pressure.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a remote sensing image segmentation method and apparatus based on convolutional neural networks. Background Technology

[0002] In the field of computer vision, image segmentation refers to the process of subdividing a digital image into multiple image sub-regions. Its purpose is to simplify or change the image representation, making it easier to understand and analyze. Semantic segmentation of remote sensing images classifies each pixel in an image based on land cover type and is an important research direction in remote sensing image processing. Due to the large differences in the scale of land features and the complex boundaries of land features contained in remote sensing images, accurately extracting features from remote sensing images is quite difficult, making precise segmentation of remote sensing images challenging.

[0003] Convolutional neural networks (CNNs) are deep learning models primarily used for image recognition and computer vision tasks. Currently, the mainstream method for remote sensing image segmentation involves transmitting data acquired by satellites to ground station servers, and then processing it using CNN models to achieve high-performance results. However, this method has at least the following limitations: First, it requires significant bandwidth resources because remote sensing images typically contain a large amount of information and data, and frequent transmission of raw data to ground station servers consumes substantial bandwidth. Second, it is slow because the communication distance between satellites and ground station servers is relatively long, especially when transmitting large amounts of data, resulting in significant transmission delays. Furthermore, satellites are primarily powered by solar energy, which limits their transmission power and speed. Third, due to various interferences affecting the link channel, the bit error rate of transmitted data is relatively high, leading to lower transmission performance.

[0004] Processing remote sensing image data from low Earth orbit satellites can alleviate the above limitations to some extent, but deploying high-performance convolutional neural network models on low Earth orbit satellites presents resource constraints and consumes a large amount of computing resources. Summary of the Invention

[0005] The purpose of this invention is to provide a remote sensing image segmentation method and apparatus based on convolutional neural networks, so as to solve the technical problems of existing technologies that require a large amount of bandwidth or computing resources, have long transmission delays, and have low performance.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] A remote sensing image segmentation method based on convolutional neural networks includes:

[0008] A convolutional neural network is trained using a ground station server based on a remote sensing image dataset. During training, for each layer of the convolutional neural network, a seed feature map is obtained by performing a convolution operation on the input data according to a seed convolution kernel. The seed feature map is then subjected to a nonlinear transformation according to a preset rule to obtain multiple mapping feature maps. The seed feature map and the mapping feature map are used as the input data for the next layer.

[0009] Wherein, the seed convolution kernel is a convolution kernel generated using a random seed, the seed feature map is a feature map generated using the seed convolution kernel, the mapping feature map is other feature maps in this layer besides the seed feature map, and the hyperparameters of the nonlinear transformation are randomly initialized and stored in random numbers;

[0010] The seed convolutional kernels and random numbers corresponding to each layer of the trained convolutional neural network are sent to a resource-constrained low Earth orbit satellite using a ground station server.

[0011] The trained convolutional neural network is deployed using the low-Earth orbit satellite according to the seed convolutional kernel and the random number, and the segmentation task of the captured remote sensing image is completed based on the trained convolutional neural network.

[0012] Optionally, after deploying the trained convolutional neural network using the low-Earth orbit satellite according to the seed convolutional kernel and the random number, the process further includes:

[0013] The trained convolutional neural network is updated using the near-Earth orbit satellite based on the seed convolutional kernel and the random number.

[0014] Optionally, performing a nonlinear transformation on the seed feature map according to preset rules to obtain multiple mapped feature maps includes:

[0015] Multiple mapped feature maps are obtained by performing a nonlinear transformation on the seed feature map according to the nonlinear transformation function.

[0016] Optionally, the nonlinear transformation function is a sine function.

[0017] Optionally, the parameters of the sine function include at least:

[0018] Phase and angular frequency: Phase represents the offset of a sine wave, and angular frequency represents the vibration frequency of a sine wave.

[0019] The present invention also provides a remote sensing image segmentation device based on a convolutional neural network, comprising:

[0020] The neural network training module is used to train a convolutional neural network using a ground station server based on a remote sensing image dataset to obtain a trained convolutional neural network. During training, for each layer of the convolutional neural network, the input data is convolved according to the seed convolution kernel to obtain a corresponding seed feature map. The seed feature map is then nonlinearly transformed according to a preset rule to obtain multiple mapping feature maps. The seed feature map and the mapping feature map are used as the input data for the next layer.

[0021] Wherein, the seed convolution kernel is a convolution kernel generated using a random seed, the seed feature map is a feature map generated using the seed convolution kernel, the mapping feature map is other feature maps in this layer besides the seed feature map, and the hyperparameters of the nonlinear transformation are randomly initialized and stored in random numbers;

[0022] The neural network data transmission module is used to send the seed convolution kernels and random numbers corresponding to each of the trained convolutional neural networks to a resource-constrained low Earth orbit satellite using a ground station server.

[0023] The neural network deployment and usage module is used to deploy the trained convolutional neural network using the low Earth orbit satellite according to the seed convolutional kernel and the random number, and to complete the segmentation task of the captured remote sensing image based on the trained convolutional neural network.

[0024] Optionally, it also includes:

[0025] The neural network update module is used to update the trained convolutional neural network using the near-Earth orbit satellite based on the seed convolutional kernel and the random number.

[0026] Optionally, the neural network training module performs a nonlinear transformation on the seed feature map according to a preset rule to obtain multiple mapped feature maps, including:

[0027] The neural network training module performs a nonlinear transformation on the seed feature map according to a nonlinear transformation function to obtain multiple mapped feature maps.

[0028] Optionally, the nonlinear transformation function is a sine function.

[0029] Optionally, the parameters of the sine function include at least:

[0030] Phase and angular frequency: Phase represents the offset of a sine wave, and angular frequency represents the vibration frequency of a sine wave.

[0031] This invention discloses a remote sensing image segmentation method and apparatus based on a convolutional neural network (CNN). The method includes: training a CNN using a ground station server based on a remote sensing image dataset to obtain a trained CNN; during training, for each layer of the CNN, performing a convolution operation on the input data according to a seed convolution kernel to obtain a corresponding seed feature map, performing a nonlinear transformation on the seed feature map according to a preset rule to obtain multiple mapping feature maps, and using the seed feature map and the mapping feature map as input data for the next layer; wherein the seed convolution kernel is a convolution kernel generated using a random seed, the seed feature map is a feature map generated using the seed convolution kernel, and the mapping feature map is other feature maps in that layer besides the seed feature map; the hyperparameters of the nonlinear transformation are randomly initialized and stored in random numbers; the ground station server sends the seed convolution kernels and the random numbers corresponding to each layer of the trained CNN to a resource-constrained low Earth orbit (LEO) satellite; the LEO satellite deploys the trained CNN according to the seed convolution kernels and the random numbers, and completes the segmentation task of the captured remote sensing image based on the trained CNN.

[0032] In view of this, the beneficial effects of this invention are:

[0033] In this invention, when training a convolutional neural network, the ground station server only needs to perform a sliding convolution operation at each layer of the model to generate a seed feature map. Then, based on the seed feature map, multiple other mapped feature maps are generated through nonlinear transformations, avoiding a large number of sliding window operations and thus significantly reducing the amount of floating-point computation. To complete the deployment of the convolutional neural network model, the ground station server only needs to send a small number of seed convolution kernels and random numbers to the low Earth orbit satellite. Compared with the traditional method of sending all model parameters, this invention can significantly reduce the number of parameters transmitted from the ground station server to the low Earth orbit satellite, thereby greatly reducing the consumption of bandwidth resources, reducing the amount of data transmission to alleviate the problem of long latency when transmitting large amounts of data, and deploying the model at the cost of less bandwidth consumption.

[0034] In existing remote sensing services for low Earth orbit satellites, this invention achieves a smaller number of parameters compared to other methods, greatly reducing bandwidth pressure in satellite services. It also reduces floating-point calculations in low Earth orbit satellites, simplifies satellite model deployment and computational pressure, and achieves an optimal trade-off between minimizing the number of parameters and maximizing performance. This greatly promotes the development of remote sensing services and research on reducing the complexity of convolutional neural network models. Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating the method of the present invention;

[0036] Figure 2 This is a schematic diagram of the system framework for ground station server-assisted training and updating of CNN models in this invention;

[0037] Figure 3 This is a schematic diagram of the SineFM algorithm in this invention;

[0038] Figure 4 This is a schematic diagram illustrating the accuracy results of nonlinear transformation using polynomial functions in this invention.

[0039] Figure 5 This is a schematic diagram illustrating the accuracy results of nonlinear transformation using radial basis functions in this invention;

[0040] Figure 6 This is a schematic diagram illustrating the accuracy results of nonlinear transformation using different feature generation functions in this invention;

[0041] Figure 7 This is a diagram comparing the floating-point computational complexity of several models;

[0042] Figure 8 This is a diagram comparing the parameter quantities of several models;

[0043] Figure 9 This is a schematic diagram of the experimental results of the angular frequency in the sine function used in this invention;

[0044] Figure 10 This is a schematic diagram of the experimental results of the phase in the sine function used in this invention;

[0045] Figure 11 This is a schematic diagram of the experimental results of the fan-out ratio in the sine function used in this invention;

[0046] Figure 12 This is a schematic diagram of the experimental results regarding the number of random seeds in the sine function used in this invention;

[0047] Figure 13 This is a schematic diagram of the structure of an embodiment of the device of the present invention. Detailed Implementation

[0048] Terminology Explanation:

[0049] Convolutional Neural Networks (CNNs) are deep learning algorithms widely used in image recognition, speech recognition, and other fields. The basic idea of ​​CNNs is to extract image features using convolution operations and then perform tasks such as classification or regression through the neural network. In convolution operations, the network processes each pixel in the image, and through continuous convolution and downsampling operations, it ultimately obtains a high-level feature representation of the image. These features can be used for tasks such as image classification, object detection, and image segmentation. For example, in image recognition tasks, CNNs can identify various types of images and achieve accuracy surpassing human capabilities to a certain extent. In the field of natural language processing, CNNs are also widely used in tasks such as text classification and sentiment analysis. Furthermore, CNNs are also used in applications such as speech recognition and video analysis.

[0050] Floating-point operations and model parameters: Floating-point operations refer to numerical calculations involving decimals performed in a computer. In convolutional neural networks (CNNs), floating-point operations are used for convolution operations, activation function calculations, pooling operations, etc., on the input data. This is an important indicator of the computational resources required for a CNN model. Model parameters refer to the parameters that need to be learned in the network, typically including convolutional kernels, weights of fully connected layers, biases, etc. In CNNs, these parameters need to be continuously adjusted through the backpropagation algorithm to minimize the network's loss function on the training data. After training, these parameters can effectively extract feature information from data such as images and speech, thereby achieving various machine learning tasks. The number of floating-point operations and model parameters in CNNs is usually very large, especially in deep networks. This not only consumes a lot of computational resources but also leads to longer model training times, thus requiring optimization and compression.

[0051] Semantic segmentation is a computer vision task that aims to segment an image into distinct semantic regions, each representing a specific object or part of the image. Unlike traditional image classification tasks, semantic segmentation requires not only identifying objects in an image but also distinguishing them from other objects and the background.

[0052] mIoU (mean Intersection over Union) is a metric for evaluating the performance of semantic segmentation models. It calculates the ratio of the intersection to the union of the predicted and actual segmentations. Specifically, it calculates the intersection-union ratio for each class and averages it to determine the model's average accuracy. A higher mIoU indicates better model performance.

[0053] Convolutional kernel: In convolutional neural networks (CNNs), a convolutional kernel is a set of parameters used to perform a convolution operation on the input image. A convolutional kernel consists of weights and a bias term; these weights are automatically learned during the training of the CNN. The convolution operation is one of the core operations in CNNs; it extracts feature information from the image step by step by sliding the convolutional kernel across the input image. By using different convolutional kernels and combining their outputs, CNNs can extract more feature information from the input image.

[0054] Feature maps: In convolutional neural networks (CNNs), feature maps are the output images obtained by convolving the input image with convolutional kernels in convolutional layers. Each feature map represents the response of a different feature in the input image. By applying different convolutional kernels in convolutional layers, different features in the input image can be captured, such as edges, textures, and shapes. Furthermore, feature maps from multiple convolutional layers can be combined to form higher-level features. Feature maps are a crucial concept in CNNs and are key to their ability to effectively learn and extract image features.

[0055] Nonlinear Transformations: In deep learning, to better extract and represent data features, multiple different linear transformations are typically used to construct neural networks. However, the combination of multiple linear transformations remains linear and cannot represent more complex data structures, such as nonlinear data relationships and distributed representations. Therefore, to address this issue, nonlinear transformations are introduced to help neural networks better handle nonlinear data relationships and obtain better feature representations. Nonlinear transformations can be implemented using activation functions, such as Sigmoid and ReLU. These activation functions can map the results of linear transformations to a nonlinear space, thereby expanding the representational capabilities of neural networks.

[0056] When using nonlinear transformations, it is usually necessary to specify some hyperparameters to control the properties and effects of the nonlinear transformation. These include the type, parameters, and location of the activation function, the drop-out rate, and the normalization parameters. These hyperparameters can affect the model's performance and learning effect, and need to be carefully adjusted and selected.

[0057] This invention provides a remote sensing image segmentation method and apparatus based on convolutional neural networks to solve the technical problems of existing technologies, such as the need for large amounts of bandwidth or computing resources, long transmission delays, and low performance.

[0058] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Preferred embodiments of the invention are shown in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0060] Convolutional Neural Networks (CNNs) are deep learning models proposed by Yann LeCun et al. in the early 1990s, primarily used for image recognition and computer vision tasks. The success of CNNs stems from their unique advantages in image processing, such as translation invariance and the ability to extract local features.

[0061] The technological background of convolutional neural networks can be traced back to research in neuroscience and computer vision. In neuroscience, researchers discovered that neurons in the visual cortex exhibit a high degree of selectivity for specific visual stimuli; that is, they respond only to visual stimuli of a particular orientation, shape, or color. This discovery of neuronal selectivity inspired the design of convolution operations in convolutional neural networks, which use convolution kernels to perform convolution operations on images to extract local features such as different orientations, shapes, and colors.

[0062] On the other hand, feature extraction algorithms in computer vision have always been an important research area. Traditional computer vision algorithms typically use hand-designed feature extractors, which require expert knowledge and extensive experimentation to determine. Convolutional neural networks, however, can automatically learn feature extractors through backpropagation, thereby reducing reliance on expert knowledge and improving the accuracy and efficiency of feature extraction.

[0063] With the continuous development of computer hardware and further research into deep learning algorithms, convolutional neural network models have achieved overwhelming success in fields such as computer vision, natural language processing, and speech recognition, and using them to provide high-quality services has become mainstream.

[0064] Furthermore, sufficient computing resources are a crucial support for training high-performance convolutional neural networks, leading to a surge in research on server-assisted convolutional neural network training in recent years. However, existing training methods are highly dependent on the network conditions between the server and the device. Unstable or disconnected networks can significantly degrade the user experience.

[0065] Traditional methods for processing remote sensing image data involve transmitting the data from satellites to ground-based servers, and then using convolutional neural network models for processing to achieve high-performance results. These technological costs have resulted in significant obstacles in combining neural networks and satellite technology, despite substantial advancements, into practical applications.

[0066] Preprocessing low-Earth orbit satellite remote sensing imagery data can alleviate the aforementioned limitations to some extent. Using the transmitted results instead of the original remote sensing data can significantly save bandwidth resources while reducing transmission latency and bit error rate. However, this also presents two challenges:

[0067] First, low-Earth orbit (LEO) satellites have limited resources, while convolutional neural network (CNN) models are computationally intensive. Even performing convolution operations during the inference phase of a CNN model consumes significant computational resources because convolution operations require sliding the convolution kernel across the remotely sensed image. This process is performed multiple times, resulting in numerous floating-point operations. For example, ResNet-50 requires 4.1 billion floating-point operations to process an image of size 224×224. Therefore, deploying high-performance CNN models on resource-constrained LEO satellites is challenging.

[0068] Second, bandwidth resources are limited, while convolutional neural network models have a large number of parameters. Low-Earth orbit satellites typically have a lifespan of five years or more, while convolutional neural network models are developing rapidly. To achieve better service quality, these models need frequent updates. However, high-performance convolutional neural network models usually contain a large number of parameters, posing a challenge to the transmission of large parameters during model updates.

[0069] This invention implements a framework for using a ground station server to assist in training and updating a convolutional neural network model for processing remote sensing images by low Earth orbit (LEO) satellites, and is suitable for low Earth orbit satellites with limited resources.

[0070] Please see Figure 1 This invention provides an embodiment of a remote sensing image segmentation method based on a convolutional neural network, comprising:

[0071] S100: The ground station server is used to train a convolutional neural network based on the remote sensing image dataset to obtain a trained convolutional neural network. During training, for each layer of the convolutional neural network, the input data is convolved according to the seed convolutional kernel to obtain the corresponding seed feature map. The seed feature map is nonlinearly transformed according to the preset rules to obtain multiple mapping feature maps. The seed feature map and the mapping feature map are used as the input data for the next layer.

[0072] Wherein, the seed convolution kernel is a convolution kernel generated using a random seed, the seed feature map is a feature map generated using the seed convolution kernel, the mapping feature map is other feature maps in this layer besides the seed feature map, and the hyperparameters of the nonlinear transformation are randomly initialized and stored in random numbers;

[0073] S200: Using a ground station server, the seed convolution kernels and random numbers corresponding to each layer of the trained convolutional neural network are sent to a resource-constrained low Earth orbit satellite;

[0074] S300: The low-Earth orbit satellite deploys the trained convolutional neural network according to the seed convolutional kernel and the random number, and completes the segmentation task of the captured remote sensing image according to the trained convolutional neural network.

[0075] This invention implements a framework for ground station server-assisted training and updating of convolutional neural network models, such as... Figure 2 As shown, this method is used to process remote sensing images for low Earth orbit satellites, addressing the problems encountered in current solutions for processing remote sensing images: processing satellite remote sensing images on ground station servers consumes a large amount of satellite transmission bandwidth resources, is slow, and has a high error rate; while deploying high-performance convolutional neural network models on low Earth orbit satellites has resource limitations and consumes a large amount of computing resources.

[0076] Please see Figure 2 The deployment process of convolutional neural network models for low Earth orbit satellites is as follows: First, the required convolutional neural network is trained using a ground server to obtain a trained convolutional neural network. Then, the trained convolutional neural network is sent to the low Earth orbit satellite for deployment or updating. Next, the low Earth orbit satellite captures remote sensing images and uses the trained convolutional neural network to analyze the captured remote sensing images for rapid response.

[0077] In step S100, a convolutional neural network is trained using a ground station server based on a remote sensing image dataset to obtain a trained convolutional neural network.

[0078] Please see Figure 3 This application provides a low-cost feature map generation algorithm, SineFM, which aims to replace the sliding window method with a nonlinear transformation method to generate feature maps, thereby reducing the amount of computation required to run convolutional neural networks.

[0079] The following is SineFM's explanation:

[0080] The core idea of ​​the SineFM algorithm is to restrict the convolutional neural network to learn only one (or a few) seed feature maps at each layer, and then, through nonlinear transformation, generate or add as many desired feature maps as possible at each layer.

[0081] It should be noted that whether each layer of a convolutional neural network learns only one seed feature map or a limited number of seed feature maps can be configured differently for different layers of different convolutional neural networks. For general convolutional neural networks, the number of convolutional kernels in each layer is very large, resulting in a large amount of computation. Learning only one or a limited number of seed feature maps can still greatly reduce the amount of computation. In a preferred embodiment, each layer of the convolutional neural network in this example can learn only one seed convolutional kernel parameter, that is, each layer learns only one seed feature map.

[0082] For a typical convolutional neural network, backpropagation is required to update the weights and biases in layers such as convolutional layers, pooling layers, and fully connected layers. The key point of the SineFM algorithm is that these parameters related to the enhancement features do not need to be updated or learned because they are generated entirely by nonlinear transformations and are no longer updated using backpropagation.

[0083] In this embodiment, multiple random numbers can be used to store and reproduce the hyperparameters of the nonlinear transformation. This allows the ground station server to complete the deployment and updating of the trained convolutional neural network model simply by sending a small number of seed convolutional kernels and random numbers to the low Earth orbit satellite. Therefore, the SineFM algorithm is suitable for ground station servers, assisting in the deployment and updating of convolutional neural network models for low Earth orbit satellites.

[0084] To complete the deployment and updating of the convolutional neural network model, the ground station server in this application only needs to send a small number of seed convolution kernels and random numbers to the low Earth orbit satellite. Compared with the traditional method of sending all model parameters, this application can significantly reduce the number of parameters transmitted from the ground station server to the low Earth orbit satellite, thereby greatly reducing the consumption of bandwidth resources, reducing the amount of data transmission to alleviate the problem of long latency when transmitting a large amount of data, and at the same time, updating the model at the cost of less bandwidth consumption.

[0085] This invention significantly reduces the floating-point computation cost of convolutional neural networks by using the SineFM algorithm to generate feature maps (including seed feature maps and mapped feature maps). Furthermore, training the convolutional neural network using a ground station server is advantageous because low-Earth orbit satellites, due to size and weight limitations, cannot provide sufficient computational resources for training high-performance convolutional neural network models, while ground station servers have ample computing resources.

[0086] In one embodiment of the present invention, the SineFM algorithm specifies that each layer in the convolutional neural network model only needs to learn one convolutional kernel parameter, which is called the seed convolutional kernel and is generated by a random seed. First, in the first layer of the convolutional neural network, a feature map (called the seed feature map) is generated using the original input image data and the seed convolutional kernel. Subsequently, the seed feature map generates other feature maps (i.e., mapped feature maps) for that layer based on certain preset rules, such as a nonlinear transformation function. This avoids generating all feature maps by using multiple convolutional kernels on the input data one by one. Using this method to generate feature maps in each layer can significantly reduce the computational load while maintaining performance similar to other convolutional neural networks.

[0087] It should be noted that during training, for the first layer of the convolutional neural network, the input data is the original input remote sensing image data. A seed convolution kernel is used to perform a convolution operation on the original input remote sensing image data to obtain the corresponding seed feature map. According to a preset rule, the seed feature map is nonlinearly transformed to obtain multiple mapping feature maps. The seed feature map and the mapping feature map are used as the input data for the next layer. For each layer in the convolutional neural network other than the first layer, the seed feature map and mapping feature map obtained from the previous layer are used as the input data for the next layer.

[0088] During training, for a given layer (such as a convolutional layer), for the input data x of that layer, this invention generates seed convolution kernels using random seeds s, and performs convolution operations on the input data using a small number of seed convolution kernels W(s) (*), thereby learning a small number (one or several) of seed feature maps y. (s) As shown in equation (1):

[0089] y (s) =W (s) *x (1)

[0090] Then, by analyzing the seed feature map y (s) The remaining feature map y is generated by applying a nonlinear transformation φ(·). (g) That is, multiple mapping feature maps are generated from the seed feature map through nonlinear transformation, as shown in equation (2):

[0091]

[0092] Where, β i It is the exponent of a single-term function applied to the i-th channel of the seed feature map, yi (g) This represents the i-th mapping feature map generated by the nonlinear transformation.

[0093] It should be noted that performing a nonlinear transformation on the seed feature map will generate multiple mapped feature maps. The index of the mapped feature map is represented by i, which can also be understood as the i-th channel of the seed feature map.

[0094] The framework proposed in this application has achieved good performance in semantic segmentation services. The proposed convolutional neural network model based on the SineFM algorithm has several times fewer parameters and floating-point computations compared to the best existing schemes, and is suitable for low Earth orbit satellites with limited resources.

[0095] The following is a pseudocode example of the SineFM algorithm. The pseudocode for SineFM in a PyTorch-like format is shown below:

[0096]

[0097] In this implementation, the seed convolution kernel is used as part of the convolution kernel group during forward propagation, i.e., [y (s) , ~y (g) The input data x undergoes initial processing through these seed convolutional kernels and is then transformed into m response maps by a nonlinear transformation φ(·), i.e., the self.conv function in the diagram. These m response maps are then passed through an activation gate, such as the ReLU activation function, to become m feature maps. These m feature maps are linearly combined using m learnable weights, which can be operated on through a 1×1 convolutional layer. α represents the 1×1 convolution weights, and α(·) is an activation, such as the func function in the code above, which can be specified as the ReLU function. Finally, the m feature maps are passed through the self.linear function to output a feature map y_g that has undergone linear transformation and normalization (normalize function).

[0098] The above process can be represented by equation (3):

[0099]

[0100] In a certain layer of a convolutional neural network, a small square of size C·k·k at a certain location in the input image is vectorized into a column vector x with C·k·k rows. π And vectorize all the convolutional filters generated by this layer into a C·k·k matrix w. For each position π, x π Multiply the value by each column of w, and then apply a non-linear function (such as ReLU) to obtain an output feature value d. π This process ultimately yields an output feature map, where each output feature value is obtained by convolving the entire input image with a convolutional filter. In a standard convolutional neural network, a single output feature value d is generated at the corresponding position π on the feature map. π .

[0101] Similarly, each value of the output feature map is a direct result of convolving the entire input map x with a convolution kernel w. This microprocess can be represented by equation (4):

[0102] d π =σ relu (w T ;x π (4)

[0103] For the layer proposed in this invention (using the monomial as the choice of the φ(·) feature generation function for illustration), each seed feature in the seed feature map is the input data x and the seed convolution kernel w. (s) The direct result of convolution. Then, for the exponential coefficients β1, β2, ..., β... m The seed features are subjected to a nonlinear transformation to obtain a set of m feature maps. β m It is predefined and will not be updated during training. The corresponding output feature map value. It is also the corresponding element in the m-response mapping through parameters α1, α2, ... α m Perform a linear combination of 1×1 convolutions.

[0104] It should be noted that seed features are the feature values ​​in the seed feature map.

[0105] The above process is shown in equation (5):

[0106]

[0107] Where, φβ i (·) is the exponent β i A point-by-point, single-item expansion. Comparison. and d π We consider the following two cases:

[0108] (1) When d π When c = 0, relu =σ relu When ([···])≥0, there always exists a vector α∈R m ×1, making

[0109] However, when (2)d π When c > 0, it is obvious that when c relu When =0, the approximation does not hold.

[0110] Therefore, in c relu Under the assumption that it is not a vector of all zeros, approximation It will be established.

[0111] Table 1 shows the nonlinear transformation functions in the SineFM algorithm, as follows:

[0112] Table 1

[0113]

[0114] In this invention, many nonlinear functions can be used to generate feature maps. This embodiment primarily considers monomials, polynomials, radial basis functions, and sine functions because of their well-studied theoretical properties and popularity in the prior art. This invention empirically determines the selection of feature generating functions through reduction analysis. For families of polynomials and radial basis functions, this invention has multiple candidate functions; the optimal choice is first determined within their respective families.

[0115] Appendix Figure 4 To be continued Figure 6 The experimental results are described, showing that the Legendre function outperforms other alternative polynomial functions, all radial basis functions exhibit similar performance, the Gaussian function is slightly stable, and the sine function outperforms other candidate functions. Therefore, the sine function is used to generate feature maps for the main results proposed in this invention. That is, in this embodiment, the sine function can be used as a nonlinear transformation function to nonlinearly transform the seed feature map to generate multiple mapped feature maps.

[0116] Tables 2 to 5, Appendix Figure 7 and attached Figure 8 The results are shown in Tables 2 to 5, which are the results on four remote sensing image datasets. The framework proposed in this invention achieves the least amount of floating-point computation and the least amount of model parameter transmission.

[0117] Table 2

[0118]

[0119] Table 3

[0120]

[0121] Table 4

[0122]

[0123] Table 5

[0124]

[0125] like Figure 7As shown, we consider two widely used architectures (VGG11 and ResNet18) and compare the floating-point computation and learnable parameters of semantic segmentation applications based on the SineFM model. CNN, GhostNet, and MonoCNN contain numerous sliding window operations for generating feature maps, resulting in significant floating-point computation. However, the method provided in this invention only requires a sliding operation at each layer of the model to generate a seed feature map. Then, multiple other mapped feature maps are generated based on the seed feature map through nonlinear transformations, avoiding numerous sliding window operations and thus greatly reducing the floating-point computation.

[0126] Since all convolutional parameters in convolutional neural networks and GhostNet need to be learned, the ground station server needs to send all convolutional layer parameters to the low Earth orbit satellite, resulting in a large amount of model parameter transmission. In the framework proposed in this invention, each layer only needs to transmit one seed convolutional kernel parameter and one random number. This is because this invention can generate a seed feature map based on this seed convolutional kernel, and then generate other feature maps, i.e., mapped feature maps, for that layer based on the seed feature map and the nonlinear transformation. In this invention, the hyperparameters of the nonlinear transformation are randomly initialized and kept fixed, so these hyperparameters can be saved and copied using random numbers. Therefore, compared with transmitting all model parameters, the framework proposed in this invention can greatly reduce the amount of model parameter transmission and consistently achieve better performance.

[0127] In summary, the advantages of the framework proposed in this invention are: (1) replacing the feature mapping generated by nonlinear transformation with a feature mapping based on a sliding window effectively reduces the floating-point computation of the model. (2) the nonlinear transformation parameters are randomly generated and fixed, effectively reducing the learnable parameters of the model. (3) the nonlinear transformation function regularizes the model, thereby improving its performance. Therefore, the framework proposed in this invention is particularly suitable for ground station servers to assist in the deployment and updating of trained convolutional neural network models for near-Earth orbit satellites, in order to provide high-quality services.

[0128] Appendix Figure 9 To be continued Figure 12 For the analysis of several hyperparameters of the nonlinear transformation, the performance of the sine function used in this invention in generating mapping features depends critically on the selection of hyperparameters. The sine function is a periodic function, expressed by formula (6):

[0129] y=A*sin(ω*x+φ) (6)

[0130] Where A represents amplitude, ω represents angular frequency, φ represents phase, x represents independent variable, and y represents dependent variable.

[0131] Amplitude A represents the maximum offset of the sine wave, which is half the distance between the highest and lowest points in the waveform. Angular frequency ω represents the vibration velocity of the sine wave, measured in radians per second, and is equal to 2π / T, where T is the period of the sine wave, i.e., the distance between two adjacent peaks (or troughs). Phase φ represents the offset of the sine wave, i.e., the position of the waveform on the x-axis. The phase and angular frequency of the sine function are important parameters controlling the shape of the sine wave.

[0132] To understand the influence of angular frequency ω and phase ψ in a sine function, this invention gradually changes the parameter boundaries while keeping all other factors constant. Figure 9 and Figure 10 It was observed that setting the lower bound a to ≥1 improves performance. Specifically, the present invention determines that sampling ω and ψ from [1,2] and [1,5] respectively yields the best results. On the other hand, the present invention can simultaneously create multiple sine functions with different sets of ω and ψ, and uses a hyperparameter (i.e., fan-out ratio) to control this process. A reduction analysis was performed to understand the effect of the fan-out ratio. Figure 11 It can be seen that having more sine functions (i.e., a larger fan-out ratio) results in better performance. In particular, this invention achieves optimal performance when the fan-out ratio is set to 5.

[0133] Finally, the computational complexity of the method of the present invention can be controlled by the number of seed convolution kernels. Figure 12 The trade-off between performance and computational complexity is described. In this invention, using 16 seeded convolutional kernels achieves the optimal trade-off between minimizing the number of parameters and maximizing performance.

[0134] In this invention, a mapped feature map is generated from a seed feature map based on a nonlinear transformation function. The parameters of this nonlinear transformation function are not trained but randomly initialized. Therefore, the parameters of the nonlinear transformation function can be saved and reproduced using random numbers. Using the framework provided in this application, when deploying and updating a trained convolutional neural network model, the ground station server only needs to transmit one seed convolutional kernel parameter and one random number for each layer of the convolutional neural network model. The ground station server only needs to send some seed convolutional kernels and random numbers to the low Earth orbit satellite to complete the deployment and updating of the convolutional neural network model.

[0135] This application provides a theoretical analysis of layers based on the SineFM algorithm, proving that it can approximate standard convolutional layers well while achieving better performance. Extensive experiments were conducted on several remote sensing image datasets, achieving a reduction in the number of parameters and the amount of floating-point computation while slightly improving the mLOU index.

[0136] It is understood that the convolutional neural network in this invention is an efficient deep learning model for edge computing environments. By deploying the trained convolutional neural network on low Earth orbit satellites and utilizing the satellite's computing resources to process remote sensing images (such as semantic segmentation), the need to transmit raw data back to the ground for processing can be reduced, thereby improving processing efficiency and response speed. Each low Earth orbit satellite is equivalent to an edge computing node.

[0137] Please see Figure 13 The present invention also provides an embodiment of a remote sensing image segmentation device based on a convolutional neural network, comprising:

[0138] The neural network training module 11 is used to train a convolutional neural network using a ground station server based on a remote sensing image dataset to obtain a trained convolutional neural network. During training, for each layer of the convolutional neural network, the input data is convolved according to the seed convolution kernel to obtain a corresponding seed feature map. The seed feature map is then nonlinearly transformed according to a preset rule to obtain multiple mapping feature maps. The seed feature map and the mapping feature map are used as the input data for the next layer.

[0139] Wherein, the seed convolution kernel is a convolution kernel generated using a random seed, the seed feature map is a feature map generated using the seed convolution kernel, the mapping feature map is other feature maps in this layer besides the seed feature map, and the hyperparameters of the nonlinear transformation are randomly initialized and stored in random numbers;

[0140] The neural network data transmission module 22 is used to send the seed convolution kernels and random numbers corresponding to each of the trained convolutional neural networks to a resource-constrained low Earth orbit satellite using a ground station server;

[0141] The neural network deployment and usage module 33 is used to deploy the trained convolutional neural network using the low Earth orbit satellite according to the seed convolutional kernel and the random number, and to complete the segmentation task of the captured remote sensing image according to the trained convolutional neural network.

[0142] In a preferred embodiment, this example may further include:

[0143] The neural network update module 44 is used to update the trained convolutional neural network using the near-Earth orbit satellite based on the seed convolutional kernel and the random number.

[0144] In a preferred embodiment, the neural network training module performs a nonlinear transformation on the seed feature map according to a preset rule to obtain multiple mapped feature maps, including: the neural network training module performs a nonlinear transformation on the seed feature map according to a nonlinear transformation function to obtain multiple mapped feature maps.

[0145] Optionally, the nonlinear transformation function is a sine function.

[0146] Optionally, the parameters of the sine function include at least: phase and angular frequency, where phase represents the offset of the sine wave and angular frequency represents the vibration frequency of the sine wave.

[0147] In existing remote sensing services for low Earth orbit satellites, this invention achieves fewer parameters compared to other methods, greatly reducing bandwidth pressure in satellite services. It also reduces floating-point calculations in low Earth orbit satellites, simplifies satellite model deployment and computational pressure, and greatly promotes the development of remote sensing services and research on reducing the complexity of convolutional neural network models.

[0148] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0149] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0150] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0151] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0152] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0153] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A remote sensing image segmentation method based on convolutional neural networks, characterized in that, include: The convolutional neural network is trained using a ground station server based on a remote sensing image dataset. During training, for each layer of the convolutional neural network, the input data is convolved according to the seed convolution kernel to obtain the corresponding seed feature map. The seed feature map is then nonlinearly transformed according to a preset rule to obtain multiple mapping feature maps. The seed feature map and the mapping feature map are used as the input data for the next layer. Wherein, the seed convolution kernel is a convolution kernel generated using a random seed, the seed feature map is a feature map generated using the seed convolution kernel, the mapping feature map is other feature maps in this layer besides the seed feature map, and the hyperparameters of the nonlinear transformation are randomly initialized and stored in random numbers; The seed convolutional kernels and random numbers corresponding to each layer of the trained convolutional neural network are sent to a resource-constrained low Earth orbit satellite using a ground station server. The trained convolutional neural network is deployed using the low-Earth orbit satellite according to the seed convolutional kernel and the random number, and the segmentation task of the captured remote sensing image is completed according to the trained convolutional neural network. Multiple mapped feature maps are obtained by performing a nonlinear transformation on the seed feature map according to a preset rule, including: Multiple mapped feature maps are obtained by performing a nonlinear transformation on the seed feature map according to a nonlinear transformation function; The nonlinear transformation function is a sine function.

2. The remote sensing image segmentation method based on convolutional neural networks according to claim 1, characterized in that, After deploying the trained convolutional neural network using the low-Earth orbit satellite based on the seed convolutional kernel and the random number, the process further includes: The trained convolutional neural network is updated using the near-Earth orbit satellite based on the seed convolutional kernel and the random number.

3. The remote sensing image segmentation method based on convolutional neural networks according to claim 1, characterized in that, The parameters of the sine function include at least: Phase and angular frequency: Phase represents the offset of a sine wave, and angular frequency represents the vibration frequency of a sine wave.

4. A remote sensing image segmentation device based on a convolutional neural network, characterized in that, include: The neural network training module is used to train a convolutional neural network using a ground station server based on a remote sensing image dataset to obtain a trained convolutional neural network. During training, for each layer of the convolutional neural network, the input data is convolved according to the seed convolution kernel to obtain the corresponding seed feature map. The seed feature map is then nonlinearly transformed according to a preset rule to obtain multiple mapping feature maps. The seed feature map and the mapping feature map are used as the input data for the next layer. Wherein, the seed convolution kernel is a convolution kernel generated using a random seed, the seed feature map is a feature map generated using the seed convolution kernel, the mapping feature map is other feature maps in this layer besides the seed feature map, and the hyperparameters of the nonlinear transformation are randomly initialized and stored in random numbers; The neural network data transmission module is used to send the seed convolution kernels and random numbers corresponding to each of the trained convolutional neural networks to a resource-constrained low Earth orbit satellite using a ground station server. The neural network deployment and usage module is used to deploy the trained convolutional neural network using the low Earth orbit satellite according to the seed convolutional kernel and the random number, and to complete the segmentation task of the captured remote sensing image according to the trained convolutional neural network. The neural network training module performs a nonlinear transformation on the seed feature map according to preset rules to obtain multiple mapped feature maps, including: The neural network training module performs a nonlinear transformation on the seed feature map according to a nonlinear transformation function to obtain multiple mapped feature maps; The nonlinear transformation function is a sine function.

5. The remote sensing image segmentation device based on a convolutional neural network according to claim 4, characterized in that, Also includes: The neural network update module is used to update the trained convolutional neural network using the near-Earth orbit satellite based on the seed convolutional kernel and the random number.

6. The remote sensing image segmentation device based on a convolutional neural network according to claim 4, characterized in that, The parameters of the sine function include at least: Phase and angular frequency: Phase represents the offset of a sine wave, and angular frequency represents the vibration frequency of a sine wave.

Images