Intelligent extraction method of remote sensing information based on large batch deep learning

Abstract

The application discloses a remote sensing information intelligent extraction method based on large batch deep learning, and utilizes LMDB and Horovod to realize distributed loading of remote sensing data and large batch deep learning processing, and utilizes learning rate preheating to accelerate the convergence speed of model weights; on the basis of the distributed model training framework Horovod and the memory mapping type database LMDB, in view of the problem that the framework occupies more memory resources during operation, a batch generation strategy is proposed to reduce the memory load; on the basis of the LAMB weight updating scheme, a hierarchical momentum decay strategy and a halving projection switching rule are proposed to optimize the learning rate adaptive preheating of the early convergence process of LAMB, so that the convergence speed of LAMB and the accuracy of gradient descent are improved.

Description

Technical Field

[0001] This invention relates to the field of remote sensing technology, and more specifically to a method for intelligent extraction of remote sensing information based on large-scale deep learning. Background Technology

[0002] With the development of remote sensing technology and the increasing popularity of the Internet, the diversification of remote sensing data acquisition methods has led to an explosive growth in the amount of information data. Utilizing relevant algorithms to process, analyze, and store massive amounts of data, and extracting its practical value for use in daily life and production, is a major challenge. Deep learning methods, by learning the inherent patterns and representational levels of sample data, can automatically and efficiently extract potentially meaningful data from satellite imagery, aerial imagery, and other data, gradually replacing traditional machine learning methods and becoming a research hotspot in many application fields. However, the increasing volume of remote sensing data means that the training process of deep learning models often takes several days or even months for the model weights to fully converge, and this enormous time overhead severely hinders the progress of related work. Currently, cluster computing based on multi-GPU parallel processing is used as the main acceleration method, which can effectively compensate for the computational and storage bottlenecks of single-machine processing and meet the needs of efficient processing of large-capacity data. Furthermore, with the continuous development of cluster computing technology, the development of parallel computing programming models such as CUDA and ROCM, and the support of common deep learning frameworks (such as PyTorch and TensorFlow) for cluster computing, a solid foundation has been laid for solving more complex distributed deep learning problems.

[0003] After years of research and refinement, the Horovod distributed deep learning framework, designed and developed by Uber, has garnered significant attention and favor from researchers and has been widely applied in related industries. The framework employs the industry-recognized Layer-wise Adaptive Moments optimizer for batch training (LAMB) algorithm, supplemented by ring communication, tensor fusion, and gradient compression to complete ensemble communication operations such as allreduce and allgather in MPI. This allows Horovod to integrate well with mainstream deep learning frameworks and provide efficient distributed acceleration on large-scale GPU clusters, demonstrating significant application value and potential for big data processing and interpretation. Horovod encapsulates distributed scheduling, distributed communication, and gradient computation on underlying deep learning frameworks such as Tensorflow, enabling large-scale data-parallel model training tasks. The overall design adopts a layered approach, primarily comprising a resource scheduling layer, a training framework layer, a communication framework layer, a distributed strategy layer, and an external access API layer. The resource scheduling layer shields the upper layers from the processes of resource allocation, management, and reclamation. The model training layer employs the LAMB algorithm to complete batch sample training tasks for weight updates, offering advantages such as high computational efficiency and fast update speed, while effectively overcoming the accuracy loss caused by batch expansion. The communication framework layer uses the industry-standard MPI method for inter-node communication. The distributed strategy layer uses Allreduce within a single node and Ring-Allreduce between nodes to achieve high-performance gradient aggregation in different scenarios. The application layer provides simple and easy-to-use programming interfaces such as DistribOptimizer, enabling better integration with existing Tensorflow and PyTorch programming methods.

[0004] Using the Horovod framework for data-parallel model training presents the following problems: 1. Although increasing the parallelism scale can effectively improve computational efficiency, loading subsets still incurs significant time overhead when the number of processes is limited. 2. Each process processes only a small batch of data per iteration, while subsets occupy substantial storage resources until the training task ends, resulting in significant resource waste. 3. As the scale of data parallelism continues to expand, the LAMB optimizer algorithm in large-batch environments cannot achieve satisfactory gradient descent results, exhibiting slow model weight convergence and decreased model prediction accuracy.

[0005] Therefore, providing a method for intelligent extraction of large-scale deep learning remote sensing information with higher computational accuracy and more memory-efficient processing has become a worthwhile research topic. Summary of the Invention

[0006] The purpose of this invention is to provide a method for intelligent extraction of remote sensing information in large-scale deep learning with higher computational accuracy and lower memory consumption. It utilizes LMDB and Horovod to achieve distributed loading and large-scale deep learning processing of remote sensing data, and accelerates the convergence speed of model weights through learning rate preheating. Based on the distributed model training framework Horovod and the memory-mapped database LMDB, this invention proposes a batch generation strategy to reduce memory load, addressing the issue of high memory consumption during framework runtime. Furthermore, based on the LAMB weight update scheme, it proposes a hierarchical momentum decay strategy and a halving projection switching rule to adaptively preheat the learning rate in the early convergence process of LAMB, thereby improving the convergence speed and gradient descent accuracy of LAMB.

[0007] The objective of this invention is achieved as follows:

[0008] The intelligent extraction method for remote sensing information based on large-scale deep learning includes the following steps:

[0009] Step S1: Establish a large-scale parallel training framework for data:

[0010] Remote sensing data is stored in key-value format using the memory-mapped database LMDB, and data items are located and retrieved using pointers that consume a small amount of memory. Then, it is combined with the Horovod framework to distribute the loaded sub-data evenly to multiple GPU devices for processing, and to improve the efficiency of large-capacity data processing by using multi-machine parallelism to achieve large-batch training.

[0011] Step S2: Reduce memory usage during training by using a batch generation strategy.

[0012] On a single GPU device in step S1, a batch generation method is used to divide the sub-data in step S1 into batches. In each iteration, only one batch of data is taken out for gradient calculation and model weight update, which effectively reduces the consumption of memory resources during model training.

[0013] Step S3, Batch Training Optimizer Algorithm:

[0014] In step 2, when updating the model weights, the MD-LAMB optimizer algorithm is used to accelerate the convergence speed of model weights in a large batch environment by "warming up" the learning rate. A switching point rule is designed to terminate the "warming up" process at an appropriate time to ensure the stability of subsequent weight updates.

[0015] The specific steps of step S1 are as follows: Since there are significant differences in the types and sizes of remote sensing data, the first step is to normalize the data to be processed so that the data to be processed is limited to a certain range, thereby eliminating the adverse effects of outlier sample data.

[0016] (1)

[0017] in Represents the sample mean. The variance of the sample is represented by i, which is the component label. The standardization correction is performed item by item by calculating the mean and variance of the individual data to make the characteristics of each dimension of the data more balanced.

[0018] LMDB is used to create the dataset file, and the Lmdb.Environment method is used to create the database environment. The preprocessed image blocks are stored in the database environment in key-value format, where the key is the data item and its sequence number, and the value is the binary form of the data item, which is converted using the Tobytes method. Finally, the Commit method is used to commit the storage transaction and complete the data storage.

[0019] The horovod framework is initialized via Hvd.init, which binds each GPU to a process to avoid resource conflicts and assigns a unique process number (rank) to each GPU for resource allocation. At the same time, model weights with rank 0 are transmitted to other processes via Broadcast, thereby completing the synchronous initialization of model weights between processes.

[0020] Each process accesses the LMDB's data storage environment through the Env.begin() function, obtains a pointer to a data item, and loads the specified data into memory by traversing it. To prevent different processes from repeatedly reading the same data, the remainder method is used to determine the ownership of the data item.

[0021] (2)

[0022] Where index is the pointer sequence number, N is the number of data items, and Final-rank is the program number to which it belongs;

[0023] The model training task is carried out based on a subset of data items read from LMDB. The descent gradient of the model weights is calculated by the error between the predicted value and the ideal value. To ensure the synchronous update of the model weights, AllReduce and Allgather provided by Hvd are used to communicate the gradients between nodes. At the end of the training task, each process will have the same model weights to evaluate the model accuracy.

[0024] The specific steps of step S2 are as follows:

[0025] During initialization, the entire dataset is divided into batches. All processes traverse the dataset once and obtain a pointer list Cur_List, which is used to store the pointer to the first data item in each batch.

[0026] At the beginning of each iteration, the Cur_List in rank 0 is randomly shuffled and broadcast to other processes, so that all processes obtain a unified shuffled list;

[0027] Each process reads a batch of data based on the scrambled list and the pointer to the first data item to update the weights of the deep learning model; after obtaining the local gradient, the process gradient is passed and communicated through the Ring-Allreduce algorithm to obtain the global gradient to get the weight update result;

[0028]

[0029] Global Grads = Ring-Allreduce(Local Grads)(4)

[0030] Global Weight = Optimizer(Global Grads)(5)

[0031] Local Grads are the local gradients calculated by each process based on the batch of data read. Global Grads are the global gradients obtained by integrating them through a ring algorithm. Finally, the global weights are obtained by the optimizer algorithm.

[0032] The specific steps of step S3 are as follows:

[0033] Adaptive learning rate correction includes two steps: warm-up and termination. The warm-up step accelerates the convergence of model weights by increasing the learning rate. The termination step stops the warm-up process when the model weights converge. The weight update scheme is as follows:

[0034] (6)

[0035] (7)

[0036] (8)

[0037] (9)

[0038] Wherein, constant factor To prevent the denominator from being 0; let , Let be the L2 norm value of the Adam gradient of the layer after the t-th iteration. It is used to measure the convergence of the current layer; This represents the value of the time-varying parameters of the algorithm after the t-th iteration. Represents L2 norm, To prevent the denominator from being 0 for smaller numbers, For attenuation coefficient, The base learning rate; the scheme uses Adam gradients. As the main direction of updates, among them The first-order momentum matrix, It is a second-order momentum matrix. This achieves a regularization effect;

[0039] use As the expected value of the optimal gradient direction, a larger value indicates a faster convergence rate when the gradient is larger, while a smaller value indicates a reduced momentum decay rate to prevent deviation from the optimal solution; simultaneously, the hyperparameters... <1 is used to control the weight of the historical rate of change, so that the growth of the learning rate is also affected by the accumulated historical rate of change, thus alleviating the gradient divergence problem.

[0040] The convergence trend is predicted using the three most recent Adam gradients, and the convergence status of the model layers is determined.

[0041] (10)

[0042] (11)

[0043] (12)

[0044] In each iteration At the end, take the last three times. , , Construct the gradient rate of change vectors for the t-th, t-1, and t-2th iterations. , The convergence degree of the model layer is determined by the mean of the half-projection of the two; when the mean of the half-projection is less than a threshold... Stop The momentum decays, and finally, a modified Sigmoid function is used to restore the magnitude of the momentum, which is defined as:

[0045] (13)

[0046] in These are hyperparameters and can be adjusted. The value controls the degree of momentum recovery, deviation Then it is determined by the model convergence point. If all layers in the model satisfy the switching rule at the t-th iteration, The calculation rules are as follows:

[0047] .

[0048] The beneficial effects of this invention are: 1. Reduced memory resource consumption: By combining with the LMDB database, this invention enables data location using pointers, allowing Horovod to locate data items without loading the entire large dataset into memory during dataset processing, instead utilizing pointers with lower memory consumption. Based on the combination of Horovod and LMDB, a batch generation strategy is proposed for batch data processing, enabling the computation of large datasets with minimal memory consumption by loading and processing only one batch of data at a time.

[0049] 2. Faster Model Convergence: This invention designs an adaptive preheating optimizer algorithm to "warm up" and optimize the large-scale model training process. In the hierarchical momentum decay strategy, the algorithm can provide different learning rates for different layers based on their convergence status and appropriately amplify them using the layer-based gradient L2 norm, enabling the model to reach convergence faster. Then, a halving projection switching rule is designed to determine the optimal convergence state of the model and stop the preheating process to ensure the stability of subsequent convergence. With the optimization of the above strategies, this invention achieves faster convergence speed and effectively guarantees accuracy compared to the original algorithm. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the data parallel remote sensing data processing framework combining a batch generation strategy proposed in this invention;

[0051] Figure 2 This is a schematic diagram of the MD-LAMB optimizer method proposed in this invention;

[0052] Figure 3 Three small datasets were used in the implementation examples of this invention;

[0053] Figure 4 This is a large dataset used in an embodiment of the present invention;

[0054] Figure 5 This refers to the memory usage rate during the runtime of the framework of this invention;

[0055] Figure 6 This refers to the time speedup ratio of the framework of this invention under different degrees of parallelism;

[0056] Figure 7This invention presents a comparison of the convergence of MD-LAMB and the original LAMB under different batch sizes;

[0057] Figure 8 The image shows the extraction results of various deep learning models of this invention in a 20k environment.

[0058] Figure 9 This diagram illustrates the computational efficiency of the framework of this invention under different degrees of parallelism. Detailed Implementation

[0059] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0060] The specific steps of steps S1 and S2 are as follows: Figure 1 As shown. First, the remote sensing image is segmented and standardized, as shown in Equation (1). Preprocessing is used to address the effects of different image sizes and singular values. At the same time, all image blocks are divided into single batches (e.g., 32). The framework assigns a pointer to the first image block of each batch and constructs a pointer list Cur_List for image location.

[0061] After the initialization operation of GPU binding in the Horovod framework, the framework allocates resources according to the unique process number. Different image blocks will obtain their assigned process number according to equation (2). At the beginning of each iteration, Rank0 will randomly shuffle the Cur_List and broadcast it to other GPUs to randomly shuffle the list. Then, each GPU reads the batch data according to the shuffled list and performs independent forward propagation and backward update processes as shown in equations (3), (4), and (5). The gradients of different processes are aggregated through AllReduce and Allgather to achieve weight integration.

[0062] S3 specific steps are as follows: Figure 2 As shown, the framework employs an adaptive preheating optimizer algorithm for the preheating process of weight updates. Weight updates are performed according to equation (6). The learning rate is adjusted to increase the degree of amplification, so that the model can reach the convergence state more quickly. At the same time, at the end of each iteration, the model convergence is judged according to formulas (10), (11), and (12). When the conditions are met, the warm-up process is stopped to ensure the stability of subsequent convergence.

[0063] . Figure 3 This is a Massachusetts Buildings Dataset consisting of 151 aerial images of the Boston area with a spatial resolution of 1m, covering an area of ​​approximately 340 square kilometers. Each image has a resolution of 1500x1500 pixels; the Wuhan Aerial Dataset includes 450 images from Christchurch and New Zealand. The dataset contains 0.075m aerial imagery. The original dataset consists of 8189 images (512x512 pixels) and corresponding labels. The dataset was divided into a training set of 4736 images (130,500 buildings) and a test set of 3452 images (56,500 buildings). The Wuhan Satellite Dataset II (WHU Satellite Dataset II) covers 550 cities in East Asia. The ground resolution is 0.45m, containing a total of 34,085 buildings. The original imagery consists of 17,388 data samples of 512×512 pixels, of which 13,662 pixels (containing 25,749 buildings) are used for model training, and the remaining 3,726 pixels (containing 8,358 buildings) are used for model testing. To accommodate memory space limitations, during actual training, the images from the three datasets are uniformly cut to 128x128 pixels and standardized before being fed into the model.

[0064] Figure 4 The Phoenix, USA building dataset showcased consists of 19 levels of 0.5m spatial resolution remote sensing satellite imagery taken by Google Earth between May and August 2021, covering an area of ​​2225.3575 square kilometers, with a data size of 58.61GB. The sample labels were generated from building vector maps provided by the OSM (Open Street Map) open-source wiki map and manually corrected to improve accuracy. The dataset is divided into a 4:1 ratio, 1792*1792 resolution training set (1825 images) and a test set (499 images). To accommodate memory constraints, each image was further cropped to 128*128 pixels and standardized during preprocessing before being fed into the model.

[0065] Figure 5To assess memory usage during processing of the three datasets, eight GPUs were used for acceleration, with a total batch size of 256. Optimized is the framework that integrates batch generation strategies, while Original is the original framework. At the same parallel scale, the F1 accuracy of the two frameworks on different datasets is similar, but there are significant differences in CPU and memory usage. On the Massa dataset, the optimized algorithm reduced CPU and memory usage by 8GB and the memory utilization rate by 1.2% compared to the original algorithm; for the Wuhan aerial photography dataset, it was 58GB and 3.1%; and for the Wuhan satellite dataset, it was 32GB and 0.8%. Because the batch generator algorithm loads only one batch of data into memory at a time, it does not need to completely partition the entire dataset into memory, and the CPU only needs to record pointer sequences instead of the data itself, thus greatly saving memory overhead. At the same time, the efficient pointer mapping operation also effectively alleviates the latency problem caused by I / O communication.

[0066] Figure 6 The study demonstrates the time speedup for three datasets under different parallelism levels. While the three datasets differ in size and number of images, they all achieve similar speedup results as parallelism increases, indicating that the batch generation strategy has good stability and scalability. Because the strategy loads only one batch of data into memory per iteration, it reduces the framework's memory requirements, allowing it to support larger datasets without compromising computational efficiency as the data size increases.

[0067] Figure 7 This image shows a convergence comparison between the proposed MD-LAMB and the original LAMB under batch sizes of 7k, 10k, 12k, 15k, 17k, and 20k. MD-LAMB and LAMB have similar F1 accuracy during convergence, but MD-LAMB achieves a 1.7x convergence speedup compared to LAMB at a batch size of 20k, demonstrating higher model convergence efficiency. This speedup is generally positively correlated with batch size. To accelerate the convergence process, MD-LAMB uses a warm-up method to increase the learning rate, enabling the model to reach the convergence point faster without deviating from the optimal direction or experiencing gradient divergence. Furthermore, the switching rule automatically terminates the warm-up process; for example, in batches of 7k and 10k, the warm-up process is terminated only in the 7th and 8th training rounds, resulting in relatively similar model convergence points.

[0068] Figure 8The paper showcases the building extraction results of popular deep learning models SegNet, RAPNet, and MGAR under 20k batch and MD-LAMB optimization. It can be seen that the three structures effectively solve the problem of small target adhesion under the optimization of the algorithm, ensuring individual independence. In addition, they can also identify large building targets, achieving good large batch training results.

[0069] Figure 9 The computational efficiency of MD-LAMB and Phoenix data under different degrees of parallelism and models is demonstrated. Overall, the number of GPUs used is directly proportional to the computational efficiency. However, in the comparison with the ideal situation corresponding to a single node, it can be clearly seen that although the utilization rate of the devices declines with the increase of the number of devices, a good acceleration effect can still be achieved.

Claims

1. A method for intelligent extraction of remote sensing information based on large-scale deep learning, characterized in that: Includes the following steps: Step S1: Establish a large-scale parallel training framework for data: Remote sensing data is stored in key-value format using the memory-mapped database LMDB, and data items are located and read using pointers that consume a small amount of memory. Then, it is combined with the Horovod framework to distribute the loaded sub-data evenly to multiple GPU devices for processing, and large-scale training is achieved by using multi-machine parallelism. Step S2: Reduce memory usage during training by using a batch generation strategy. On a single GPU device in step S1, the sub-data in step S1 is divided into batches using a batch generation method. In each iteration, only one batch of data is taken out for gradient calculation and model weight update. Step S3: Design a large-scale training optimizer algorithm: The MD-LAMB optimizer algorithm is used when updating the model weights in step S2. The specific steps of step S3 are as follows: Adaptive learning rate correction includes two steps: warm-up and termination. The warm-up step accelerates the convergence of model weights by increasing the learning rate. The termination step stops the warm-up process when the model weights converge. The weight update scheme is as follows: （6） （7） （8） ; where constant factor To prevent the denominator from being 0; let , Let be the L2 norm value of the Adam gradient of the layer after the t-th iteration. It is used to measure the convergence of the current layer; This represents the value of the time-varying parameters of the algorithm after the t-th iteration. Represents L2 norm, To prevent the denominator from being 0 for smaller numbers, For attenuation coefficient, The base learning rate; the scheme uses Adam gradients. As the main direction of updates, among them The first-order momentum matrix, It is a second-order momentum matrix. This achieves a regularization effect; use As the expected value of the optimal gradient direction, a larger value indicates a faster convergence rate when the gradient is larger, while a smaller value indicates a reduced momentum decay rate to prevent deviation from the optimal solution; simultaneously, the hyperparameters... <1 is used to control the weight of historical rate of change; The convergence trend is predicted using the three most recent Adam gradients, and the convergence status of the model layers is determined. （10） （11） （12） At the end of each iteration, take the three closest values. , , Construct the gradient rate of change vectors for the t-th, t-1, and t-2th iterations. , The convergence degree of the model layer is determined by the mean of the half-projection of the two; when the mean of the half-projection is less than a threshold... Stop The momentum decays, and finally, a modified Sigmoid function is used to restore the magnitude of the momentum, which is defined as: （13） in These are hyperparameters, adjusted by... The value controls the degree of momentum recovery, deviation Then it is determined by the model convergence point. If all layers in the model satisfy the switching rule at the t-th iteration, The calculation rules are as follows: （14）。 2. The intelligent remote sensing information extraction method based on large-scale deep learning according to claim 1, characterized in that: The specific steps of step S1 are as follows: Since there are significant differences in the types and sizes of remote sensing data, the first step is to normalize the data to be processed so that the data to be processed is limited to a certain range, thereby eliminating the adverse effects of outlier sample data. ;in Represents the sample mean. The variance of the sample is represented by i, which is the component label. The standardization correction is performed item by item by calculating the mean and variance of the individual data to make the characteristics of each dimension of the data more balanced. LMDB is used to create the dataset file, and the Lmdb.Environment method is used to create the database environment. The preprocessed image blocks are stored in the database environment in key-value format, where the key is the data item and its sequence number, and the value is the binary form of the data item, which is converted using the Tobytes method. Finally, the Commit method is used to commit the storage transaction and complete the data storage. The horovod framework is initialized via Hvd.init, which binds each GPU to a process to avoid resource conflicts and assigns a unique process number (rank) to each GPU for resource allocation. At the same time, model weights with rank 0 are transmitted to other processes via Broadcast, thereby completing the synchronous initialization of model weights between processes. Each process accesses the LMDB's data storage environment through the Env.begin() function, obtains a pointer to a data item, and loads the specified data into memory by traversing it. To prevent different processes from repeatedly reading the same data, the remainder method is used to determine the ownership of the data item. （2） Where index is the pointer sequence number, N is the number of data items, and Final-rank is the program number to which it belongs; The model training task is carried out based on a subset of data items read from LMDB. The descent gradient of the model weights is calculated by the error between the predicted value and the ideal value. To ensure the synchronous update of the model weights, AllReduce and Allgather provided by Hvd are used to communicate the gradients between nodes. At the end of the training task, each process will have the same model weights to evaluate the model accuracy.

3. The intelligent remote sensing information extraction method based on large-scale deep learning according to claim 1, characterized in that: The specific steps of step S2 are as follows: During initialization, the entire dataset is divided into batches. All processes traverse the dataset once and obtain a pointer list Cur_List, which is used to store the pointer to the first data item in each batch. At the beginning of each iteration, the Cur_List in rank 0 is randomly shuffled and broadcast to other processes, so that all processes obtain a unified shuffled list; Each process reads a batch of data based on the scrambled list and the pointer to the first data item to update the weights of the deep learning model; after obtaining the local gradient, the process gradient is passed and communicated through the Ring-Allreduce algorithm to obtain the global gradient to get the weight update result; ； Global Grads = Ring-Allreduce(Local Grads)(4); Global Weight = Optimizer(Global Grads)(5); Local Grads are the local gradients calculated by each process based on the batch of data read. Global Grads are the global gradients obtained by integrating them through a ring algorithm. Finally, the global weights are obtained by the optimizer algorithm.

Images