Remote sensing large model pre-training method and device, electronic equipment and storage medium

Abstract

The application provides a remote sensing large model pre-training method and device, electronic equipment and storage medium, comprising: dividing the received multi-spectral remote sensing image into multiple image blocks; calculating a remote sensing index for representing the environmental characteristics of the image block; calculating a semantic saliency index according to the remote sensing index; determining the current training round of the remote sensing large model, and calculating the final mask score of the image block according to the current training round, the total training round, and the semantic saliency index; selecting a target image block that meets the preset mask requirement in the multiple image blocks to perform mask to obtain a mask image block set; updating the parameters of the remote sensing large model according to the mask image block set; increasing the current training round, and again determining a new mask image block set according to a new mask final score to update the parameters of the remote sensing large model again until the new current training round is equal to the total training round. The application can improve the generalization ability and robustness of the remote sensing large model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing, and in particular to a remote sensing large model pre-training method and device, electronic equipment and a storage medium. BACKGROUND

[0002] In related technologies, pre-training and fine-tuning have become the core paradigm in the field of remote sensing image interpretation. Among them, pre-training based on masked autoencoder stands out due to its powerful ability, which can learn general feature representation by reconstructing the masked image area. The image is divided into small blocks and most of the areas are randomly masked, and then the network is forced to restore the global image only with the remaining context; in this process, the model is forced to extract general representations, laying a robust and transferable feature base for subsequent interpretation tasks.

[0003] However, due to the large width of multispectral remote sensing images and the open scene, the same pixel often mixes multiple ground objects such as buildings, vegetation, soil and water, and the spatial heterogeneity is extremely high. When the masked autoencoder is randomly masked, it is easy to mask all the "key discriminant blocks", and the remaining context may come from irrelevant categories, or may be dominated by large homogeneous objects due to the imbalance in proportion, causing the decoder to be forced to fit the average texture, and the complex boundaries and small structures are smoothed as noise; the reconstructed background looks smooth, but loses the multi-class edge and transition zone information required for actual interpretation, making the pre-training network insufficient to model the structure prior of complex scenes.

[0004] The original masked autoencoder completely randomly masks without considering the collaborative semantics between multispectral channels. For remote sensing images, random masking ignores the spatial neighborhood continuity, which easily cuts continuous water or farmland into fragmented fragments, causing the encoder to tend to learn local spectral clustering rather than joint spatial-spectral structure. The lack of semantic guided masking strategy ultimately hinders the effective mining and aggregation of meaningful spatial-spectral features by the model.

[0005] Therefore, there is an urgent need for a new remote sensing large model pre-training method to solve the above problems. SUMMARY

[0006] Therefore, the present application provides a remote sensing large model pre-training method and device, electronic equipment and a storage medium, which can improve the generalization ability and robustness of the remote sensing large model.

[0007] The first aspect of the embodiment of the present application provides a remote sensing large model pre-training method, comprising the following steps: dividing a received multi-spectral remote sensing image into a plurality of image blocks; for each image block, calculating a remote sensing index for representing environmental features of the image block; calculating a semantic saliency index for representing semantic information richness of the image block according to the remote sensing index; determining a current training round of a remote sensing large model, and calculating a mask final score of the image block according to the current training round, a preset total training round of the remote sensing large model, and the semantic saliency index; selecting a target image block satisfying a preset mask requirement in the mask final score from the plurality of image blocks to perform mask to obtain a mask image block set; updating parameters of the remote sensing large model according to the mask image block set; in the case that it is detected that the current training round is less than the total training round, increasing the current training round, calculating a new mask final score of the image block again according to the increased current training round, and determining a new mask image block set again according to the new mask final score to update the parameters of the remote sensing large model again until the new current training round is equal to the total training round.

[0008] In a possible implementation manner, the remote sensing index comprises a normalized difference vegetation index, a normalized difference water index, and a normalized building index; the calculating the remote sensing index for representing the environmental features of the image block comprises: calculating the normalized difference vegetation index according to the following formula: ; wherein, is the normalized difference vegetation index, is a near-infrared band of the multi-spectral remote sensing image, is a red band of the multi-spectral remote sensing image; the normalized difference water index is calculated according to the following formula: ; wherein, is the normalized difference water index, is a green band of the multi-spectral remote sensing image; and the normalized building index is calculated according to the following formula: ; wherein, is the normalized building index, is a short-wave infrared band of the multi-spectral remote sensing image.

[0009] In a possible implementation manner, each image block comprises P is an integer greater than 1; the calculation of the remote sensing index for representing the image block environment characteristics comprises: calculating the normalized difference vegetation index, the normalized difference water index and the normalized building index corresponding to each pixel point in the image block; the calculation of the semantic saliency index for representing the semantic information richness degree of the image block according to the remote sensing index comprises: for each image block, the first mean value and the first standard deviation of the normalized difference vegetation index of each pixel point, the second mean value and the second standard deviation of the normalized difference water index, and the third mean value and the third standard deviation of the normalized building index are calculated respectively; and the semantic saliency index is calculated according to the first mean value, the first standard deviation, the second mean value, the second standard deviation, the third mean value and the third standard deviation.

[0010] In a possible implementation, the calculation of the semantic saliency index according to the first mean value, the first standard deviation, the second mean value, the second standard deviation, the third mean value and the third standard deviation comprises: the semantic saliency index is calculated according to the following formula: ; wherein, the semantic saliency index, K is the number of remote sensing indexes, the mean value of the kth remote sensing index, the standard deviation of the kth remote sensing index, is a smoothing constant.

[0011] In a possible implementation, the calculation of the mask final score of the image block according to the current training round, the total training round of the preset remote sensing large model and the semantic saliency index comprises: the mask final score is calculated according to the following formula: ; ; wherein, the current training round, the total training round, the training progress of the remote sensing large model, the size of between 0 and 1, the semantic saliency index, random noise between 0 and 1.

[0012] In a possible implementation, the selection of the target image block satisfying the preset mask requirement in the mask final score in the plurality of image blocks to perform the mask to obtain a mask image block set comprises: the mask image block set is obtained according to the following formula: ; wherein, the mask image block set, ​a number of image blocks into which the multispectral remote sensing image is divided, a mask ratio.

[0013] In a possible implementation, the updating the parameters of the remote sensing large model according to the set of mask image blocks comprises: calculating an average absolute error and a normalized average absolute error corresponding to each mask image block in the set of mask image blocks; and updating the parameters of the remote sensing large model according to the average absolute error and the normalized average absolute error.

[0014] In a second aspect, the embodiments of the present application further provide a remote sensing large model pre-training apparatus, comprising: a division module, a first calculation module, a second calculation module, a third calculation module, a mask module, a parameter updating module, and a detection module; the division module is configured to divide a received multispectral remote sensing image into a plurality of image blocks; the first calculation module is configured to calculate, for each of the image blocks, a remote sensing index used to represent environmental features of the image block; the second calculation module is configured to calculate, according to the remote sensing index, a semantic saliency index used to represent a semantic information richness degree of the image block; the third calculation module is configured to determine a current training round of a remote sensing large model, and calculate a mask final score of the image block according to the current training round, a preset total training round of the remote sensing large model, and the semantic saliency index; the mask module is configured to select, from the plurality of image blocks, a target image block whose mask final score meets a preset mask requirement, and perform masking on the target image block to obtain a set of mask image blocks; the parameter updating module is configured to update parameters of the remote sensing large model according to the set of mask image blocks; and the detection module is configured to detect whether the current training round is less than the total training round, and in a case where the detection module detects that the current training round is less than the total training round, increase the current training round, the third calculation module calculates a new mask final score of the image block again according to the increased current training round, the mask module determines a new set of mask image blocks again according to the new mask final score, and the parameter updating module updates the parameters of the remote sensing large model again, until the detection module detects that the new current training round is equal to the total training round.

[0015] In a third aspect, the embodiments of the present application further provide an electronic device, comprising a processor and a memory, the memory being configured to store instructions, and the processor being configured to invoke the instructions in the memory, so that the electronic device performs the remote sensing large model pre-training method according to the first aspect.

[0016] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium storing computer instructions, when the computer instructions run on an electronic device, the electronic device is caused to perform the remote sensing large model pre-training method according to the first aspect.

[0017] Compared with the related art, the embodiments of the present application have at least the following advantages: by calculating a remote sensing index representing the environmental characteristics of the image block, then calculating the semantic saliency index of the image block based on the remote sensing index, and then calculating the mask final score of the image block according to the current training round and the total training round of the remote sensing large model and the semantic saliency index, and finally selecting a target image block that meets the preset mask requirement in the mask final score from the plurality of image blocks for masking, the remote sensing index widely used in the field of remote sensing can be fully utilized as prior knowledge to dynamically guide the mask strategy, so that the remote sensing large model can pay more attention to the spatial-spectral region with rich information in the training process. It is worth noting that, unlike simply relying on random masking, the present application can highlight the region with potential semantic value, and improve the discriminability and semantic sensitivity of the remote sensing large model in representation learning. In addition, by increasing the current training round and determining a new mask image block set in the next round when it is detected that the current training round is less than the total training round, the mask center is smoothly transitioned from the "information-rich area" to the "information-poor area". Without introducing any random noise, the remote sensing large model first focuses on the salient spectral features, then systematically mines the background and edge regions, realizes global-local consistent spatial-spectral representation learning, and automatically learns the features of ground objects under the guidance of the physical semantics of the remote sensing index, which significantly improves the accuracy and robustness of the downstream tasks.

[0018] The technical effects obtained by the above-mentioned second aspect, third aspect and fourth aspect are similar to the technical effects obtained by the corresponding technical means in the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 A step flowchart of the remote sensing large model pre-training method provided by an embodiment of the present application;

[0020] Figure 2 A functional module diagram of the remote sensing large model pre-training device provided by an embodiment of the present application;

[0021] Figure 3 A structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to enable a clearer understanding of the above-mentioned objects, features and advantages of the present application, the present application will now be described in detail with the following description and specific embodiments. It should be noted that the embodiments and the features in the embodiments of the present application can be combined with each other without conflict, if possible.

[0023] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. The described embodiments are merely some embodiments of the present application, but not all embodiments of the present application.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description herein is for describing some embodiments of the present application only and is not intended to be limiting of the present application.

[0025] It is further noted that the terms "comprise", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0026] In the present application, "at least one" means one or more, and "multiple" means two or more than two. The "and / or" describes the relationship between the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth" and the like (if any) in the specification and claims of the present application and the drawings are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0027] In the embodiments of the present application, the words "exemplary" or "for example" are used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the exemplary or example embodiments are presented as a means of enabling some aspects of the present application.

[0028] For the purpose of facilitating understanding, some explanations of concepts related to the embodiments of the present application are given as examples for reference.

[0029] Multispectral remote sensing image: is obtained by multispectral imaging equipment, each band image records the spectral radiation information of ground objects in the same ground range at different wavebands. Compared with ordinary RGB images, the core difference lies in the larger number of channels, which can capture more abundant spectral characteristics of ground objects.

[0030] Remote sensing large model: is a deep learning-based AI model, which is used to process remote sensing data, automatically identify ground objects, monitor changes and extract information.

[0031] Pre-training: is the basic stage of artificial intelligence model development, which learns general features through large-scale data, and can be divided into two stages of unsupervised pre-training and supervised fine-tuning. There are essential differences in target, data and application level between direct training.

[0032] Please refer to Figure 1 , Figure 1 is the step flow chart of an embodiment of the remote sensing large model pre-training method of the present application. According to different needs, the order of steps in the flow chart can be changed, and some steps can be omitted.

[0033] It should be noted that the remote sensing large model pre-training method of the embodiments of the present application can be applied to the pre-training scene of the remote sensing large model, and the execution subject thereof can be a remote sensing large model pre-training device, for example, the remote sensing large model can be pre-trained by the remote sensing large model pre-training device. Of course, the remote sensing large model pre-training method of the embodiments of the present application can also be applied to other scenes that need to pre-train the remote sensing large model, which is not limited in the present application.

[0034] The specific process of the embodiment is shown in Figure 1 , including the following steps:

[0035] S101, divide the received multispectral remote sensing image into multiple image blocks.

[0036] In some embodiments, the pre-training framework of the remote sensing large model is composed of an encoder and a decoder, and the encoder part first divides the received multispectral remote sensing image into image blocks of a fixed size.

[0037] Specifically, the encoder divides the multispectral remote sensing image into L image blocks according to a sliding window with a size of Each image block includes pixel points, and L and P are both integers greater than 1.

[0038] S102, for each image block, calculate a remote sensing index for representing the environmental characteristics of the image block.

[0039] In some embodiments, the remote sensing indices include: a normalized difference vegetation index, a normalized difference water index, and a normalized building index; calculating the remote sensing indices used to characterize the environmental features of an image patch includes: calculating the normalized difference vegetation index according to the following formula: ;in, The normalized differential vegetation index, For the near-infrared band of multispectral remote sensing images, The red band of the multispectral remote sensing image is used; the normalized difference water index is calculated using the following formula: ;in, The normalized difference water index, The green band of the multispectral remote sensing image is used; the normalized building index is calculated using the following formula: ;in, For the normalized building index, This refers to the shortwave infrared band of multispectral remote sensing images.

[0040] It is understood that the calculation of remote sensing indices in this embodiment includes calculating the normalized difference vegetation index, the normalized difference water index, and the normalized building index corresponding to each pixel in the image patch.

[0041] S103, Calculate the semantic saliency index based on the remote sensing index to characterize the richness of semantic information in image patches.

[0042] In some embodiments, calculating a semantic saliency index to characterize the semantic information richness of an image patch based on a remote sensing index includes: calculating, for each image patch, a semantic saliency index. The first mean and first standard deviation of the normalized difference vegetation index, the second mean and second standard deviation of the normalized difference water index, and the third mean and third standard deviation of the normalized building index are calculated for each pixel. The semantic significance index is calculated based on the first mean, first standard deviation, second mean, second standard deviation, third mean, and third standard deviation.

[0043] Specifically, the semantic saliency index is calculated using the following formula:

[0044] ;in, Here, K is the semantic saliency index, and K is the number of remote sensing indices. Let be the mean of the k-th remote sensing index. Let be the standard deviation of the k-th remote sensing index. This is the smoothing constant.

[0045] It is worth noting that the mean value is used to reflect the semantic certainty of the image block: the higher the mean value, the richer the semantic information contained in the image block, and thus the stronger the class discrimination ability and the more important the information. The standard deviation is used to represent the heterogeneity within the image block, reflect the reconstruction difficulty of the region, and measure the dispersion degree of information within it. On this basis, the semantic saliency index (SSI) is introduced to measure the semantic importance of each image block. The higher the SSI value, the richer the semantic information, the stronger the discriminability and the lower the internal heterogeneity, so it is more suitable for reconstruction and model training.

[0046] S104, determining the current training round of the remote sensing large model, and calculating the mask final score of the image block according to the current training round, the total training round of the preset remote sensing large model, and the semantic saliency index.

[0047] In some embodiments, the mask final score is calculated according to the following formula: ;

[0048] ; wherein, is the current training round, is the total training round, is the training progress of the remote sensing large model, the size is between 0 and 1, is the semantic saliency index, is random noise between 0 and 1.

[0049] S105, selecting target image blocks that meet the preset mask requirements from the plurality of image blocks to perform masking, to obtain a set of masked image blocks.

[0050] In some embodiments, the set of masked image blocks is obtained according to the following formula:

[0051] ; wherein, is the set of masked image blocks, is the number of image blocks into which the multispectral remote sensing image is divided, is the mask ratio.

[0052] Specifically, the low importance region to be masked is selected based on the mask final score. To this end, first, the scores of all image blocks of each image are sorted, and the corresponding patch indexes are arranged in descending order. The first ×L target image blocks are selected for masking.

[0053] S106, updating the parameters of the remote sensing large model according to the set of masked image blocks.

[0054] In some embodiments, updating the parameters of the remote sensing large model according to the mask image block set comprises: calculating the average absolute error and the normalized average absolute error corresponding to each mask image block in the mask image block set; and updating the parameters of the remote sensing large model according to the average absolute error and the normalized average absolute error.

[0055] Specifically, the average absolute error and the normalized average absolute error are calculated according to the following formula:

[0056] ; ; wherein, the average absolute error, the normalized average absolute error, the true pixel of the masked image block, the pixel of the corresponding image block reconstructed and generated by the remote sensing large model.

[0057] It is worth noting that the average absolute error only calculates the L1 reconstruction residual of the visible pixel to drive the encoder to extract high-frequency texture and edge information with light overhead; the normalized average absolute error normalizes the above residual according to the data range, eliminates the influence of dimension and dynamic range, and makes the remote sensing large model give equal punishment to “overestimation” and “underestimation”, preventing semantic drift caused by brightness deviation. The two jointly form the “high mask rate + visible area double-scale reconstruction” task, which forces the network to learn a transferable representation with absolute accuracy and relative robustness without increasing the amount of calculation, providing a general visual foundation for subsequent downstream tasks.

[0058] S107, detecting whether the current training round is less than the total training round, and in the case of detecting that the current training round is less than the total training round, increasing the current training round, and executing S104; otherwise, ending the process.

[0059] Specifically, in the case of detecting that the current training round is less than the total training round, the current training round is increased by 1, and then S104 is executed.

[0060] For ease of understanding, the pre-training method of the remote sensing large model of the present embodiment is specifically illustrated as follows:

[0061] 1. Collect multispectral remote sensing images (take Sentinel-2 satellite L2A data as an example, a total of 12 spectral channels), and uniformly crop the images to obtain image blocks with a size of 224x224. To enhance the generalization ability of the model, random rotation, flipping, spectral disturbance and other data enhancement operations can be performed on the basis of cropping.

[0062] 2. Based on the multispectral characteristics, calculate the commonly used spectral indices in remote sensing (such as normalized vegetation index NDVI, water index NDWI, building index NDBI, etc.), as the domain knowledge source for subsequent dynamic mask.

[0063] 3. Construct a deep learning model using a pre-trained framework with an encoder and decoder structure.

[0064] 4. Input the multispectral remote sensing image after data enhancement operation into the deep learning model. The encoder first divides the input image into fixed-size image blocks. Then, based on the corresponding multispectral channels of each image block, calculate the remote sensing index features, and separately calculate the mean and standard deviation of each index in the image block. The mean is used to measure the adequacy of semantic information in the image block, and the standard deviation is used to describe the dispersion and complexity of the spectral features within the image block. On this basis, calculate the semantic saliency index of each image block to measure its contribution in the spatial-spectral representation learning process.

[0065] 5. According to the progress information of the training process (i.e. the ratio of the current training round to the total training rounds) and the semantic saliency index of the image block, calculate the final mask score of the image block, and adopt a semantic adaptive progressive dynamic mask strategy. In the early stage of training, focus on masking the image block regions with high semantic importance, forcing the model to dig deep discriminative features in the case of relatively insufficient available information; as the training rounds progress, the mask center gradually shifts to the image block regions with relatively scarce semantic information, allowing the model to further learn the background, structure, and weak texture regions, thereby achieving global spatial-spectral representation optimization with "first saliency - then edge". Finally, perform mask operation on the top-ranked image blocks based on the final score, and complete the construction of dynamic noise-free mask.

[0066] 6. Input the visible image blocks that are not masked into the encoder, and after passing through multiple layers of self-attention mechanism and nonlinear transformation, obtain compressed and discriminative high-dimensional feature representation. Then, input the encoder output and masked image blocks into the decoder, and the decoder restores the occluded areas through feature interaction and reconstruction mapping, thereby achieving high-quality reconstruction of the input image. Through end-to-end training, the model can fully absorb the physical semantic information contained in the remote sensing index, adaptively learn the differences between land cover classes and spectral structure features, and significantly improve the generalization ability and robustness of the downstream tasks of the model.

[0067] 7. Calculate the difference between the reconstruction result and the original image on the visible image blocks, and the loss function is composed of mean square error and normalized mean square error, considering both spatial structure and spectral features. The optimization process uses an adaptive optimizer, combined with learning rate warm-up and cosine decay scheduling.

[0068] 8. Repeat steps (3) to (7) until a set number of iterations is reached or the model converges, completing the pre-training.

[0069] Compared with the related art, the embodiments of the present application have at least the following advantages: by calculating a remote sensing index representing the environmental characteristics of an image block, then calculating a semantic saliency index of the image block based on the remote sensing index, and then calculating a mask final score of the image block according to the current training round and the total training round of the remote sensing large model and the semantic saliency index, and finally selecting a target image block that meets the preset mask requirement in the mask final score from multiple image blocks for masking, the remote sensing index widely used in the field of remote sensing can be fully utilized as prior knowledge to dynamically guide the mask strategy, so that the remote sensing large model can focus on the spatial-spectral region with rich information in the training process. It is worth noting that, unlike simply relying on random masking, the present application can highlight the region with potential semantic value, and improve the discriminability and semantic sensitivity of the remote sensing large model in representation learning. In addition, by increasing the current training round and determining a new mask image block set in the next round when it is detected that the current training round is less than the total training round, the mask center is smoothly transitioned from the "information-rich area" to the "information-poor area". Without introducing any random noise, the remote sensing large model first focuses on the salient spectral features, then systematically mines the background and edge regions, realizes global-local consistent spatial-spectral representation learning, and automatically learns the features of ground objects under the guidance of the physical semantics of the remote sensing index, significantly improving the accuracy and robustness of downstream tasks.

[0070] Based on the same idea as the remote sensing large model pre-training method in the above embodiments, the present application also provides a remote sensing large model pre-training device, which can be used to execute the remote sensing large model pre-training method described above. For ease of illustration, only the parts related to the embodiments of the present application are shown in the structural diagram of the remote sensing large model pre-training device embodiment, and those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, which can include more or fewer components than the illustrated, or combine certain components, or different component arrangements.

[0071] As shown in Figure 2 The remote sensing large model pre-training device 20 includes a division module 201, a first calculation module 202, a second calculation module 203, a third calculation module 204, a mask module 205, a parameter update module 206, and a detection module 207. In some embodiments, the above-mentioned modules can be programmable software instructions stored in a memory and executable by a processor. It can be understood that in other embodiments, the above-mentioned modules can also be program instructions or firmware fixed in the processor.

[0072] The division module 201 is configured to divide the received multispectral remote sensing image into a plurality of image blocks;

[0073] The first calculation module 202 is configured to calculate, for each of the image blocks, a remote sensing index for representing an environmental feature of the image block.

[0074] The second calculation module 203 is configured to calculate, according to the remote sensing index, a semantic saliency index for representing a semantic information richness of the image block.

[0075] The third calculation module 204 is configured to determine a current training round of a remote sensing large model, and calculate a mask final score of the image block according to the current training round, a preset total training round of the remote sensing large model, and the semantic saliency index.

[0076] The mask module 205 is configured to select, from the plurality of image blocks, a target image block whose mask final score meets a preset mask requirement, and perform masking on the target image block to obtain a set of masked image blocks.

[0077] The parameter updating module 206 is configured to update parameters of the remote sensing large model according to the set of masked image blocks.

[0078] The detection module 207 is configured to detect whether the current training round is less than the total training round. In a case where the detection module detects that the current training round is less than the total training round, the current training round is increased, the third calculation module 204 calculates a new mask final score of the image block again according to the increased current training round, the mask module 205 determines a new set of masked image blocks again according to the new mask final score, and the parameter updating module 206 updates the parameters of the remote sensing large model again, until the detection module 207 detects that a new current training round is equal to the total training round.

[0079] The remote sensing large model pre-training apparatus 20 provided by the above embodiment can implement the technical solutions described in the remote sensing large model pre-training method embodiments. The principles of implementation of the above modules or units can be referred to the corresponding content in the remote sensing large model pre-training method embodiments, which will not be described herein again.

[0080] Please refer to Figure 3 , Figure 3 FIG. 1 is a schematic diagram of an embodiment of an electronic device according to the present application. In the embodiment of the present application, the electronic device 300 includes a processor 301, a memory 302, and a display 303. Figure 3 Only some components of the electronic device 300 are shown, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented.

[0081] The processor 301 may, in some embodiments, be a central processing unit (CPU), a microprocessor, or other data processing chip, for running program codes stored in the memory 302 or processing data, such as the remote sensing large model pre-training method in the present application.

[0082] In some embodiments, the processor 301 can be a single server or a group of servers. The group of servers can be centralized or distributed. In some embodiments, the processor 301 can be local or remote. In some embodiments, the processor 301 can be implemented on a cloud platform. In an embodiment, the cloud platform can include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an inter-organizational cloud, a multi-cloud, etc., or any combination thereof.

[0083] The memory 302 may, in some embodiments, be an internal storage unit of the electronic device 300, such as a hard disk or a memory of the electronic device 300. The memory 302 may, in other embodiments, also be an external storage device of the electronic device 300, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the electronic device 300.

[0084] Further, the memory 302 can also include both an internal storage unit and an external storage device of the electronic device 300. The memory 302 is used to store application software installed on the electronic device 300 and various types of data.

[0085] The display 303 may, in some embodiments, be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, etc. The display 303 is used to display information of the electronic device 300 and to display visualized user application programs. The components 301-303 of the electronic device 300 communicate with each other through a system bus.

[0086] In an embodiment, when the processor 301 executes the remote sensing large model pre-training program in the memory 302, the following steps can be implemented:

[0087] The received multi-spectral remote sensing image is divided into a plurality of image blocks;

[0088] For each of the image blocks, a remote sensing index is calculated for representing environmental features of the image block;

[0089] According to the remote sensing index, a semantic saliency index is calculated for representing the richness of semantic information of the image block;

[0090] determine a current training round of the remote sensing large model, and calculate a mask final score of the image block according to the current training round, a preset total training round of the remote sensing large model, and the semantic saliency index;

[0091] select a target image block satisfying a preset mask requirement from the image blocks according to the mask final scores, and perform mask on the target image block to obtain a mask image block set;

[0092] update parameters of the remote sensing large model according to the mask image block set;

[0093] In a case where it is detected that the current training round is less than the total training round, the current training round is increased, a new mask final score of the image block is calculated again according to the increased current training round, and a new mask image block set is determined again according to the new mask final score to update the parameters of the remote sensing large model again, until the new current training round is equal to the total training round.

[0094] It should be understood that, in addition to the above functions, the processor 301 can also implement other functions when executing the remote sensing large model pre-training program in the memory 302. For details, refer to the description of the corresponding method embodiments.

[0095] Further, the type of the electronic device 300 is not specifically limited, and the electronic device 300 can be a mobile phone, a tablet computer, a personal digital assistant (PDA), a wearable device, a laptop, or the like. Exemplary embodiments of the portable electronic device include, but are not limited to, a portable electronic device running an IOS, an android, a microsoft, or other operating system. The above portable electronic device can also be other portable electronic devices, such as a laptop having a touch-sensitive surface (e.g., a touch panel). It should also be understood that, in some other embodiments of the present application, the electronic device 300 can also be a desktop computer having a touch-sensitive surface (e.g., a touch panel).

[0096] Correspondingly, the embodiments of the present application also provide a computer readable storage medium for storing computer readable programs or instructions, which can implement the steps or functions of the remote sensing large model pre-training method provided by the above method embodiments when executed by a processor.

[0097] Those skilled in the art can understand that all or part of the processes of the above-mentioned embodiment methods can be instructed by a computer program to relevant hardware (such as a processor, a controller, etc.) to be completed, and the computer program can be stored in a computer readable storage medium. Among them, the computer readable storage medium is a magnetic disk, an optical disk, a read-only memory or a random access memory, etc.

[0098] The remote sensing large model pre-training method, device, electronic equipment and computer readable storage medium provided by the present application are described in detail above. The principles and implementation modes of the present application are described by applying specific examples. The above description of the embodiments is only used to help understand the method and its core idea of the present application. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A remote sensing large model pre-training method, characterized in that, The method comprises the following steps: dividing a received multispectral remote sensing image into a plurality of image blocks; for each of the image blocks, calculating a remote sensing index for representing environmental characteristics of the image block; calculating a semantic saliency index for representing the richness of semantic information of the image block according to the remote sensing index; determining a current training round of a remote sensing large model, and calculating a mask final score of the image block according to the current training round, a preset total training round of the remote sensing large model, and the semantic saliency index; selecting a target image block that meets a preset mask requirement from the plurality of image blocks for masking to obtain a mask image block set; updating parameters of the remote sensing large model according to the mask image block set; in a case where it is detected that the current training round is less than the total training round, increasing the current training round, calculating a new mask final score of the image block again according to the increased current training round, and determining a new mask image block set again according to the new mask final score to update the parameters of the remote sensing large model again until the new current training round is equal to the total training round; The remote sensing indexes include a normalized difference vegetation index, a normalized difference water index, and a normalized building index; each of the image blocks includes pixels, and P is an integer greater than 1. the calculating of the semantic saliency index for representing the richness of semantic information of the image block according to the remote sensing index comprises: For each of the image blocks, respectively calculate a first mean and a first standard deviation of the normalized difference vegetation index, a second mean and a second standard deviation of the normalized difference water index, and a third mean and a third standard deviation of the normalized built-up index for the pixel points in the image block. calculating the semantic saliency index according to the first mean value, the first standard deviation, the second mean value, the second standard deviation, the third mean value, and the third standard deviation.

2. The remote sensing large model pre-training method according to claim 1, wherein the calculating of the remote sensing index for representing the environmental characteristics of the image block comprises: calculating the normalized difference vegetation index according to the following formula: ; wherein is the normalized difference vegetation index, is a near infrared band of the multispectral remote sensing image, is a red band of the multispectral remote sensing image; calculating the normalized difference water index according to the following formula: ; wherein is the normalized difference water index, is the green band of the multispectral remote sensing image; calculating the normalized building index according to the following formula: ; wherein, is the normalized building index, is the short-wave infrared band of the multispectral remote sensing image.

3. The remote sensing large model pre-training method according to claim 2, wherein the calculating of the remote sensing index for representing the environmental characteristics of the image block comprises: calculating the normalized difference vegetation index, the normalized difference water index, and the normalized building index corresponding to each of the pixel points in the image block.

4. The remote sensing large model pre-training method according to claim 1, characterized in that, the calculating of the semantic saliency index according to the first mean value, the first standard deviation, the second mean value, the second standard deviation, the third mean value, and the third standard deviation comprises: calculating the semantic saliency index according to the following formula: ; wherein, is the semantic saliency index, K is the number of remote sensing indices, is the mean of the kth remote sensing index, is the standard deviation of the kth remote sensing index, is a smoothing constant.

5. The remote sensing large model pre-training method according to claim 1, characterized in that, the calculating of the mask final score of the image block according to the current training round, the preset total training round of the remote sensing large model, and the semantic saliency index comprises: calculating the mask final score according to the following formula: ； ; wherein, is the current training epoch, is the total training epochs, is the training progress of the remote sensing large model, is between 0 and 1, is the semantic saliency index, is a random noise between 0 and 1.

6. The remote sensing large model pre-training method according to claim 5, characterized in that, the selecting of the target image block that meets the preset mask requirement from the plurality of image blocks for masking to obtain the mask image block set comprises: obtaining the mask image block set according to the following formula: ; wherein, is the set of mask image blocks, is the number of image blocks the multispectral remote sensing image is divided into, is the mask ratio.

7. The remote sensing large model pre-training method according to any one of claims 1 to 6, characterized in that, the updating of the parameters of the remote sensing large model according to the mask image block set comprises: calculating the mean absolute error and the standardized mean absolute error corresponding to each mask image block in the mask image block set; updating parameters of the remote sensing large model according to the average absolute error and the normalized average absolute error.

8. A remote sensing large model pre-training apparatus, characterized in that, comprising: a division module, a first calculation module, a second calculation module, a third calculation module, a mask module, a parameter updating module, and a detection module; the division module is configured to divide a received multi-spectral remote sensing image into a plurality of image blocks; the first calculation module is configured to calculate, for each of the image blocks, a remote sensing index for representing environmental features of the image block; the second calculation module is configured to calculate, according to the remote sensing index, a semantic saliency index for representing a semantic information richness of the image block; the third calculation module is configured to determine a current training round of a remote sensing large model, and calculate a mask final score of the image block according to the current training round, a preset total training round of the remote sensing large model, and the semantic saliency index; the mask module is configured to select, from the plurality of image blocks, a target image block whose mask final score meets a preset mask requirement, and obtain a mask image block set; the parameter updating module is configured to update parameters of the remote sensing large model according to the mask image block set; the detection module is configured to detect whether the current training round is less than the total training round, and in a case where the detection module detects that the current training round is less than the total training round, increase the current training round, and the third calculation module calculates a new mask final score of the image block again according to the increased current training round, the mask module determines a new mask image block set again according to the new mask final score, and the parameter updating module updates the parameters of the remote sensing large model again, until the detection module detects that a new current training round is equal to the total training round; The remote sensing indexes include a normalized difference vegetation index, a normalized difference water index, and a normalized building index; each of the image blocks includes pixels, and P is an integer greater than 1. the calculation of the semantic saliency index according to the remote sensing index comprises: For each of the image blocks, respectively calculate a first mean and a first standard deviation of the normalized difference vegetation index, a second mean and a second standard deviation of the normalized difference water index, and a third mean and a third standard deviation of the normalized built-up index for the pixel points in the image block. calculating the semantic saliency index according to the first mean value, the first standard deviation, the second mean value, the second standard deviation, the third mean value, and the third standard deviation. 9.An electronic device comprising a processor and a memory, wherein, The memory is configured to store instructions, and the processor is configured to invoke the instructions in the memory, so that the electronic device executes the remote sensing large model pre-training method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, which, when executed on an electronic device, cause the electronic device to execute the remote sensing large model pre-training method according to any one of claims 1 to 7.

Images