Remote sensing basic model pre-training method and device based on context enhancement

Abstract

The embodiment of the invention discloses a remote sensing basic model pre-training method and device based on context enhancement. A specific embodiment of the method comprises the following steps: acquiring a ground remote sensing image set and a remote sensing basic model; performing image block division on each ground remote sensing image to generate an image block vector sequence; performing mask processing on each image block vector sequence to generate a mask image block vector sequence and an original pixel value; inputting each mask image block vector sequence into the remote sensing basic model to obtain a reconstructed pixel value set; inputting each image block vector sequence into the remote sensing basic model to obtain a prediction pixel value set; the total loss value of the remote sensing basic model is determined, and the total loss value is the sum of the reconstruction loss value, the prediction loss value and the consistency loss value; and according to the total loss value, performing back propagation parameter updating on the remote sensing basic model to obtain a pre-trained remote sensing basic model. According to the embodiment, accurate remote sensing features can be extracted, and waste of computing resources is reduced.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of remote sensing image perception, and specifically to a remote sensing basic model pre-training method and device based on context enhancement. Background Art

[0002] With the advancement of remote sensing technology and aerospace technology, it is becoming easier to collect earth observation remote sensing data through remote sensing, and the scale of satellite image data is also increasing. At the same time, self-supervised learning performs well in processing large amounts of unlabeled data, and shows better performance than supervised learning in tasks such as classification, segmentation, and detection. At present, when pre-training the remote sensing basic model, the method usually adopted is: the contrastive learning pre-training method based on self-supervised learning, and the representation learning of remote sensing images through a large number of data enhancement technologies to obtain the pre-trained remote sensing basic model.

[0003] However, when the above method is used to pre-train the remote sensing basic model, the following technical problems often occur: First, contrastive learning methods usually need to organize pre-training data in time and geographic dimensions, often need to build and store a large number of negative samples, and compare positive and negative samples, resulting in the need for large memory and computing resources during training. In addition, when dealing with complex remote sensing image feature extraction tasks, contrastive learning-based models often show poor generalization ability and are difficult to effectively extract accurate remote sensing features, thus consuming a lot of computing resources. Second, when pre-training remote sensing models using self-supervised learning methods based on mask image modeling, due to the uneven distribution of target density in remote sensing images, a high mask rate will lead to a loss of contextual information, which will affect the model's overall semantic understanding of the image, reduce the performance of the pre-trained remote sensing basic model, and further reduce the accuracy of remote sensing image classification, segmentation or detection tasks.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0005] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0006] Some embodiments of the present disclosure propose a remote sensing basic model pre-training method and device based on context enhancement to solve one or more of the technical problems mentioned in the above background technology section.

[0007] In a first aspect, some embodiments of the present disclosure provide a remote sensing basic model pre-training method based on context enhancement, the method comprising: obtaining a ground remote sensing image set and a remote sensing basic model; performing image block division on each ground remote sensing image in the ground remote sensing image set to generate an image block vector sequence, and obtaining an image block vector sequence set; performing masking processing on each image block vector sequence in the image block vector sequence set to generate a masked image block vector sequence and an original pixel value, and obtaining a masked image block vector sequence set and an original pixel value set; inputting each masked image block vector sequence in the masked image block vector sequence set into the remote sensing basic model to obtain a reconstructed pixel value set; inputting each image block vector sequence in the image block vector sequence set into the remote sensing basic model to obtain a predicted pixel value set; determining the total loss value of the remote sensing basic model based on the reconstructed pixel value set, the predicted pixel value set and the original pixel value set, wherein the total loss value is the sum of the reconstruction loss value, the prediction loss value and the consistency loss value; performing back-propagation parameter update on the remote sensing basic model based on the total loss value to obtain a pre-trained remote sensing basic model.

[0008] In a second aspect, some embodiments of the present disclosure provide a remote sensing basic model pre-training device based on context enhancement, the device comprising: an acquisition unit, configured to acquire a ground remote sensing image set and a remote sensing basic model; an image block division unit, configured to perform image block division on each ground remote sensing image in the ground remote sensing image set to generate an image block vector sequence, and obtain an image block vector sequence set; a mask processing unit, configured to perform mask processing on each image block vector sequence in the image block vector sequence set to generate a mask image block vector sequence and an original pixel value, and obtain a mask image block vector sequence set and an original pixel value set; a pixel reconstruction unit, configured to reconstruct the mask image block vector Each mask image block vector sequence in the sequence set is input into the above remote sensing basic model to obtain a reconstructed pixel value set; the prediction pixel unit is configured to input each image block vector sequence in the above image block vector sequence set into the above remote sensing basic model to obtain a predicted pixel value set; the determination unit is configured to determine the total loss value of the above remote sensing basic model based on the above reconstructed pixel value set, the above predicted pixel value set and the above original pixel value set, wherein the above total loss value is the sum of the reconstruction loss value, the prediction loss value and the consistency loss value; the back propagation parameter updating unit is configured to perform back propagation parameter update on the above remote sensing basic model according to the above total loss value to obtain a pre-trained remote sensing basic model.

[0009] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0010] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.

[0011] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the remote sensing basic model pre-training method based on context enhancement of some embodiments of the present disclosure, the waste of computing resources can be reduced. Specifically, the reason for the waste of computing resources is that the contrastive learning method usually needs to organize pre-training data in time and geographical dimensions, often needs to build and store a large number of negative samples, and compare positive and negative samples, resulting in the need for large memory and computing resources during training. In addition, when processing complex remote sensing image feature extraction tasks, the model based on contrastive learning often exhibits poor generalization ability and is difficult to effectively extract accurate remote sensing features. Based on this, the remote sensing basic model pre-training method based on context enhancement of some embodiments of the present disclosure first obtains a ground remote sensing image set and a remote sensing basic model. Secondly, each ground remote sensing image in the above ground remote sensing image set is divided into image blocks to generate an image block vector sequence to obtain an image block vector sequence set. Then, each image block vector sequence in the above image block vector sequence set is masked to generate a mask image block vector sequence and an original pixel value, and a mask image block vector sequence set and an original pixel value set are obtained. In order to solve the problem of context information loss caused by uneven target density, the original image block is used as a reconstruction template instead of directly deleting the masked image block. Then, each mask image block vector sequence in the mask image block vector sequence set is input into the remote sensing basic model to obtain a reconstructed pixel value set. Through the mask reconstruction method, the model's ability to extract image details is enhanced, thereby improving the generalization ability of the model in subsequent tasks. Then, each image block vector sequence in the image block vector sequence set is input into the remote sensing basic model to obtain a predicted pixel value set. A context enhancement generation branch with shared parameters is introduced to promote semantic reasoning by providing meaningful context information during the reconstruction process. Afterwards, the total loss value of the remote sensing basic model is determined based on the reconstructed pixel value set, the predicted pixel value set and the original pixel value set. Among them, the total loss value is the sum of the reconstruction loss value, the prediction loss value and the consistency loss value. By minimizing the gap between the original pixel value and the reconstructed value of the mask block, the model is guided to learn more context information. Finally, according to the total loss value, the back-propagation parameters of the remote sensing basic model are updated to obtain a pre-trained remote sensing basic model. This avoids the consumption of memory and computing resources caused by organizing positive and negative samples and performing comparative learning of positive and negative samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0013] Figure 1 is a flowchart of some embodiments of a remote sensing basic model pre-training method based on context enhancement according to the present disclosure; Figure 2 It is a framework diagram of some embodiments of the remote sensing basic model pre-training method based on context enhancement according to the present disclosure; Figure 3 is a reconstruction schematic diagram of some embodiments of the remote sensing basic model pre-training method based on context enhancement according to the present disclosure; Figure 4 is a building segmentation result diagram according to some embodiments of the remote sensing basic model pre-training method based on context enhancement disclosed in the present invention; Figure 5 It is a structural schematic diagram of some embodiments of the remote sensing basic model pre-training device based on context enhancement according to the present disclosure; Figure 6 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0014] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0015] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0016] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0017] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0018] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0019] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0020] Figure 1 The flowchart 100 of some embodiments of the remote sensing basic model pre-training method based on context enhancement according to the present disclosure is shown. The remote sensing basic model pre-training method based on context enhancement comprises the following steps: Step 101, obtaining a ground remote sensing image set and a remote sensing basic model.

[0021] In some embodiments, the execution subject of the context-enhanced remote sensing basic model pre-training method can obtain a ground remote sensing image set and a remote sensing basic model. Among them, the ground remote sensing image set can be obtained through a public data set. Here, the above-mentioned remote sensing basic model can be a feature extractor commonly used in various visual tasks, which is not specifically limited here. The above-mentioned visual tasks may include but are not limited to: semantic segmentation, target detection, ground object classification, land cover classification, etc.

[0022] As an example, the public datasets may include but are not limited to at least one of the following: WorldView-3 dataset, SpaceNet building segmentation dataset, DOTA (Dataset for Object Detection in Aerial Images), etc. The feature extractors may include but are not limited to at least one of the following: Swin Transformer (window transformer model), ResNet (residual network), EfficientNet (efficient network), etc.

[0023] Step 102 , performing image block division on each ground remote sensing image in the ground remote sensing image set to generate an image block vector sequence, thereby obtaining an image block vector sequence set.

[0024] In some embodiments, the execution entity may divide each ground remote sensing image in the ground remote sensing image set into image blocks to generate an image block vector sequence, thereby obtaining an image block vector sequence set.

[0025] In some optional implementations of some embodiments, the execution subject divides each ground remote sensing image in the ground remote sensing image set into image blocks to generate an image block vector sequence to obtain the image block vector sequence set, which may include the following steps: In the first step, the following image block division steps are performed on each ground remote sensing image in the above ground remote sensing image set to generate an image block vector sequence and obtain an image block vector sequence set: Sub-step 1: Divide the ground remote sensing image into multiple non-overlapping image blocks to obtain an image block sequence. The image blocks are square images with the same width and height. One ground remote sensing image corresponds to one image block sequence.

[0026] In practice, given a ground remote sensing image ,in, x represents Ground remote sensing images, represents the set of real numbers, H and W Represents the height and width of the image respectively, C represents the number of channels in the image. The size of the image block obtained by division can be P×P, Above P Can be used to represent the height and width of an image block.

[0027] Sub-step 2: Flatten each image block in the above image block sequence into an image block vector to obtain an image block vector sequence, wherein the image block vector can be obtained by determining the pixel value in the image block as the value of the corresponding position in the image block vector in a row-first or column-first order.

[0028] In practice, we can get the image block vector sequence ,in, Represents the sequence of image block vectors. N represents the number of image blocks, where .

[0029] Step 103 , performing mask processing on each image block vector sequence in the image block vector sequence set to generate a mask image block vector sequence and an original pixel value, thereby obtaining a mask image block vector sequence set and an original pixel value set.

[0030] In some embodiments, the execution entity may perform mask processing on each image block vector sequence in the image block vector sequence set to generate a mask image block vector sequence and an original pixel value, thereby obtaining a mask image block vector sequence set and an original pixel value set.

[0031] In some optional implementations of some embodiments, the execution subject performs mask processing on each image block vector sequence in the image block vector sequence set to generate a mask image block vector sequence and an original pixel value, and obtains the mask image block vector sequence set and the original pixel value set, which may include the following steps: In the first step, the following random masking operation is performed on each image block vector sequence in the above image block vector sequence set to obtain a masked image block vector sequence set and an original pixel value set: Sub-step 1: According to a preset masking rate, each image block vector sequence number corresponding to the image block vector sequence is determined as the image block vector sequence number to be masked, and a set of image block vector sequence numbers to be masked is obtained. The masking rate is the ratio between the number of masked graphic blocks and the number of image blocks. Here, a random algorithm can be set to determine the sequence number corresponding to the image block vector in the image block vector sequence that meets the masking rate as the image block vector sequence number to be masked, and a set of image block vector sequence numbers to be masked is obtained.

[0032] In practice, assuming the mask rate is Q, the above-mentioned set of vector numbers of the image blocks to be masked can have The vector number of the image block to be masked.

[0033] Sub-step 2, converting the image block vectors in the above-mentioned image block vector sequence corresponding to each image block vector sequence number to be masked in the above-mentioned image block vector sequence number set into preset mask vectors to obtain a mask image block vector sequence. The above-mentioned mask vector can be a vector of the same size as the image block vector. The values ​​in the above-mentioned mask vector can all be 1. The above-mentioned mask image block vector sequence contains unconverted image block vectors and mask vectors, wherein the number of mask vectors can be the product of the number of image blocks and the mask rate.

[0034] In practice, the sequence of mask image patch vectors can be expressed as .

[0035] Sub-step three, determining the value of the image block vector in the above-mentioned image block vector sequence corresponding to each image block vector sequence number to be masked in the above-mentioned image block vector sequence number set as the original pixel value. Among them, one image block vector sequence corresponds to one original pixel value. The above-mentioned original pixel value is a matrix composed of image block vectors corresponding to each image block vector sequence number to be masked.

[0036] Step 104: input each mask image block vector sequence in the mask image block vector sequence set into the remote sensing basic model to obtain a reconstructed pixel value set.

[0037] In some embodiments, the execution entity may input each mask image block vector sequence in the mask image block vector sequence set into the remote sensing basic model to obtain a reconstructed pixel value set.

[0038] In some optional implementations of some embodiments, the execution subject inputs each mask image block vector sequence in the mask image block vector sequence set into the remote sensing basic model to obtain a reconstructed pixel value set, which may include the following steps: In the first step, a block embedding operation is performed on each mask image block vector sequence in the above mask image block vector sequence set to generate mask image features, thereby obtaining a mask image feature set. Wherein, a block embedding operation can be performed on each mask image block vector sequence in the above mask image block vector sequence set to generate mask image features, thereby obtaining a mask image feature set, through a preset block embedding algorithm. The above block embedding algorithm can be a linear mapping operation or a convolution operation.

[0039] In practice, when the block embedding algorithm is a convolution operation, the size of the convolution kernel can be 4, the convolution step can be 4, the input dimension is the number of channels, and the output dimension can be 128. Here, the number of channels is C =3.

[0040] In the second step, according to the encoder, each mask image feature in the mask image feature set is subjected to implicit feature extraction to generate mask image implicit features, thereby obtaining the mask image implicit feature set. The remote sensing basic model includes an encoder and a decoder.

[0041] In practice, the mask image features after block embedding are used as the input of the encoder, and the encoder extracts implicit representations containing global and local information based on the partially visible image blocks of the input. The above encoder can be SwinTransformer, which maps the mask image features of the image blocks to high-dimensional feature representations through a hierarchical self-attention mechanism.

[0042] In the third step, according to the above decoder, feature decoding is performed on each implicit feature of the mask image in the implicit feature set of the mask image to generate a reconstructed pixel value, and a reconstructed pixel value set is obtained. Among them, the above decoder can be a multi-layer perceptron or other lightweight network. The above decoder can be a convolution layer with a convolution kernel size of 1. The above decoder is used to generate a reconstructed pixel value according to the implicit features of the mask image. Here, one implicit feature of the mask image corresponds to one reconstructed pixel value. The reconstructed pixel value is a matrix obtained by reconstructing the mask image block vector sequence, and the values ​​in the matrix are the pixel values ​​corresponding to each mask vector after reconstruction.

[0043] In practice, a reconstruction schematic diagram can be obtained according to the mask image block vector sequence and the corresponding reconstructed pixel value. The reconstruction schematic diagram obtained according to the embodiment of the present disclosure is as follows: Figure 3 shown. Figure 3 Each column from left to right is the original image and four 2-tuples. Each 2-tuple contains a mask image and a reconstruction diagram. The mask rates of the four 2-tuples from left to right are 70%, 75%, 80%, and 85%, respectively. Figure 3 It can be seen that the embodiments of the present disclosure can restore mask patches with high quality, especially in the case of high masking rates. Figure 3It can be seen that the embodiments of the present disclosure can provide robust features for remote sensing image analysis.

[0044] Step 105: input each image block vector sequence in the image block vector sequence set into the remote sensing basic model to obtain a predicted pixel value set.

[0045] In some embodiments, the execution entity may input each image block vector sequence in the image block vector sequence set into the remote sensing basic model to obtain a predicted pixel value set.

[0046] In some optional implementations of some embodiments, the execution subject inputs each image block vector sequence in the image block vector sequence set into the remote sensing basic model to obtain a predicted pixel value set, which may include the following steps: In the first step, according to the context enhancement branch encoder, implicit features are extracted from each image block vector sequence in the image block vector sequence set to generate image implicit features and obtain an image implicit feature set. The context enhancement branch encoder has the same structure as the encoder and shares parameters.

[0047] In the second step, according to the context enhancement branch decoder, feature decoding is performed on each image implicit feature in the image implicit feature set to generate a predicted pixel value, thereby obtaining a predicted pixel value set. The context enhancement branch decoder has the same structure as the decoder and shares parameters.

[0048] In practice, the above step 105 can be used as a context enhancement generation branch to receive the image block vector sequence and extract the image implicit features with rich context information through the encoder with shared parameters. Then, the above image implicit features are input into the same decoder to generate predicted pixel values. The process of context enhancement generation branch is shown in the following formula: .

[0049] in, Represents the predicted pixel value. Represents the first image block vector sequence i image patch vectors. is an indicator function, when the image block vector When masked, the function value is 1, otherwise it is 0. The symbol ⊙ represents the element-by-element product operation. f Indicates the encoder. represents the decoder. The context enhancement generation branch uses As a template, it promotes the imitation learning of the mask image reconstruction branch in terms of feature extraction and semantic reasoning by providing meaningful contextual information.

[0050] Step 106, determining the total loss value of the remote sensing basic model according to the reconstructed pixel value set, the predicted pixel value set and the original pixel value set.

[0051] In some embodiments, the execution entity may determine the total loss value of the remote sensing basic model based on the reconstructed pixel value set, the predicted pixel value set and the original pixel value set, wherein the total loss value is the sum of the reconstruction loss value, the prediction loss value and the consistency loss value.

[0052] In some optional implementations of some embodiments, the execution subject determines the total loss value of the remote sensing basic model according to the reconstructed pixel value set, the predicted pixel value set and the original pixel value set, which may include the following steps: The first step is to determine the reconstruction loss value between each reconstructed pixel value in the above reconstructed pixel value set and each original pixel value in the above original pixel value set according to a preset reconstruction loss function. The above reconstruction loss function is shown in the following formula: .

[0053] in, Represents the reconstruction loss value, and one reconstructed pixel value set corresponds to one reconstruction loss value. Represents the set of reconstructed pixel values. Represents a raw set of pixel values. Represents the number of mask pixels, where the number of mask pixels is the total number of elements in the mask vector corresponding to each mask image block vector sequence in the mask image block vector sequence set. It is used to calculate and The distance function of the similarity between them. Among them, the above reconstruction loss function uses the mean absolute error (L1 Loss) to calculate the reconstruction loss value.

[0054] In practice, the reconstruction learning objective is achieved by minimizing the distance between the original pixel values ​​and the reconstructed pixel values ​​of the mask patch.

[0055] The second step is to determine the prediction loss value between each predicted pixel value in the predicted pixel value set and each original pixel value in the original pixel value set according to a preset prediction loss function. The prediction loss function is shown in the following formula: .

[0056] in, Represents the prediction loss value. Represents a set of predicted pixel values.

[0057] The third step is to determine the consistency loss value between each reconstructed pixel value in the above-mentioned reconstructed pixel value set and each predicted pixel value in the above-mentioned predicted pixel value set according to a preset consistency loss function. The above-mentioned context consistency loss function is shown in the following formula: .

[0058] in, Represents the consistency loss value. Dist ( ) represents the similarity function, which is used to guide the model to better learn context information. Here, the mean absolute error (L1 Loss) can be used as Dist ( ) function.

[0059] In practice, the above consistency loss value can alleviate the pre-training convergence instability problem caused by missing information.

[0060] In the fourth step, the sum of the reconstruction loss value, the prediction loss value and the consistency loss value is determined as the total loss value. The total loss value can be determined by the following formula.

[0061] .

[0062] in, Represents the total loss value.

[0063] In practice, meaningful contextual information is provided to encourage the encoder to simulate feature extraction and learn semantic inference.

[0064] Step 107, based on the total loss value, back-propagation parameters of the remote sensing basic model are updated to obtain a pre-trained remote sensing basic model.

[0065] In some embodiments, the execution entity may perform back-propagation parameter update on the remote sensing basic model according to the total loss value to obtain a pre-trained remote sensing basic model.

[0066] In some optional implementations of some embodiments, the execution subject performs back propagation parameter update on the remote sensing basic model according to the total loss value to obtain a pre-trained remote sensing basic model, which may include the following steps: In the first step, in response to the total loss value being greater than a preset threshold, back-propagation parameters of the remote sensing basic model are updated according to the total loss value to obtain an initial remote sensing basic model and the pre-training step is performed again using the initial remote sensing basic model as the remote sensing basic model. The pre-training step includes steps 102 to 107.

[0067] In practice, the remote sensing basic model can be back-propagated and parameter updated by gradient descent, such as Figure 2The framework diagram is shown in Figure 1. Pre-training is performed through multi-task learning to learn feature representations with rich semantics and local information.

[0068] In the second step, in response to the total loss value being less than or equal to a preset threshold, the initial remote sensing basic model is determined as a pre-trained remote sensing basic model, and the pre-trained remote sensing basic model is stored.

[0069] Optionally, the above step 107 further includes the following steps: The first step is to obtain the remote sensing image to be measured, wherein the earth observation image taken by the satellite can be used as the remote sensing image to be measured.

[0070] In the second step, the pre-trained remote sensing basic model and the initial segmentation head network are fine-tuned to obtain a building segmentation remote sensing model. In which, a sample training data set is first obtained. The sample training data set includes a sample remote sensing image set and a corresponding sample building segmentation result atlas. The sample building segmentation result map includes at least one building segmentation area and a corresponding category label. Secondly, the pre-trained remote sensing basic model and the initial segmentation head network together constitute an initial building segmentation remote sensing model. The output of the pre-trained remote sensing basic model is the input of the initial segmentation head network. The initial segmentation head network can be a head network of Mask RCNN. Afterwards, the sample remote sensing image set is used as the input of the initial building segmentation remote sensing model, and the sample building segmentation result atlas is used as the output of the initial building segmentation remote sensing model to train and obtain the building segmentation remote sensing model.

[0071] In practice, the pre-trained remote sensing basic model obtained through the training of the above steps 101-107 can better extract the features of the remote sensing image to be tested and effectively learn the feature representation of high-level semantics and fine-grained details.

[0072] The third step is to input the above-mentioned remote sensing image to be tested into the above-mentioned building segmentation remote sensing model to obtain a building segmentation result map and display the above-mentioned building segmentation result map, wherein the above-mentioned building segmentation result map includes building segmentation areas and corresponding category labels.

[0073] In some optional implementations of some embodiments, the backbone network in Mask RCNN can be selected as the encoder, and the detection head in Mask RCNN can be selected as the initial segmentation head network. The building segmentation result obtained by training on the SpaceNet building segmentation dataset is shown in FIG. Figure 4 As shown. Among them, Figure 4The first column is the ground truth image (GT), and the second, third and fourth columns are respectively the building segmentation result images obtained by the supervised learning training method (Supervised Learning, Sup), the seasonal contrast unsupervised training method (Seasonal contrast, SeCo) and the pre-training method (Ours) proposed in the embodiment of the present disclosure. It can be seen that the present invention has a better effect on building segmentation, such as Figure 4 As shown in the yellow box in the figure.

[0074] In practice, target detection can also be performed on the remote sensing image to be tested. Specifically, the pre-trained remote sensing basic model and the initial target detection head network are first fine-tuned to obtain a target detection remote sensing model. Afterwards, the remote sensing image to be tested is input into the target detection remote sensing model to obtain a target detection result map. The target detection result map contains entity annotation boxes and corresponding entity categories. Finally, the target detection result map can be transmitted to a local display device or a remote display terminal for display.

[0075] The above optional steps and step 106-step 107 and their related contents are an inventive point of an embodiment of the present disclosure, which solves the above technical problem 2 "reducing the accuracy of classification, segmentation or detection tasks of remote sensing images". The factors that lead to the above technical problems are often as follows: When the remote sensing model is pre-trained by the self-supervised learning method based on mask image modeling, due to the uneven distribution of target density in the remote sensing image, a high mask rate will lead to the loss of context information, which will affect the model's overall semantic understanding of the image. If the above factors are solved, the performance of the pre-trained remote sensing basic model can be improved. In order to achieve this effect, first, the reconstruction learning goal is achieved by minimizing the distance between the original pixel value and the reconstructed value of the mask block. Then, the context information is learned by calculating the prediction loss between the predicted pixel value and the original pixel value. After that, the pre-trained remote sensing basic model is obtained by jointly training according to the reconstruction loss value, the prediction loss value and the consistency loss value. Secondly, the above pre-trained remote sensing basic model and the multi-task detection head can be fine-tuned and trained by multi-task learning to obtain a multi-task remote sensing model. Thus, feature representations with rich semantics and local information can be learned. Finally, the multi-task remote sensing model can be used to perform visual processing tasks such as building segmentation or target detection on the remote sensing image to be tested, thereby improving the accuracy of classification, segmentation or detection tasks on remote sensing images.

[0076] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the remote sensing basic model pre-training method based on context enhancement of some embodiments of the present disclosure, the waste of computing resources can be reduced. Specifically, the reason for the waste of computing resources is that the contrastive learning method usually needs to organize pre-training data in time and geographical dimensions, often needs to build and store a large number of negative samples, and compare positive and negative samples, resulting in the need for large memory and computing resources during training. In addition, when processing complex remote sensing image feature extraction tasks, the model based on contrastive learning often exhibits poor generalization ability and is difficult to effectively extract accurate remote sensing features. Based on this, the remote sensing basic model pre-training method based on context enhancement of some embodiments of the present disclosure first obtains a ground remote sensing image set and a remote sensing basic model. Secondly, each ground remote sensing image in the above ground remote sensing image set is divided into image blocks to generate an image block vector sequence to obtain an image block vector sequence set. Then, each image block vector sequence in the above image block vector sequence set is masked to generate a mask image block vector sequence and an original pixel value, and a mask image block vector sequence set and an original pixel value set are obtained. In order to solve the problem of context information loss caused by uneven target density, the original image block is used as a reconstruction template instead of directly deleting the masked image block. Then, each mask image block vector sequence in the mask image block vector sequence set is input into the remote sensing basic model to obtain a reconstructed pixel value set. Through the mask reconstruction method, the model's ability to extract image details is enhanced, thereby improving the generalization ability of the model in subsequent tasks. Then, each image block vector sequence in the image block vector sequence set is input into the remote sensing basic model to obtain a predicted pixel value set. A context enhancement generation branch with shared parameters is introduced to promote semantic reasoning by providing meaningful context information during the reconstruction process. Afterwards, the total loss value of the remote sensing basic model is determined based on the reconstructed pixel value set, the predicted pixel value set and the original pixel value set. Among them, the total loss value is the sum of the reconstruction loss value, the prediction loss value and the consistency loss value. By minimizing the gap between the original pixel value and the reconstructed value of the mask block, the model is guided to learn more context information. Finally, according to the total loss value, the back-propagation parameters of the remote sensing basic model are updated to obtain a pre-trained remote sensing basic model. This avoids the consumption of memory and computing resources caused by organizing positive and negative samples and comparing positive and negative samples.

[0077] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a remote sensing basic model pre-training device based on context enhancement, and these device embodiments are similar to Figure 1 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0078] like Figure 5 As shown, some embodiments of the remote sensing basic model pre-training device 500 based on context enhancement include: an acquisition unit 501, an image block division unit 502, a mask processing unit 503, a reconstructed pixel unit 504, a predicted pixel unit 505, a determination unit 506, and a back propagation parameter updating unit 507. Among them, the acquisition unit 501 is configured to acquire a ground remote sensing image set and a remote sensing basic model; the image block division unit 502 is configured to perform image block division on each ground remote sensing image in the above-mentioned ground remote sensing image set to generate an image block vector sequence, and obtain an image block vector sequence set; the mask processing unit 503 is configured to perform mask processing on each image block vector sequence in the above-mentioned image block vector sequence set to generate a mask image block vector sequence and an original pixel value, and obtain a mask image block vector sequence set and an original pixel value set; the reconstructed pixel unit 504 is configured to input each mask image block vector sequence in the above-mentioned mask image block vector sequence set into the above-mentioned A remote sensing basic model is provided to obtain a reconstructed pixel value set; a prediction pixel unit 505 is configured to input each image block vector sequence in the image block vector sequence set into the remote sensing basic model to obtain a predicted pixel value set; a determination unit 506 is configured to determine a total loss value of the remote sensing basic model according to the reconstructed pixel value set, the predicted pixel value set and the original pixel value set, wherein the total loss value is the sum of the reconstruction loss value, the prediction loss value and the consistency loss value; a back propagation parameter updating unit 507 is configured to perform back propagation parameter update on the remote sensing basic model according to the total loss value to obtain a pre-trained remote sensing basic model.

[0079] It is understood that the units described in the device 500 are similar to those described in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 500 and the units included therein, and will not be described in detail here.

[0080] Reference below Figure 6 , which shows a structural schematic diagram of an electronic device (eg, a computing device) 600 suitable for implementing some embodiments of the present disclosure. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0081] like Figure 6As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory 602 or a program loaded from a storage device 608 to a random access memory 603. Various programs and data required for the operation of the electronic device 600 are also stored in the random access memory 603. The processing device 601, the read-only memory 602, and the random access memory 603 are connected to each other via a bus 604. An input / output interface 605 is also connected to the bus 604.

[0082] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 6 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0083] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a read-only memory 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0084] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0085] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0086] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains a ground remote sensing image set and a remote sensing basic model; performs image block division on each ground remote sensing image in the ground remote sensing image set to generate an image block vector sequence, thereby obtaining an image block vector sequence set; performs mask processing on each image block vector sequence in the image block vector sequence set to generate a mask image block vector sequence and an original pixel value, thereby obtaining a mask image block vector sequence set and an original pixel value set; inputs each mask image block vector sequence in the mask image block vector sequence set into the remote sensing basic model to obtain a reconstructed pixel value set; inputs each image block vector sequence in the image block vector sequence set into the remote sensing basic model to obtain a predicted pixel value set; determines a total loss value of the remote sensing basic model based on the reconstructed pixel value set, the predicted pixel value set and the original pixel value set, wherein the total loss value is the sum of the reconstruction loss value, the predicted loss value and the consistency loss value; and performs back-propagation parameter update on the remote sensing basic model based on the total loss value to obtain a pre-trained remote sensing basic model.

[0087] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0088] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0089] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be arranged in a processor, for example, may be described as: a processor including an acquisition unit, an image block division unit, a mask processing unit, a reconstruction pixel unit, a prediction pixel unit, a determination unit, and a back propagation parameter update unit. Among them, the names of these units do not constitute a limitation on the units themselves in some cases. For example, the acquisition unit may also be described as a "unit for acquiring a ground remote sensing image set and a remote sensing basic model."

[0090] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0091] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A remote sensing basic model pre-training method based on context enhancement, comprising: Obtain ground remote sensing image sets and remote sensing basic models; Performing image block division on each ground remote sensing image in the ground remote sensing image set to generate an image block vector sequence, thereby obtaining an image block vector sequence set; Performing mask processing on each image block vector sequence in the image block vector sequence set to generate a mask image block vector sequence and an original pixel value, thereby obtaining a mask image block vector sequence set and an original pixel value set; Inputting each mask image block vector sequence in the mask image block vector sequence set into the remote sensing basic model to obtain a reconstructed pixel value set; Inputting each image block vector sequence in the image block vector sequence set into the remote sensing basic model to obtain a predicted pixel value set; Determine a total loss value of the remote sensing basic model according to the reconstructed pixel value set, the predicted pixel value set and the original pixel value set, wherein the total loss value is the sum of the reconstruction loss value, the prediction loss value and the consistency loss value; According to the total loss value, back-propagation parameters of the remote sensing basic model are updated to obtain a pre-trained remote sensing basic model.

2. The method according to claim 1, wherein: The step of dividing each ground remote sensing image in the ground remote sensing image set into image blocks to generate an image block vector sequence to obtain an image block vector sequence set comprises: The following image block division steps are performed on each ground remote sensing image in the ground remote sensing image set to generate an image block vector sequence, thereby obtaining an image block vector sequence set: Dividing the ground remote sensing image into a plurality of non-overlapping image blocks to obtain an image block sequence, wherein the image blocks are square images with the same width and height; Each image block in the image block sequence is flattened into an image block vector to obtain an image block vector sequence.

3. The method according to claim 1, wherein: The step of performing mask processing on each image block vector sequence in the image block vector sequence set to generate a mask image block vector sequence and an original pixel value, and obtaining the mask image block vector sequence set and the original pixel value set, comprises: The following random masking operation is performed on each image block vector sequence in the image block vector sequence set to obtain a masked image block vector sequence set and an original pixel value set: According to a preset masking rate, each corresponding image block vector sequence number in the image block vector sequence is determined as the image block vector sequence number to be masked, so as to obtain a set of image block vector sequence numbers to be masked; Converting the image block vectors in the image block vector sequence corresponding to each image block vector sequence number to be masked in the image block vector sequence number set to a preset mask vector to obtain a mask image block vector sequence; The value of the image block vector in the image block vector sequence corresponding to each image block vector sequence number to be masked in the image block vector sequence number set to be masked is determined as the original pixel value.

4. The method according to claim 1, wherein: The step of inputting each mask image block vector sequence in the mask image block vector sequence set into the remote sensing basic model to obtain a reconstructed pixel value set, and Inputting each image block vector sequence in the image block vector sequence set into the remote sensing basic model to obtain a predicted pixel value set, including: Performing a block embedding operation on each mask image block vector sequence in the mask image block vector sequence set to generate a mask image feature, thereby obtaining a mask image feature set; According to the encoder, performing implicit feature extraction on each mask image feature in the mask image feature set to generate a mask image implicit feature, thereby obtaining a mask image implicit feature set, wherein the remote sensing basic model includes an encoder and a decoder; According to the decoder, feature decoding is performed on each mask image implicit feature in the mask image implicit feature set to generate a reconstructed pixel value, thereby obtaining a reconstructed pixel value set; According to the context enhancement branch encoder, performing implicit feature extraction on each image block vector sequence in the image block vector sequence set to generate an image implicit feature, thereby obtaining an image implicit feature set, wherein the context enhancement branch encoder has the same structure as the encoder and shares parameters; According to the context enhancement branch decoder, feature decoding is performed on each image implicit feature in the image implicit feature set to generate a predicted pixel value, thereby obtaining a predicted pixel value set, wherein the context enhancement branch decoder has the same structure as the decoder and shares parameters.

5. The method according to claim 1, wherein: The method further comprises: Acquire the remote sensing image to be measured; Fine-tune the pre-trained remote sensing basic model and the initial segmentation head network to obtain a building segmentation remote sensing model; The remote sensing image to be measured is input into the building segmentation remote sensing model to obtain a building segmentation result map and display the building segmentation result map, wherein the building segmentation result map includes building segmentation areas and corresponding category labels.

6. A remote sensing basic model pre-training device based on context enhancement, comprising: An acquisition unit configured to acquire a ground remote sensing image set and a remote sensing basic model; An image block division unit is configured to perform image block division on each ground remote sensing image in the ground remote sensing image set to generate an image block vector sequence, thereby obtaining an image block vector sequence set; a mask processing unit configured to perform mask processing on each image block vector sequence in the image block vector sequence set to generate a mask image block vector sequence and an original pixel value, and obtain a mask image block vector sequence set and an original pixel value set; A pixel reconstruction unit is configured to input each mask image block vector sequence in the mask image block vector sequence set into the remote sensing basic model to obtain a reconstructed pixel value set; A predicted pixel unit, configured to input each image block vector sequence in the image block vector sequence set into the remote sensing basic model to obtain a predicted pixel value set; A determination unit is configured to determine a total loss value of the remote sensing base model according to the reconstructed pixel value set, the predicted pixel value set and the original pixel value set, wherein the total loss value is the sum of the reconstruction loss value, the prediction loss value and the consistency loss value; The back propagation parameter updating unit is configured to perform back propagation parameter updating on the remote sensing basic model according to the total loss value to obtain a pre-trained remote sensing basic model.

7. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images