A visual model training method, a remote sensing image segmentation method and related devices

Abstract

The application discloses a visual model training method, a remote sensing image segmentation method and related devices, and relates to the technical field of computers. Specifically, the application first performs model pre-training on a visual model on a pre-training data set. The pre-trained visual model has good representation capability for remote sensing images. Then, model parameters of the pre-trained visual model are configured as initial parameters of a basic model, and the pre-configured basic model is trained on data sets corresponding to related tasks and irrelevant tasks of a target task, so as to obtain a pre-trained basic model. Finally, the pre-trained basic model is trained on a data set corresponding to the target task. Since the pre-trained basic model has general representation capability, the pre-trained basic model can be optimized by using a small amount of training data corresponding to the target task, so that the processing capability of the model for the target task is improved, and the target task is reduced to depend on the training data of the target task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a visual model training method, a remote sensing image segmentation method and related devices. BACKGROUND

[0002] Remote sensing image refers to an image or photo of the earth surface obtained through remote sensing technology, which is usually taken or recorded by a satellite, an airplane, a drone or other remote sensing platforms. At present, with the continuous improvement of remote sensing infrastructure and the vigorous development of artificial intelligence (AI) technology, remote sensing image analysis based on AI technology such as deep learning is gradually widely applied in crop analysis, forestry analysis, fishery analysis, city analysis and other fields, and plays an important role and value in agriculture, forestry, fishery, hydrology and other industries.

[0003] However, due to the low resolution of remote sensing images, manual field surveying and mapping are often needed to realize the labeling task of remote sensing images, so that the configuration cost of remote sensing image training data is very high. Moreover, it takes a long time to accumulate enough training data, and the lack of data quantity of training data limits the function of deep learning-based remote sensing analysis scheme, and cannot realize the remote sensing image recognition task with high accuracy and high generalization ability.

[0004] Therefore, how to reduce the dependence of a specified type of remote sensing analysis scheme on a specified type of remote sensing image training data has become a technical problem to be solved by those skilled in the art. SUMMARY

[0005] In view of the above problems, the present application provides a visual model training method, a remote sensing image segmentation method and related devices to realize the purpose of reducing the dependence of a specified type of remote sensing analysis scheme on a specified type of training data.

[0006] The specific scheme is as follows:

[0007] The first aspect of the present application provides a visual model training method, comprising:

[0008] obtaining a pre-training dataset, a meta-training first dataset and a meta-training second dataset; wherein the pre-training dataset, the meta-training first dataset and the meta-training second dataset each comprise a plurality of pairs of remote sensing images labeled with corresponding labels; a pair of remote sensing images in the pre-training dataset is composed of two enhanced remote sensing images, and the two enhanced remote sensing images are obtained by performing different data enhancement processing on the same original remote sensing image; a pair of remote sensing images in the meta-training first dataset and the meta-training second dataset is composed of two remote sensing images with the same label, or two remote sensing images with different labels, or two enhanced remote sensing images corresponding to the same original remote sensing image; the label of each pair of remote sensing images is composed of the labels of the two remote sensing images constituting the pair of remote sensing images, the labels in the pre-training dataset and the meta-training first dataset correspond to at least one remote sensing image recognition task, and any task in the at least one remote sensing image recognition task is a related task or an unrelated task of a target task; the labels in the meta-training second dataset correspond to the target task;

[0009] model pre-training is performed on the pre-configured visual model on the pre-training dataset, and model parameters are updated according to an overall loss, to obtain a pre-trained visual model; the overall loss is a weighted loss of a contrastive loss and a cross-entropy loss;

[0010] model parameters of the pre-trained visual model are configured as initial parameters of a base model, to obtain a pre-configured base model, and the pre-configured base model comprises an attention module;

[0011] model training is performed on the pre-configured base model on the meta-training first dataset, and model parameters are updated according to the overall loss, to obtain a pre-trained base model;

[0012] model training is performed on the pre-trained base model on the meta-training second dataset, and model parameters are updated according to the overall loss, to obtain a target visual model for implementing the target task.

[0013] The second aspect of the application provides a visual model training device, comprising:

[0014] The training data acquisition unit is configured to acquire a pre-training data set, a meta-training first data set, and a meta-training second data set; wherein the pre-training data set, the meta-training first data set, and the meta-training second data set each include a plurality of pairs of remote sensing images labeled with corresponding labels; a pair of remote sensing images in the pre-training data set is composed of two enhanced remote sensing images, and the two enhanced remote sensing images are obtained by performing different data enhancement processing on the same original remote sensing image; a pair of remote sensing images in the meta-training first data set and the meta-training second data set is composed of two remote sensing images with the same label, or composed of two remote sensing images with different labels, or composed of two enhanced remote sensing images corresponding to the same original remote sensing image; the label of each pair of remote sensing images is composed of the labels of the two remote sensing images constituting the pair of remote sensing images; the labels in the pre-training data set and the meta-training first data set correspond to at least one remote sensing image recognition task, and any one of the at least one remote sensing image recognition task is a related task or an unrelated task of a target task; the labels in the meta-training second data set correspond to the target task;

[0015] The pre-training unit is configured to perform model pre-training on a preconfigured visual model based on the pre-training data set, and update model parameters based on an overall loss to obtain a pre-trained visual model; the overall loss is a weighted loss of a contrastive loss and a cross-entropy loss.

[0016] The meta-training unit is configured to configure the model parameters of the pre-trained visual model as initial parameters to a base model to obtain a preconfigured base model, wherein the preconfigured base model includes an attention module; perform model training on the preconfigured base model based on the meta-training first data set, and update model parameters based on the overall loss to obtain a pre-trained base model; perform model training on the pre-trained base model based on the meta-training second data set, and update model parameters based on the overall loss to obtain a target visual model for implementing the target task.

[0017] The third aspect of the present application provides a remote sensing image segmentation method, comprising:

[0018] Acquiring a remote sensing image to be segmented;

[0019] Calling the target visual model to process the remote sensing image to be segmented to obtain a segmentation result of a specified segmentation object in the remote sensing image to be segmented;

[0020] The target visual model is trained according to the visual model training method in the first aspect, and the target task is a remote sensing image segmentation task of the specified segmentation object, the related task of the target task includes a remote sensing image segmentation task of another segmentation object, and the irrelevant task of the target task includes at least one of a classification task, a positioning task and a detection task of a remote sensing image.

[0021] The fourth aspect of the present application provides an image processing device, comprising at least one processor and a memory connected to the processor, wherein:

[0022] The memory is used to store a computer program;

[0023] The processor is used to execute the computer program to implement the visual model training method in the first aspect or the remote sensing image segmentation method in the third aspect.

[0024] The fifth aspect of the present application provides a storage medium, the storage medium carries one or more computer programs, when the one or more computer programs are executed by an electronic device, the electronic device can implement the visual model training method in the first aspect or the remote sensing image segmentation method in the third aspect.

[0025] Through the above technical solution, the visual model is first pre-trained on the pre-training data set. Since the similar image pairs in the pre-training data set include two enhanced remote sensing images obtained by enhancing the same original remote sensing image, and are labeled with corresponding labels, that is, the self-supervised, supervised and contrast learning methods are used in the pre-training stage, the pre-trained visual model has good representation capability for remote sensing images. On this basis, the model parameters of the pre-trained visual model are configured as initial parameters of the basic model, and the pre-configured basic model is trained on the meta-training first data set corresponding to the related task and the irrelevant task of the target task. Since the remote sensing image pairs in the meta-training first data set meet the conditions of being obtained by enhancing the same original remote sensing image, or meeting the conditions of having the same label, or meeting the conditions of having different labels, based on this, self-supervised data enhancement, same label data grouping and different label grouping are performed in the meta-training stage, that is, the model training in multiple modes is adopted, and the pre-trained basic model with general representation capability is obtained. Finally, the pre-trained basic model is retrained on the meta-training second data set corresponding to the target task. Since the pre-trained basic model has general representation capability, the pre-trained basic model can be optimized by using a small amount of training data corresponding to the target task, that is, the processing capability of the model for the specified target task can be improved.

[0026] Based on this, by means of two-stage training based on the combination of supervised, self-supervised and contrastive learning, the application can increase the feature distance between classes and reduce the feature distance within classes by relying on less sample data, thereby training a target visual model with high accuracy based on a small amount of sample data, reducing the dependence of a specified type of remote sensing analysis scheme on a specified type of remote sensing image training data, thereby reducing the cost and cycle of data procurement, providing a basis for implementing a target task based on a small amount of target task sample data, and quickly supporting new function expansion. BRIEF DESCRIPTION OF DRAWINGS

[0027] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments, and are not intended to limit the application. Moreover, the same reference numerals are used throughout the various drawings to designate identical parts. In the drawings:

[0028] Figure 1 a flowchart of a visual model training method provided by the application;

[0029] Figure 2 a full flowchart of visual model training is illustrated;

[0030] Figure 3 a flowchart of pre-training is illustrated;

[0031] Figure 4 a flowchart of meta-training is illustrated;

[0032] Figure 5 a structural diagram of a visual model training device provided by the application;

[0033] Figure 6 a structural diagram of an image processing device provided by the application. DETAILED DESCRIPTION

[0034] The embodiments of the application are described below with reference to the accompanying drawings. The terms used in the embodiment part of the application are only used to explain the specific embodiments of the application, and are not intended to limit the application. Those skilled in the art can know that with the development of technology and the appearance of new scenarios, the technical solutions provided by the embodiments of the application are also applicable to similar technical problems.

[0035] Figure 1 is a flowchart of a visual model training method according to an embodiment of the application, Figure 2 a full flowchart of visual model training is illustrated. In combination with Figure 1 and Figure 2 as shown, the method can include the following steps:

[0036] Step S101, obtaining a pre-training dataset, a meta-training first dataset and a meta-training second dataset.

[0037] The pre-training dataset, the meta-training first dataset and the meta-training second dataset each include a plurality of pairs of remote sensing images labeled with corresponding labels.

[0038] The sample data in different datasets have different characteristics. Specifically, the pair of remote sensing images in the pre-training dataset is composed of two enhanced remote sensing images obtained by different data enhancement processing on the same original remote sensing image; the pair of remote sensing images in the meta-training first dataset and the meta-training second dataset is composed of two remote sensing images with the same label, or two remote sensing images with different labels, or two enhanced remote sensing images corresponding to the same original remote sensing image.

[0039] The label of each pair of remote sensing images in the pre-training dataset, the meta-training first dataset and the meta-training second dataset is composed of the labels of the two remote sensing images constituting the pair of remote sensing images.

[0040] The sample labels in different datasets also have different characteristics. Specifically, the labels in the pre-training dataset and the meta-training first dataset correspond to at least one remote sensing image recognition task, and any one of the at least one remote sensing image recognition task is a related task or an unrelated task of a target task; the labels in the meta-training second dataset correspond to the target task. It should be noted that the visual model trained based on the training scheme provided in the present application is a visual model suitable for solving the target task; the target task and its related task and unrelated task are all computer vision tasks suitable for processing remote sensing images.

[0041] Step S102, model pre-training of a pre-configured visual model on the pre-training dataset, and updating of model parameters according to the overall loss to obtain a pre-trained visual model.

[0042] It should be noted that in the field of computer vision, pre-training can usually be performed on a large-scale image dataset (such as ImageNet) to learn the bottom features (such as edges, textures, etc.) and high-level features (such as objects, scenes, etc.) in images. The pre-trained model can serve as a starting point for other computer vision tasks, and can be fine-tuned to adapt to new tasks and datasets. Based on this, in the field of remote sensing image recognition, model pre-training can be performed based on remote sensing image sample data of non-target tasks.

[0043] For example, in combination with the above-mentioned pre-training dataset, the meta-training first dataset and the meta-training second dataset, the pre-training dataset includes a plurality of pairs of remote sensing images labeled with corresponding labels, and the labels correspond to at least one remote sensing image recognition task, and any one of the at least one remote sensing image recognition task is a related task or an unrelated task of a target task; the meta-training first dataset includes a plurality of pairs of remote sensing images labeled with corresponding labels, and the labels correspond to at least one remote sensing image recognition task, and any one of the at least one remote sensing image recognition task is a related task or an unrelated task of a target task; the meta-training second dataset includes a plurality of pairs of remote sensing images labeled with corresponding labels, and the labels correspond to the target task. Figure 2The pre-training stage schematic diagram shows that a set of input data processed by the visual model CNN can include two enhanced remote sensing images (e.g., enhanced remote sensing image s and enhanced remote sensing image t) obtained by the same original remote sensing image after different data enhancement processing (e.g., data enhancement s and data enhancement t); based on the label of the set of input data and the processing result of the visual model CNN on the set of input data, the contrastive loss and the cross-entropy loss corresponding to the set of input data can be obtained, and the overall loss is obtained by weighting the two, that is, the overall loss is the weighted loss of the contrastive loss and the cross-entropy loss; through multiple rounds of gradient iteration, the learning rate and the optimization strategy are adjusted to improve the representation understanding ability of the pre-trained visual model.

[0044] It should be noted that pre-training is used to train the model with a large amount of data, so that the model learns the general feature representation, and provides a basis for optimizing the performance of the model on specific tasks. Through pre-training, the model training task in the few-shot scenario or the model training task when the computing resources are limited can be solved to a certain extent.

[0045] Step S103, the model parameters of the pre-trained visual model are configured as initial parameters of the base model to obtain a pre-configured base model.

[0046] The pre-configured base model includes an attention module. That is, the base structure of the base model trained in the meta-training stage can be consistent with the base structure of the visual model trained in the pre-training stage, but the base model trained in the meta-training stage adds an attention mechanism to improve the relevance understanding ability of the base model; and the base structure of the base model uses the parameters of the visual model in the pre-training stage for initial parameter configuration.

[0047] Step S104, model training is performed on the pre-configured base model on the meta-training first data set, and the model parameters are updated according to the overall loss to obtain a pre-trained base model.

[0048] For example, in combination with Figure 2As shown in the meta-training stage schematic diagram, a set of input data processed by the base model CNN can be represented as remote sensing image 1 and remote sensing image 2, which can specifically include: two enhanced remote sensing images constituting a similar image pair, or two remote sensing images constituting a same-label image pair, or two remote sensing images constituting a different-label image pair. Among them, the same-label image pair is composed of two remote sensing images with the same label, and the different-label image pair is composed of two remote sensing images with different labels. Based on the label of the set of input data and the processing result of the visual model CNN on the set of input data, the contrast loss and the cross-entropy loss corresponding to the set of input data can be obtained, and the overall loss is obtained after weighting the two, that is, the overall loss is the weighted loss of the contrast loss and the cross-entropy loss; by minimizing the loss function, the model parameters are optimized to improve the model accuracy.

[0049] It should be noted that in the meta-training stage, the training samples can be batched and trained to improve the evaluation indicators of the algorithm on the training task, and the parameter settings, network structure, etc. are adjusted. Based on this, the base model in the meta-training stage can absorb experience from past tasks and dynamically select bias on new tasks, thereby achieving the purpose of improving learning efficiency and generalization ability.

[0050] Step S105, model training is performed on the pre-trained base model on the meta-training second data set, and the model parameters are updated according to the overall loss, to obtain a target visual model for implementing the target task.

[0051] Based on the above, the visual model is first pre-trained on the pre-training data set. Since the similar image pairs in the pre-training data set include two enhanced remote sensing images obtained from the same original remote sensing image, and are labeled with corresponding labels, that is, the self-supervised, supervised and contrast learning methods are used in the pre-training stage, the pre-trained visual model has good representation ability for remote sensing images. On this basis, the model parameters of the pre-trained visual model are configured as initial parameters of the base model, and the pre-configured base model is trained on the meta-training first data set corresponding to the related task and the unrelated task of the target task. Since the remote sensing image pairs in the meta-training first data set meet the conditions of being enhanced from the same original remote sensing image, having the same label, or having different labels, based on this, self-supervised data enhancement, same-label data grouping and different-label grouping are performed in the meta-training stage, that is, model training in multiple modes is adopted, and a pre-trained base model with general representation ability is obtained. Finally, the pre-trained base model is retrained on the meta-training second data set corresponding to the target task. Since the pre-trained base model has general representation ability, the pre-trained base model can be fine-tuned using a small amount of training data corresponding to the target task, thereby improving the processing ability of the model for the specified target task.

[0052] Based on this, by means of two-stage training based on combining supervised, self-supervised and contrastive learning, the application can increase the feature distance between classes and reduce the feature distance within classes by relying on less sample data, thereby enabling the target visual model to be trained based on a small amount of sample data to have a high accuracy, reducing the dependence of the specified type of remote sensing analysis scheme on the specified type of remote sensing image training data, thereby reducing the cost and period of data procurement, providing a basis for implementing target tasks based on a small amount of target task sample data, and quickly supporting new function expansion.

[0053] In one or more embodiments provided by the application, the configuration process of the pre-training data set can include:

[0054] Step S201, obtaining a plurality of original remote sensing images.

[0055] Step S202, performing at least two ways of data enhancement processing on the obtained original remote sensing images to obtain at least two enhanced remote sensing images corresponding to the obtained original remote sensing images.

[0056] Step S203, grouping two enhanced remote sensing images corresponding to the same original remote sensing image and different from each other into a group to obtain a plurality of similar image pairs (which can also be referred to as similar pair pairs), and configuring labels for the obtained similar image pairs.

[0057] Step S204, generating the pre-training data set.

[0058] The pre-training data set can include a plurality of similar image pairs labeled with corresponding labels.

[0059] On the basis of the above, in combination with Figure 3 The pre-training process schematic diagram of the example, the above-mentioned step S102, model pre-training of the pre-configured visual model on the pre-training data set, and updating of the model parameters according to the overall loss, can include:

[0060] Step S301, obtaining a pre-configured visual model.

[0061] Step S302, extracting similar image pairs from the pre-training data set.

[0062] Step S303, calling two visual models sharing the model structure to process the two enhanced remote sensing images in the extracted similar image pairs, respectively.

[0063] Step S304: Based on the processing results of the extracted similar image pairs by the two visual models, calculate the first cross-entropy loss value and the first contrast loss value, and then add the first cross-entropy loss value and the first contrast loss value by weight to obtain the first overall loss value.

[0064] For example, in combination Figure 3 The pre-training process shown, where a visual model processes a remote sensing image, can include: a convolutional layer convolves the input remote sensing image to obtain a first feature map; a pooling layer pools the first feature map to obtain a second feature map; a fully connected layer (FC) processes the second feature map to obtain a first feature vector; and a dropout layer processes the first feature vector to obtain a second feature vector. Based on this, the processing results of the two visual models on the extracted similar image pairs can include the second feature maps and second feature vectors of the two remote sensing images (e.g., enhanced remote sensing image s and enhanced remote sensing image t) in the extracted similar image pair; a cross-entropy loss is calculated based on the two second feature maps and two labels (e.g., the label of enhanced remote sensing image s and the label of enhanced remote sensing image t) to obtain the aforementioned first cross-entropy loss; a contrast loss is calculated based on the two second feature vectors to obtain the aforementioned first contrast loss; and then the calculated first cross-entropy loss and first contrast loss are weighted to obtain the overall loss, i.e., the aforementioned first overall loss, providing a basis for updating model parameters.

[0065] Step S305: Update the model parameters of the visual model based on the first overall loss value, and return to the step of extracting similar image pairs from the pre-training dataset until the preset pre-training stopping condition is met, and obtain the pre-trained visual model.

[0066] The above scheme reduces model parameters by setting pooling and dropout layers, laying the foundation for improving model generalization. The weighted calculation result of the two losses is used as the overall loss, and through multiple rounds of gradient iteration, the learning rate and optimization strategy are adjusted, which improves the representational understanding ability of the pre-trained visual model to a certain extent. Based on the above, supervised and self-supervised comparative learning methods are used to train the visual model in the pre-training stage, resulting in a pre-trained visual model with good initial representation ability for remote sensing images.

[0067] Further, in the pre-training stage, a self-supervised contrastive learning method can also be adopted, that is, the model is trained on a large amount of unlabeled data, that is, unsupervised learning is performed by using the data itself as a supervision signal to train the model to learn the internal structure and representation of the data. Since self-supervised learning does not require labeled labels, it can learn from a large amount of unlabeled sample data, reducing the data labeling cost, and through the generation task, the model can obtain multiple samples from a single data, improving the utilization efficiency of the data. Based on the self-supervised contrastive learning method, the model can learn a general and bottom-level feature representation, and based on this, the learned knowledge can be transferred to a specific downstream task to solve the actual machine learning task by using a small amount of labeled data. That is, the model parameters can be updated based on the contrastive loss to iteratively obtain the pre-trained visual model.

[0068] In one or more embodiments provided in the present application, the configuration process of each meta-training data set in the meta-training first data set and the meta-training second data set can include:

[0069] Step S401, a plurality of original remote sensing images labeled with corresponding labels are obtained.

[0070] The labels of the obtained original remote sensing images match the tasks corresponding to the meta-training data set. That is, the labels of the original remote sensing images obtained when configuring the meta-training second data set correspond to the target task, and the labels of the original remote sensing images obtained when configuring the meta-training first data set correspond to the related task and the irrelevant task of the target task.

[0071] Step S402, after removing the labels labeled on the obtained original remote sensing images, at least two ways of data enhancement processing are performed on the obtained original remote sensing images to obtain at least two enhanced remote sensing images corresponding to the obtained original remote sensing images, and labels are configured for the obtained similar image pairs.

[0072] Step S403, two original remote sensing images corresponding to the same task and labeled with different labels are grouped into a group to obtain a plurality of different-label image pairs.

[0073] Step S404, two original remote sensing images corresponding to the same task and labeled with the same label are grouped into a group to obtain a plurality of same-label image pairs.

[0074] Step S405, the meta-training data set is generated.

[0075] The meta-training data set includes a plurality of remote sensing image pairs labeled with corresponding labels, and the remote sensing image pairs include similar image pairs, different-label image pairs, and same-label image pairs.

[0076] On the basis of the above, the step S104, the model training of the pre-configured base model on the meta-training first data set, and the updating of the model parameters according to the overall loss can include:

[0077] Step S501, extracting a pair of remote sensing images from the meta-training first data set.

[0078] Step S502, calling two pre-configured base models of a shared model structure to process two remote sensing images in the extracted pair of remote sensing images respectively.

[0079] Step S503, calculating a second cross-entropy loss value and a second contrastive loss value according to the processing results of the two pre-configured base models on the extracted pair of remote sensing images, and performing weighted addition on the second cross-entropy loss value and the second contrastive loss value to obtain a second overall loss value.

[0080] For example, in combination with the meta-training process shown in Figure 4 , the process of calling a base model to process a remote sensing image can include: a convolution layer performs convolution processing on the input remote sensing image to obtain a third feature map, a pooling layer (pool) performs pooling processing on the third feature map to obtain a fourth feature map, a fully connected layer (FC) processes the fourth feature map to obtain a third feature vector, and an attention mechanism layer (Atten) processes the third feature vector to obtain a fourth feature vector. On this basis, the processing results of the two visual models on the extracted pair of remote sensing images can include the fourth feature map and the fourth feature vector of each of the two remote sensing images (such as remote sensing image 1 and remote sensing image 2) in the extracted pair of remote sensing images, the calculation of the cross-entropy loss based on the two fourth feature maps and two labels (such as the label of remote sensing image 1 and the label of remote sensing image 2) to obtain the second cross-entropy loss described above, the calculation of the contrastive loss based on the two fourth feature vectors to obtain the second contrastive loss described above, and then the weighted overall loss obtained from the calculated second cross-entropy loss and second contrastive loss, that is, the second overall loss described above, provides a basis for updating the model parameters.

[0081] Step S504, updating the model parameters of the pre-configured base model based on the second overall loss value, and returning to step S501 of extracting a pair of remote sensing images from the meta-training first data set until a preset meta-training first stop condition is met, to obtain the pre-trained base model.

[0082] In the meta-training stage, a more abundant contrast learning is utilized, i.e., a positive and negative sample pair is constructed, similar samples are taken as positive samples, and dissimilar samples are taken as negative samples; on this basis, the similarity between the positive sample pairs is maximized, and the similarity between the negative sample pairs is minimized, so that the model can learn the internal structure and relationship of the data, thereby generating better data representation.

[0083] It should be noted that the meta-training stage (meta-learning) can train the model on the task to enable the model to solve other tasks. For example, in the meta-training stage, the model structure, hyperparameters, initialization parameters, etc. are first learned in the training samples of task 1, then the test samples of task 1 are evaluated, and each group of test samples can obtain a loss to represent the algorithm quality; then the same process is performed on task 2, and finally the final loss is obtained by averaging the loss of each task.

[0084] On the basis of the above, the step S105 of training the pre-trained base model on the meta-training second data set and updating the model parameters according to the overall loss can include:

[0085] Step S601, extracting a pair of remote sensing images from the meta-training second data set.

[0086] Step S602, calling two pre-trained base models sharing the model structure to process the two remote sensing images in the extracted remote sensing image pair respectively.

[0087] Step S603, calculating a third cross-entropy loss value and a third contrast loss value according to the processing results of the two pre-trained base models on the extracted remote sensing image pair, and performing weighted addition on the third cross-entropy loss value and the third contrast loss value to obtain a third overall loss value.

[0088] Step S604, updating the model parameters of the pre-trained base model based on the third overall loss value, and returning to step S601 of extracting a pair of remote sensing images from the meta-training second data set until a preset meta-training second stop condition is met, to obtain the target visual model.

[0089] Based on the above, in the meta-training stage, the model is first trained on the existing multiple remote sensing data sets, the base model learns multiple remote sensing recognition tasks, and has a general representation understanding ability, and then in the target task, the training data corresponding to the target task is used for tuning to improve the special ability of the visual model. Based on this, on the new task, only a small amount of samples are needed to make the model have good understanding ability.

[0090] Based on the above, the embodiment of the present application provides a two-stage model training scheme based on contrast learning. By integrating contrast learning into the pre-training stage and the meta-training stage, the image recognition problem of small samples is solved to some extent. Specifically, the above scheme uses self-supervised, supervised and contrast learning to train the model in the pre-training stage. In the meta-training stage, the initial parameters are configured based on the model parameters of the pre-training stage, and self-supervised data augmentation, same label data grouping and different label grouping are used in multiple mode training. Through the above scheme, less data can be relied on, the feature distance between classes is increased, and the feature distance within the class is reduced, so as to achieve an accuracy close to that of the model trained based on a large amount of training data. That is, the present application can reduce the dependence of deep learning training on training data, and provide a basis for reducing data cost and quickly supporting new tasks.

[0091] The visual model training device provided by the embodiment of the present application is described below. The visual model training device described below can be correspondingly referred to the visual model training method described above.

[0092] Figure 5 A structural schematic diagram of a visual model training device disclosed by the embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the device can include: Figure 5

[0093] The training data acquisition unit 11 is configured to acquire a pre-training data set, a meta-training first data set and a meta-training second data set. The pre-training data set, the meta-training first data set and the meta-training second data set each include a plurality of remote sensing image pairs labeled with corresponding labels. The remote sensing image pair in the pre-training data set is composed of two enhanced remote sensing images obtained by different data enhancement processing of the same original remote sensing image. The remote sensing image pair in the meta-training first data set and the meta-training second data set is composed of two remote sensing images with the same label, or composed of two remote sensing images with different labels, or composed of two enhanced remote sensing images corresponding to the same original remote sensing image. The label of each remote sensing image pair is composed of the labels of the two remote sensing images constituting the remote sensing image pair. The labels in the pre-training data set and the meta-training first data set correspond to at least one remote sensing image recognition task. Any task in the at least one remote sensing image recognition task is a related task or an unrelated task of the target task. The labels in the meta-training second data set correspond to the target task.

[0094] The pre-training unit 12 is configured to perform model pre-training on a pre-configured visual model on the pre-training data set, and update the model parameters according to the overall loss to obtain a pre-trained visual model. The overall loss is a weighted loss of the contrast loss and the cross-entropy loss.

[0095] ​The meta-training unit 13 is configured to configure model parameters of the pre-trained visual model as initial parameters of the base model to obtain a pre-configured base model, the pre-configured base model comprising an attention module; perform model training on the pre-configured base model on the meta-training first data set, and update model parameters according to the overall loss to obtain a pre-trained base model; perform model training on the pre-trained base model on the meta-training second data set, and update model parameters according to the overall loss to obtain a target visual model for implementing the target task.

[0096] In one or more embodiments provided in the present application, the apparatus can further comprise a training data configuration unit configured to configure the pre-training data set.

[0097] Based on the above, the process in which the training data configuration unit configures the pre-training data set can comprise:

[0098] Obtain a plurality of original remote sensing images;

[0099] Perform data enhancement processing on the obtained original remote sensing images in at least two ways to obtain at least two enhanced remote sensing images corresponding to the obtained original remote sensing images;

[0100] Group two enhanced remote sensing images corresponding to the same original remote sensing image and different from each other into a group to obtain a plurality of similar image pairs, and configure labels for the obtained similar image pairs;

[0101] Generate the pre-training data set, the pre-training data set comprising a plurality of similar image pairs labeled with corresponding labels.

[0102] In one or more embodiments provided in the present application, the process in which the pre-training unit 12 performs model pre-training on the pre-configured visual model on the pre-training data set and updates model parameters according to the overall loss can comprise:

[0103] Obtain a pre-configured visual model;

[0104] Extract similar image pairs from the pre-training data set;

[0105] Call two visual models sharing the model structure to process two enhanced remote sensing images in the extracted similar image pairs, respectively;

[0106] According to the processing results of the two visual models on the extracted similar image pairs, calculate a first cross-entropy loss value and a first contrast loss value, and add the first cross-entropy loss value and the first contrast loss value by weighting to obtain a first overall loss value;

[0107] update the model parameters of the visual model based on the first overall loss value, and return to performing the step of extracting similar image pairs from the pre-training data set until a preset pre-training stop condition is met, to obtain the pre-trained visual model.

[0108] In one or more embodiments provided in the present application, the apparatus can further include a training data configuration unit configured to configure the meta-training first data set and the meta-training second data set.

[0109] On the basis described above, the process of configuring each meta-training data set in the meta-training first data set and the meta-training second data set by the training data configuration unit can include:

[0110] Obtaining a plurality of original remote sensing images labeled with corresponding labels; the labels of the obtained original remote sensing images match the tasks corresponding to the meta-training data set;

[0111] After removing the labels labeled on the obtained original remote sensing images, performing at least two ways of data enhancement processing on the obtained original remote sensing images to obtain at least two enhanced remote sensing images corresponding to the obtained original remote sensing images, and configuring labels for the obtained similar image pairs;

[0112] Grouping two original remote sensing images corresponding to the same task and labeled with different labels into a group to obtain a plurality of different-label image pairs;

[0113] Grouping two original remote sensing images corresponding to the same task and labeled with the same label into a group to obtain a plurality of same-label image pairs;

[0114] Generating the meta-training data set, the meta-training data set including a plurality of remote sensing image pairs labeled with corresponding labels, the remote sensing image pairs including similar image pairs, different-label image pairs and same-label image pairs.

[0115] In one or more embodiments provided in the present application, the meta-training unit 13 can perform model training on the pre-configured base model on the meta-training first data set, and the process of updating the model parameters according to the overall loss can include:

[0116] Extracting remote sensing image pairs from the meta-training first data set;

[0117] Calling two pre-configured base models sharing the model structure to process two remote sensing images in the extracted remote sensing image pairs, respectively;

[0118] According to the processing results of the extracted remote sensing image pair by the two pre-configured base models, a second cross-entropy loss value and a second contrast loss value are calculated, and the second cross-entropy loss value and the second contrast loss value are weighted and added to obtain a second overall loss value;

[0119] The model parameters of the pre-configured base model are updated based on the second overall loss value, and the step of extracting a remote sensing image pair from the meta-training first data set is executed until a preset meta-training first stop condition is met, and the pre-trained base model is obtained.

[0120] In one or more embodiments provided in the present application, the meta-training unit 13 performs model training on the pre-trained base model on the meta-training second data set, and the process of updating the model parameters according to the overall loss can include:

[0121] extracting a remote sensing image pair from the meta-training second data set;

[0122] calling two pre-trained base models of the shared model structure to process two remote sensing images in the extracted remote sensing image pair respectively;

[0123] According to the processing results of the extracted remote sensing image pair by the two pre-trained base models, a third cross-entropy loss value and a third contrast loss value are calculated, and the third cross-entropy loss value and the third contrast loss value are weighted and added to obtain a third overall loss value;

[0124] The model parameters of the pre-trained base model are updated based on the third overall loss value, and the step of extracting a remote sensing image pair from the meta-training second data set is executed until a preset meta-training second stop condition is met, and the target visual model is obtained.

[0125] Optionally, the above target task can be any one of the classification task, the positioning task, the detection task and the segmentation task in the field of remote sensing image processing. The segmentation task is taken as an example for description.

[0126] The present application provides a remote sensing image segmentation method, which can include:

[0127] Step S701, acquiring a remote sensing image to be segmented;

[0128] Step S702, calling a target visual model to process the remote sensing image to be segmented to obtain a segmentation result of a specified segmentation object in the remote sensing image to be segmented.

[0129] The target visual model is trained according to any one of the visual model training methods, the target task is a remote sensing image segmentation task of the specified segmentation object, the related task of the target task includes a remote sensing image segmentation task of another segmentation object, and the irrelevant task of the target task includes at least one of a classification task, a positioning task and a detection task of a remote sensing image.

[0130] The remote sensing image segmentation device provided in the embodiments of the present application is described below, and the remote sensing image segmentation device described below can be correspondingly referred to the remote sensing image segmentation method described above.

[0131] The present application provides a remote sensing image segmentation device, which can include:

[0132] An image acquisition unit is configured to acquire a remote sensing image to be segmented.

[0133] An image segmentation unit is configured to call a target visual model to process the remote sensing image to be segmented, and obtain a segmentation result of a specified segmentation object in the remote sensing image to be segmented.

[0134] The target visual model is trained according to any one of the visual model training methods, the target task is a remote sensing image segmentation task of the specified segmentation object, the related task of the target task includes a remote sensing image segmentation task of another segmentation object, and the irrelevant task of the target task includes at least one of a classification task, a positioning task and a detection task of a remote sensing image.

[0135] The visual model training device or the remote sensing image segmentation device provided in the embodiments of the present application can be applied to an image processing device, such as a terminal with data processing capability, a computer, a server, etc. Optionally, Figure 6 A hardware structure block diagram of the image processing device is shown, and the hardware structure of the image processing device can include at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4. Figure 6

[0136] In the embodiments of the present application, the number of the processor 1, the communication interface 2, the memory 3 and the communication bus 4 is at least one, and the processor 1, the communication interface 2 and the memory 3 complete the communication among each other through the communication bus 4.

[0137] The processor 1 can be a central processing unit CPU, or a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application, etc.

[0138] ​The memory 3 can comprise a high-speed RAM memory and possibly also a non-volatile memory, such as at least one disk memory;

[0139] The memory is configured to store a computer program, and the processor is configured to execute the computer program, so that the image processing device is enabled to implement any of the visual model training method or the remote sensing image segmentation method described above.

[0140] The embodiments of the present application further provide a storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device is enabled to implement any of the visual model training method or the remote sensing image segmentation method provided by the embodiments of the present application.

[0141] The embodiments of the present application further provide a computer program product, which comprises computer readable instructions. When the computer readable instructions run on an electronic device, the electronic device is enabled to implement any of the visual model training method or the remote sensing image segmentation method provided by the embodiments of the present application.

[0142] Finally, it needs to be explained that, in this document, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or sequence between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0143] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The embodiments can be combined as needed, and the same and similar parts refer to each other.

[0144] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A visual model training method, characterized in that, The method comprises the following steps: obtaining a pre-training data set, a meta-training first data set and a meta-training second data set; wherein the pre-training data set, the meta-training first data set and the meta-training second data set each comprise a plurality of remote sensing image pairs labeled with corresponding labels; the remote sensing image pair in the pre-training data set is composed of two enhanced remote sensing images obtained by different data enhancement processing of the same original remote sensing image; the remote sensing image pair in the meta-training first data set and the meta-training second data set is composed of two remote sensing images with the same label, or two remote sensing images with different labels, or two enhanced remote sensing images corresponding to the same original remote sensing image; the label of each remote sensing image pair is composed of the labels of the two remote sensing images constituting the remote sensing image pair; the labels in the pre-training data set and the meta-training first data set correspond to at least one remote sensing image recognition task, and any task in the at least one remote sensing image recognition task is a related task or an unrelated task of a target task; the labels in the meta-training second data set correspond to the target task; model pre-training is performed on the pre-configured visual model based on the pre-training data set, and model parameter updating is performed based on the overall loss, to obtain a pre-trained visual model; the overall loss is a weighted loss of the contrast loss and the cross-entropy loss; the model parameters of the pre-trained visual model are configured as initial parameters of a basic model to obtain a pre-configured basic model, and the pre-configured basic model comprises an attention module; model training is performed on the pre-configured basic model based on the meta-training first data set, and model parameter updating is performed based on the overall loss, to obtain a pre-trained basic model; model training is performed on the pre-trained basic model based on the meta-training second data set, and model parameter updating is performed based on the overall loss, to obtain a target visual model for implementing the target task; the configuration process of the pre-training data set comprises the following steps: obtaining a plurality of original remote sensing images; performing data enhancement processing in at least two ways on the obtained original remote sensing images to obtain at least two enhanced remote sensing images corresponding to the obtained original remote sensing images; grouping two enhanced remote sensing images corresponding to the same original remote sensing image and different from each other to obtain a plurality of similar image pairs, and configuring labels for the obtained similar image pairs; generating the pre-training data set comprising a plurality of similar image pairs labeled with corresponding labels; model pre-training is performed on the pre-configured visual model based on the pre-training data set, and model parameter updating is performed based on the overall loss, which comprises the following steps: obtaining a pre-configured visual model; extracting similar image pairs from the pre-training data set; calling two visual models sharing the same model structure to process the two enhanced remote sensing images in the extracted similar image pairs, respectively; According to the processing results of the extracted similar image pairs by the two visual models, a first cross-entropy loss value and a first contrast loss value are calculated, and the first cross-entropy loss value and the first contrast loss value are weighted and added to obtain a first overall loss value; The model parameters of the visual model are updated based on the first overall loss value, and the step of extracting similar image pairs from the pre-training data set is returned to be executed until a pre-set pre-training stop condition is met, and the pre-trained visual model is obtained.

2. The visual model training method of claim 1, wherein, The configuration process of each meta-training first data set and meta-training second data set includes: Obtain a plurality of original remote sensing images labeled with corresponding labels; the labels of the obtained original remote sensing images match the tasks corresponding to the meta-training data set; After removing the labels labeled on the obtained original remote sensing images, at least two ways of data enhancement processing are performed on the obtained original remote sensing images to obtain at least two enhanced remote sensing images corresponding to the obtained original remote sensing images, and labels are configured for the obtained similar image pairs; Two original remote sensing images corresponding to the same task and labeled with different labels are grouped into a group to obtain a plurality of different label image pairs; Two original remote sensing images corresponding to the same task and labeled with the same label are grouped into a group to obtain a plurality of same label image pairs; The meta-training data set is generated, and the meta-training data set includes a plurality of remote sensing image pairs labeled with corresponding labels, and the remote sensing image pairs include similar image pairs, different label image pairs and same label image pairs.

3. The visual model training method of claim 2, wherein, Model training is performed on the pre-configured base model on the meta-training first data set, and the model parameters are updated according to the overall loss, including: Extracting remote sensing image pairs from the meta-training first data set; Two pre-configured base models sharing the model structure are called to process two remote sensing images in the extracted remote sensing image pairs respectively; According to the processing results of the extracted remote sensing image pairs by the two pre-configured base models, a second cross-entropy loss value and a second contrast loss value are calculated, and the second cross-entropy loss value and the second contrast loss value are weighted and added to obtain a second overall loss value; The model parameters of the pre-configured base model are updated based on the second overall loss value, and the step of extracting remote sensing image pairs from the meta-training first data set is returned to be executed until a pre-set meta-training first stop condition is met, and the pre-trained base model is obtained. 4.The visual model training method of claim 2, wherein, Model training is performed on the pre-trained base model on the meta-training second data set, and the model parameters are updated according to the overall loss, including: Extracting remote sensing image pairs from the meta-training second data set; Two pre-trained base models sharing the model structure are called to process two remote sensing images in the extracted remote sensing image pairs respectively; According to the processing results of the extracted remote sensing image pairs by the two pre-trained base models, a third cross-entropy loss value and a third contrast loss value are calculated, and the third cross-entropy loss value and the third contrast loss value are weighted and added to obtain a third overall loss value; The model parameters of the pre-trained base model are updated based on the third overall loss value, and the step of extracting remote sensing image pairs from the meta-training second data set is returned to be executed until a preset meta-training second stop condition is met, and the target visual model for implementing the target task is obtained.

5. A visual model training apparatus, characterized by comprising: Comprise: A training data acquisition unit is configured to acquire a pre-training data set, a meta-training first data set and a meta-training second data set; wherein the pre-training data set, the meta-training first data set and the meta-training second data set respectively comprise a plurality of remote sensing image pairs labeled with corresponding labels; the remote sensing image pairs in the pre-training data set are composed of two enhanced remote sensing images, and the two enhanced remote sensing images are obtained by different data enhancement processing on the same original remote sensing image; the remote sensing image pairs in the meta-training first data set and the meta-training second data set are composed of two remote sensing images with the same label, or composed of two remote sensing images with different labels, or composed of two enhanced remote sensing images corresponding to the same original remote sensing image; the label of each remote sensing image pair is composed of the labels of the two remote sensing images constituting the remote sensing image pair; the labels in the pre-training data set and the meta-training first data set correspond to at least one remote sensing image recognition task, and any task in the at least one remote sensing image recognition task is a related task or an unrelated task of the target task; the labels in the meta-training second data set correspond to the target task; A pre-training unit is configured to perform model pre-training on a pre-configured visual model on the pre-training data set, and update the model parameters according to the overall loss to obtain a pre-trained visual model; the overall loss is a weighted loss of the contrast loss and the cross-entropy loss; A meta-training unit is configured to configure the model parameters of the pre-trained visual model as initial parameters to a base model to obtain a pre-configured base model, and the pre-configured base model comprises an attention module; the pre-configured base model is trained on the meta-training first data set, and the model parameters are updated according to the overall loss to obtain a pre-trained base model; the pre-trained base model is trained on the meta-training second data set, and the model parameters are updated according to the overall loss to obtain a target visual model for implementing the target task; A training data configuration unit is configured to configure the pre-training data set; The process of configuring the pre-training data set by the training data configuration unit comprises: Obtaining a plurality of original remote sensing images; Performing at least two ways of data enhancement processing on the obtained original remote sensing images to obtain at least two enhanced remote sensing images corresponding to the obtained original remote sensing images; The two enhanced remote sensing images corresponding to the same original remote sensing image and different are grouped into a group to obtain a plurality of similar image pairs, and the obtained similar image pairs are labeled; The pre-training data set is generated, and the pre-training data set includes a plurality of similar image pairs labeled with corresponding labels; The pre-training unit pre-trains a pre-configured visual model on the pre-training data set, and the process of updating the model parameters according to the overall loss includes: obtaining a pre-configured visual model; extracting similar image pairs from the pre-training data set; calling two visual models sharing the model structure to process the two enhanced remote sensing images in the extracted similar image pairs respectively; According to the processing results of the two visual models on the extracted similar image pairs, a first cross-entropy loss value and a first contrast loss value are calculated, and the first cross-entropy loss value and the first contrast loss value are weighted and added to obtain a first overall loss value; update the model parameters of the visual model based on the first overall loss value, and return to the step of extracting similar image pairs from the pre-training data set until the pre-set pre-training stop condition is met, and obtain the pre-trained visual model.

6. A method for segmenting remote sensing images, characterized in that it comprises the steps of: The method comprises: obtaining a remote sensing image to be segmented; calling a target visual model to process the remote sensing image to be segmented to obtain a segmentation result of a specified segmentation object in the remote sensing image to be segmented; wherein the target visual model is trained according to the visual model training method of any one of claims 1-4, and the target task is a remote sensing image segmentation task of the specified segmentation object, the related task of the target task includes a remote sensing image segmentation task of other segmentation objects, and the irrelevant task of the target task includes at least one of a classification task, a positioning task and a detection task of a remote sensing image.

7. An image processing apparatus characterized by comprising: comprising at least one processor and a memory connected to the processor, wherein: the memory is used to store a computer program; the processor is used to execute the computer program to enable the image processing device to implement the visual model training method of any one of claims 1-4, or implement the remote sensing image segmentation method of claim 6.

8. A storage medium, characterized by The storage medium carries one or more computer programs, which can enable the electronic device to implement the visual model training method of any one of claims 1-4 or the remote sensing image segmentation method of claim 6 when the one or more computer programs are executed by the electronic device.

Images