Visual model training method, remote sensing image segmentation method and related devices

Abstract

The invention discloses a visual model training method, a remote sensing image segmentation method and a related device, and relates to the technical field of computers, specifically, model pre-training is performed on a visual model on a pre-training data set, and the pre-trained visual model has good characterization capability on a remote sensing image; then, using model parameters of the pre-trained visual model as initial parameters, configuring the initial parameters to the basic model, and performing model training on the pre-configured basic model on a data set corresponding to related tasks and irrelevant tasks of the target task to obtain a pre-trained basic model; and finally, training the pre-trained basic model on the data set corresponding to the target task, and since the pre-trained basic model has a universal representation capability, the pre-trained basic model is optimized by using a small amount of training data corresponding to the target task, so that the processing capability of the model on the target task can be improved, and the processing efficiency of the model on the target task is improved. And the dependence of the target task implementation on the target task training data is reduced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a visual model training method, a remote sensing image segmentation method and related devices. Background Art

[0002] Remote sensing images refer to images or photos of the earth's surface obtained through remote sensing technology, usually taken or recorded by satellites, aircraft, drones or other remote sensing platforms. At present, with the continuous improvement of remote sensing infrastructure and the vigorous development of artificial intelligence (AI) technology, remote sensing image analysis methods based on AI technologies such as deep learning have gradually been widely used in crop analysis, forestry analysis, fishery analysis, urban analysis and other fields, playing an important role and value in agriculture, forestry, fishery, hydrology and other industries.

[0003] However, due to the low resolution of remote sensing images, manual field surveying is often required to achieve the remote sensing image labeling task, which makes the configuration cost of remote sensing image training data very high; moreover, it takes a long time to accumulate enough training data, and the insufficient amount of training data leads to the limitation of the functions of remote sensing analysis solutions based on deep learning, and it is impossible to achieve remote sensing image recognition tasks with high accuracy and high generalization ability.

[0004] Therefore, how to reduce the dependence of a specified type of remote sensing analysis solution on a specified type of remote sensing image training data has become a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0005] In view of the above problems, the present application provides a visual model training method, a remote sensing image segmentation method and related devices to achieve the purpose of reducing the dependence of a specified type of remote sensing analysis scheme on a specified type of training data.

[0006] The specific plan is as follows:

[0007] The first aspect of the present application provides a visual model training method, comprising:

[0008] A pre-training data set, a first meta-training data set and a second meta-training data set are obtained; wherein the pre-training data set, the first meta-training data set and the second meta-training data set respectively include: a plurality of remote sensing image pairs annotated with corresponding labels; the remote sensing image pairs in the pre-training data set are composed of two enhanced remote sensing images, and the two enhanced remote sensing images are obtained by subjecting the same original remote sensing image to different data enhancement processes; the remote sensing image pairs in the first meta-training data set and the second meta-training data set are composed of two remote sensing images with the same label, or two remote sensing images with different labels, or two enhanced remote sensing images corresponding to the same original remote sensing image; the label of each remote sensing image pair is composed of the labels of the two remote sensing images constituting the remote sensing image pair, and the labels in the pre-training data set and the first meta-training data set correspond to at least one remote sensing image recognition task, and any one of the at least one remote sensing image recognition tasks is a related task or an unrelated task of the target task; the labels in the second meta-training data set correspond to the target task;

[0009] Pre-training a pre-configured visual model on the pre-training data set, and updating model parameters according to the overall loss to obtain a pre-trained visual model; the overall loss is a weighted loss of contrast loss and cross entropy loss;

[0010] Using the model parameters of the pre-trained visual model as initial parameters, and configuring them to the basic model to obtain a pre-configured basic model, wherein the pre-configured basic model includes an attention module;

[0011] Performing model training on the pre-configured basic model on the first meta-training data set, and updating model parameters according to the overall loss to obtain a pre-trained basic model;

[0012] The pre-trained basic model is trained on the meta-training second data set, and the model parameters are updated according to the overall loss to obtain a target visual model for achieving the target task.

[0013] A second aspect of the present application provides a visual model training device, comprising:

[0014] A training data acquisition unit is used to acquire a pre-training data set, a meta-training first data set and a meta-training second data set; wherein the pre-training data set, the meta-training first data set and the meta-training second data set respectively include: a plurality of remote sensing image pairs annotated with corresponding labels; the remote sensing image pairs in the pre-training data set are composed of two enhanced remote sensing images, and the two enhanced remote sensing images are obtained by subjecting the same original remote sensing image to different data enhancement processes; the remote sensing image pairs in the meta-training first data set and the meta-training second data set are composed of two remote sensing images with the same label, or two remote sensing images with different labels, or two enhanced remote sensing images corresponding to the same original remote sensing image; the label of each remote sensing image pair is composed of the labels of the two remote sensing images constituting the remote sensing image pair, and the labels in the pre-training data set and the meta-training first data set correspond to at least one remote sensing image recognition task, and any one of the at least one remote sensing image recognition tasks is a related task or an unrelated task of the target task; the labels in the meta-training second data set correspond to the target task;

[0015] A pre-training unit, used for performing model pre-training on a pre-configured visual model on the pre-training data set, and updating model parameters according to an overall loss to obtain a pre-trained visual model; the overall loss is a weighted loss of a contrast loss and a cross entropy loss;

[0016] A meta-training unit is used to configure the model parameters of the pre-trained visual model as initial parameters to the basic model to obtain a pre-configured basic model, wherein the pre-configured basic model includes an attention module; perform model training on the pre-configured basic model on the meta-training first data set, and update the model parameters according to the overall loss to obtain the pre-trained basic model; perform model training on the pre-trained basic model on the meta-training second data set, and update the model parameters according to the overall loss to obtain a target visual model for achieving the target task.

[0017] The third aspect of the present application provides a remote sensing image segmentation method, comprising:

[0018] Obtaining remote sensing images to be segmented;

[0019] Invoking the target visual model to process the remote sensing image to be segmented, and obtaining a segmentation result of a specified segmentation object in the remote sensing image to be segmented;

[0020] Among them, the target visual model is trained according to the visual model training method described in the first aspect above, and the target task is the remote sensing image segmentation task of the specified segmentation object, and the related tasks of the target task include: remote sensing image segmentation tasks of other segmentation objects; irrelevant tasks of the target task include at least one of the classification task, positioning task and detection task of the remote sensing image.

[0021] A fourth aspect of the present application provides an image processing device, comprising at least one processor and a memory connected to the processor, wherein:

[0022] The memory is used to store computer programs;

[0023] The processor is used to execute the computer program to implement the visual model training method of the first aspect, or to implement the remote sensing image segmentation method of the third aspect.

[0024] The fifth aspect of the present application provides a storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the visual model training method of the first aspect, or implement the remote sensing image segmentation method of the third aspect.

[0025] By means of the above technical scheme, the present application firstly pre-trains the visual model on the pre-training data set. Since the similar image pairs in the pre-training data set include two enhanced remote sensing images obtained by enhancing the same original remote sensing image and are annotated with corresponding labels, that is, self-supervision, supervised and contrastive learning methods are adopted in the pre-training stage, so that the pre-trained visual model has a good representation ability for the remote sensing images; on this basis, the model parameters of the pre-trained visual model are used as initial parameters and configured to the basic model, and the pre-configured basic model is trained on the meta-training first data set corresponding to the relevant tasks and irrelevant tasks of the target task. Since the meta-training first data set The remote sensing image pairs in the data set are either enhanced from the same original remote sensing image, or have the same label, or have different labels. Based on this, self-supervised data enhancement, same-label data grouping and different-label data grouping are performed in the meta-training stage, that is, multiple modes of model training are adopted to obtain a pre-trained basic model with general representation capabilities; finally, the pre-trained basic model is retrained on the second meta-training data set corresponding to the target task. Since the pre-trained basic model has general representation capabilities, the pre-trained basic model can be tuned using a small amount of training data corresponding to the target task, thereby improving the model's processing capabilities for the specified target task.

[0026] Based on this, through a two-stage training method based on a combination of supervised, self-supervised and contrastive learning, the present application can rely on less sample data to increase the feature distance between classes and reduce the feature distance within a class. In this way, a target visual model with a higher accuracy can be obtained based on a small amount of sample data training, reducing the dependence of a specified type of remote sensing analysis scheme on a specified type of remote sensing image training data, thereby reducing the cost and cycle of data procurement, and providing a basis for achieving target tasks based on a small amount of target task sample data, and quickly supporting the expansion of new functions. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0028] Figure 1 A flowchart of a visual model training method provided in this application;

[0029] Figure 2 This example shows the entire process of visual model training.

[0030] Figure 3 The schematic diagram of the pre-training process is illustrated;

[0031] Figure 4 An example of the meta-training process diagram is shown;

[0032] Figure 5 A schematic diagram of the structure of a visual model training device provided in this application;

[0033] Figure 6 A schematic diagram of the structure of an image processing device provided in this application. DETAILED DESCRIPTION

[0034] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation mode of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application. It is known to those skilled in the art that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0035] Figure 1 is a flow chart of a visual model training method according to an embodiment of the present application. Figure 2 This example shows the entire process of visual model training. Figure 1 and Figure 2 As shown, the method may include the following steps:

[0036] Step S101: Obtain a pre-training data set, a meta-training first data set, and a meta-training second data set.

[0037] The pre-training data set, the first meta-training data set and the second meta-training data set respectively include: a plurality of remote sensing image pairs annotated with corresponding labels.

[0038] The sample data in different data sets have different characteristics. Specifically, the remote sensing image pairs in the pre-training data set are composed of two enhanced remote sensing images, and the two enhanced remote sensing images are obtained by subjecting the same original remote sensing image to different data enhancement processes; the remote sensing image pairs in the first meta-training data set and the second meta-training data set are composed of two remote sensing images with the same label, or two remote sensing images with different labels, or two enhanced remote sensing images corresponding to the same original remote sensing image.

[0039] The label of each remote sensing image pair in the pre-training data set, the meta-training first data set, and the meta-training second data set is composed of labels of two remote sensing images constituting the remote sensing image pair.

[0040] The sample labels in different data sets also have different characteristics. Specifically, the labels in the pre-training data set and the meta-training first data set correspond to at least one remote sensing image recognition task, and any one of the at least one remote sensing image recognition task is a related task or an unrelated task of the target task; the labels in the meta-training second data set correspond to the target task. It should be noted that the visual model trained based on the training scheme provided in this application is a visual model suitable for solving the target task; the target task and its related tasks and unrelated tasks are all computer vision tasks suitable for processing remote sensing images.

[0041] Step S102: pre-training the pre-configured visual model on the pre-training data set, and updating the model parameters according to the overall loss to obtain the pre-trained visual model.

[0042] It should be noted that in the field of computer vision, pre-training can usually be performed on large-scale image datasets (such as ImageNet) to learn low-level features (such as edges, textures, etc.) and high-level features (such as objects, scenes, etc.) in images. Pre-trained models can be used as a starting point for other computer vision tasks and can be fine-tuned to adapt to new tasks and datasets. Based on this, in the field of remote sensing image recognition, model pre-training can be performed based on remote sensing image sample data of non-target tasks.

[0043] Exemplary, combined Figure 2As shown in the schematic diagram of the pre-training stage, a set of input data processed by the visual model CNN may include two enhanced remote sensing images (such as enhanced remote sensing image s and enhanced remote sensing image t) obtained by subjecting the same original remote sensing image to different data enhancement processing (such as data enhancement s and data enhancement t); based on the labels of a set of input data and the processing results of the visual model CNN on the set of input data, the contrast loss and cross entropy loss corresponding to the set of input data can be obtained, and the overall loss is obtained by weighting the two, that is, the overall loss is the weighted loss of the contrast loss and the cross entropy loss; in this way, after multiple rounds of gradient iterations, the learning rate and optimization strategy are adjusted to improve the representation and understanding ability of the pre-trained visual model.

[0044] It should be noted that pre-training uses a large amount of data to train the model so that the model can learn generalized feature representations, providing a basis for optimizing the model's performance on specific tasks. Pre-training can, to a certain extent, solve model training tasks in scenarios with few samples or when computing resources are limited.

[0045] Step S103: using the model parameters of the pre-trained visual model as initial parameters, and configuring them to the basic model to obtain a pre-configured basic model.

[0046] The preconfigured basic model includes an attention module. That is, the basic structure of the basic model trained in the meta-training phase can be consistent with the basic structure of the visual model trained in the pre-training phase, but the basic model trained in the meta-training phase adds an attention mechanism to enhance the relevance understanding ability of the basic model; and the basic structure of the basic model uses the parameters of the visual model in the pre-training phase for initial parameter configuration.

[0047] Step S104: performing model training on the pre-configured basic model on the meta-training first data set, and updating model parameters according to the overall loss to obtain a pre-trained basic model.

[0048] Exemplary, combined Figure 2As shown in the schematic diagram of the meta-training stage, a set of input data processed by the basic model CNN can be represented as remote sensing image 1 and remote sensing image 2, which can specifically include: two enhanced remote sensing images constituting a similar image pair, or two remote sensing images constituting a same-label image pair, or two remote sensing images constituting a different-label image pair. Among them, the same-label image pair is composed of two remote sensing images with the same label, and the different-label image pair is composed of two remote sensing images with different labels. Based on the label of a set of input data and the processing result of the visual model CNN on the set of input data, the contrast loss and cross entropy loss corresponding to the set of input data can be obtained, and the overall loss is obtained by weighting the two, that is, the overall loss is the weighted loss of the contrast loss and the cross entropy loss; by minimizing the loss function, the model parameters are optimized to improve the model accuracy.

[0049] It should be noted that in the meta-training phase, the training samples can be divided into batches and trained to improve the evaluation indicators of the algorithm on the training task, and adjust the parameter settings, network structure, etc. Based on this, the basic model in the meta-training phase can absorb experience from past tasks and dynamically select deviations on new tasks, thereby achieving the purpose of improving learning efficiency and generalization ability.

[0050] Step S105: performing model training on the pre-trained basic model on the meta-training second data set, and updating model parameters according to the overall loss to obtain a target visual model for achieving the target task.

[0051] Based on the above content, this embodiment first pre-trains the visual model on the pre-training data set. Since the similar image pairs in the pre-training data set include two enhanced remote sensing images obtained by enhancing the same original remote sensing image and are annotated with corresponding labels, that is, self-supervision, supervised and contrast learning methods are adopted in the pre-training stage, so that the pre-trained visual model has a good representation ability for the remote sensing image; on this basis, the model parameters of the pre-trained visual model are used as initial parameters and configured to the basic model, and the pre-configured basic model is trained on the meta-training first data set corresponding to the relevant tasks and irrelevant tasks of the target task. Since the meta-training first data set The remote sensing image pairs in the data set are either enhanced from the same original remote sensing image, or have the same label, or have different labels. Based on this, self-supervised data enhancement, same-label data grouping and different-label data grouping are performed in the meta-training stage, that is, multiple modes of model training are adopted to obtain a pre-trained basic model with general representation capabilities; finally, the pre-trained basic model is retrained on the second meta-training data set corresponding to the target task. Since the pre-trained basic model has general representation capabilities, the pre-trained basic model can be tuned using a small amount of training data corresponding to the target task, thereby improving the model's processing capabilities for the specified target task.

[0052] Based on this, through a two-stage training method based on a combination of supervised, self-supervised and contrastive learning, the present application can rely on less sample data to increase the feature distance between classes and reduce the feature distance within a class. In this way, a target visual model with a higher accuracy can be obtained based on a small amount of sample data training, reducing the dependence of a specified type of remote sensing analysis scheme on a specified type of remote sensing image training data, thereby reducing the cost and cycle of data procurement, and providing a basis for achieving target tasks based on a small amount of target task sample data, and quickly supporting the expansion of new functions.

[0053] In one or more embodiments provided in the present application, the configuration process of the pre-training data set may include:

[0054] Step S201: Acquire several original remote sensing images.

[0055] Step S202: performing at least two types of data enhancement processing on the acquired original remote sensing image to obtain at least two enhanced remote sensing images corresponding to the acquired original remote sensing image.

[0056] Step S203: grouping two different enhanced remote sensing images corresponding to the same original remote sensing image into one group to obtain a number of similar image pairs (also referred to as similar pairs), and configuring labels for the obtained similar image pairs.

[0057] Step S204: Generate the pre-training data set.

[0058] The pre-training data set may include a number of similar image pairs annotated with corresponding labels.

[0059] On the basis of the above, combined with Figure 3 The schematic diagram of the pre-training process of the example, the above step S102, pre-training the pre-configured visual model on the pre-training data set, and updating the model parameters according to the overall loss, may include:

[0060] Step S301: Obtain a pre-configured visual model.

[0061] Step S302: extract similar image pairs from the pre-training data set.

[0062] Step S303: calling the two visual models with a shared model structure to process the two enhanced remote sensing images in the extracted similar image pair respectively.

[0063] Step S304: Calculate a first cross entropy loss value and a first contrast loss value based on the processing results of the two visual models on the extracted similar image pairs, and perform weighted addition on the first cross entropy loss value and the first contrast loss value to obtain a first overall loss value.

[0064] Exemplary, combined Figure 3 In the pre-training process shown, the process of calling the visual model to process a remote sensing image may include: the convolution layer performs convolution processing on the input remote sensing image to obtain a first feature map, the pooling layer performs pooling processing on the first feature map to obtain a second feature map, the fully connected layer (FC) processes the second feature map to obtain a first feature vector, and the dropout layer processes the first feature vector to obtain a second feature vector. On this basis, the processing results of the two visual models on the extracted similar image pairs may include the second feature map and the second feature vector of each of the two remote sensing images (such as the enhanced remote sensing image s and the enhanced remote sensing image t) in the extracted similar image pair, the cross entropy loss is calculated based on the two second feature maps and two labels (such as the label of the enhanced remote sensing image s and the label of the enhanced remote sensing image t) to obtain the above-mentioned first cross entropy loss, the contrast loss is calculated based on the two second feature vectors to obtain the above-mentioned first contrast loss, and then the calculated first cross entropy loss and the first contrast loss are weighted to obtain the overall loss, that is, the above-mentioned first overall loss, which provides a basis for updating the model parameters.

[0065] Step S305, updating the model parameters of the visual model based on the first overall loss value, and returning to execute the step of extracting similar image pairs from the pre-trained data set until a preset pre-training stop condition is met to obtain the pre-trained visual model.

[0066] The above scheme reduces the model parameters by setting the pooling layer and the discarding layer, which provides a basis for improving the generalization of the model; the weighted calculation results of the two losses are used as the overall loss, and after multiple rounds of gradient iterations, the learning rate and optimization strategy are adjusted, which improves the representation and understanding ability of the pre-trained visual model to a certain extent. Based on the above content, the pre-training stage uses supervised and self-supervised comparative learning methods to train the visual model, and obtains a pre-trained visual model with good initial representation capabilities for remote sensing images.

[0067] In addition, self-supervised contrastive learning can also be used in the pre-training stage, that is, training the model on a large amount of unlabeled data, that is, using the data's own information as a supervisory signal for unsupervised learning, so as to train the model to learn the intrinsic structure and representation of the data. Since self-supervised learning does not require labeled labels, it can learn from a large amount of unlabeled sample data, which reduces the cost of data labeling, and by generating tasks, the model can obtain multiple samples from a single data, improving the efficiency of data utilization. Based on the self-supervised contrastive learning method, the model can learn a general, underlying feature representation. Based on this, by transferring the learned knowledge to specific downstream tasks, actual machine learning tasks can be solved with a small amount of labeled data. In other words, the model parameters can be updated based on the contrast loss to iterate the pre-trained visual model.

[0068] In one or more embodiments provided in the present application, the configuration process of each meta-training data set in the meta-training first data set and the meta-training second data set may include:

[0069] Step S401: Acquire a number of original remote sensing images marked with corresponding labels.

[0070] The labels of the original remote sensing images obtained match the tasks corresponding to the meta-training data set. That is, the labels of the original remote sensing images obtained when configuring the second meta-training data set correspond to the target tasks, and the labels of the original remote sensing images obtained when configuring the first meta-training data set correspond to the tasks related to the target tasks and the tasks not related to the target tasks.

[0071] Step S402: after removing the labels marked on the acquired original remote sensing images, perform at least two types of data enhancement processing on the acquired original remote sensing images to obtain at least two enhanced remote sensing images corresponding to the acquired original remote sensing images, and configure labels for the obtained similar image pairs.

[0072] Step S403: group two original remote sensing images corresponding to the same task and marked with different labels into one group to obtain a plurality of different-label image pairs.

[0073] Step S404: group two original remote sensing images corresponding to the same task and marked with the same label into one group to obtain a plurality of image pairs with the same label.

[0074] Step S405: Generate the meta-training data set.

[0075] The meta-training data set includes a number of remote sensing image pairs annotated with corresponding labels, and the remote sensing image pairs include similar image pairs, different-label image pairs and same-label image pairs.

[0076] On the basis of the above, the above step S104, performing model training on the preconfigured basic model on the first meta-training data set, and updating the model parameters according to the overall loss, may include:

[0077] Step S501: extract remote sensing image pairs from the meta-training first data set.

[0078] Step S502: calling the two pre-configured basic models of the shared model structure to process the two remote sensing images in the extracted remote sensing image pair respectively.

[0079] Step S503: Calculate a second cross entropy loss value and a second contrast loss value based on the processing results of the extracted remote sensing image pair by the two preconfigured basic models, and perform weighted addition on the second cross entropy loss value and the second contrast loss value to obtain a second overall loss value.

[0080] Exemplary, combined Figure 4 The meta-training process shown in the figure, calling the basic model to process a remote sensing image may include: the convolution layer performs convolution processing on the input remote sensing image to obtain a third feature map, the pooling layer (pool) performs pooling processing on the third feature map to obtain a fourth feature map, the fully connected layer (FC) processes the fourth feature map to obtain a third feature vector, and the attention mechanism layer (Atten) processes the third feature vector to obtain a fourth feature vector. On this basis, the processing results of the two visual models on the extracted remote sensing image pair may include the fourth feature map and fourth feature vector of each of the two remote sensing images (such as remote sensing image 1 and remote sensing image 2) in the extracted remote sensing image pair, calculating the cross entropy loss based on the two fourth feature maps and two labels (such as the label of remote sensing image 1 and the label of remote sensing image 2) to obtain the above-mentioned second cross entropy loss, calculating the contrast loss based on the two fourth feature vectors to obtain the above-mentioned second contrast loss, and then weighting the calculated second cross entropy loss and the second contrast loss to obtain the overall loss, that is, the above-mentioned second overall loss, which provides a basis for updating the model parameters.

[0081] Step S504: update the model parameters of the preconfigured basic model based on the second overall loss value, and return to execute step S501, extract remote sensing image pairs from the meta-training first data set until the preset meta-training first stop condition is met to obtain the pre-trained basic model.

[0082] In the meta-training stage, richer contrastive learning is utilized, that is, constructing positive and negative sample pairs, taking similar samples as positive samples, and taking dissimilar samples as negative samples; on this basis, by maximizing the similarity between positive sample pairs and minimizing the similarity between negative sample pairs, the model can learn the intrinsic structure and relationship of the data, thereby generating better data representation.

[0083] It should be noted that the meta-training stage (meta-learning) can be used to train the model on the task so that the model has the ability to solve other tasks. For example, in the meta-training stage, the model structure, hyperparameters, initialization parameters, etc. are first learned from the training samples of Task 1, and then evaluated based on the test samples of Task 1. Each group of tests can get a loss to characterize the quality of the algorithm; then the same process is performed on Task 2, and the final loss is finally obtained by averaging the losses of each task.

[0084] On the basis of the above, the above step S105, performing model training on the pre-trained basic model on the meta-training second data set, and updating the model parameters according to the overall loss, may include:

[0085] Step S601: extract remote sensing image pairs from the meta-training second data set.

[0086] Step S602: calling the two pre-trained basic models of the shared model structure to process the two remote sensing images in the extracted remote sensing image pair respectively.

[0087] Step S603: Calculate a third cross entropy loss value and a third contrast loss value based on the processing results of the extracted remote sensing image pair by the two pre-trained basic models, and perform weighted addition on the third cross entropy loss value and the third contrast loss value to obtain a third overall loss value.

[0088] Step S604: update the model parameters of the pre-trained basic model based on the third overall loss value, and return to execute step S601: extract remote sensing image pairs from the meta-training second data set until the preset meta-training second stop condition is met to obtain the target visual model.

[0089] Based on the above, the meta-training phase first trained the model on a variety of existing remote sensing data sets, allowing the basic model to learn a variety of remote sensing recognition tasks, giving it a general representation and understanding capability. Then, on the target task, the training data corresponding to the target task was used for tuning, improving the specialized capabilities of the visual model. Based on this, only a small number of samples are needed for the model to have a good understanding capability on the newly added tasks.

[0090] Based on the above, the embodiment of the present application provides a two-stage model training scheme based on contrastive learning, which solves the image recognition problem of small samples to a certain extent by integrating contrastive learning into the pre-training stage and the meta-training stage. Specifically, the above scheme uses self-supervision, supervised and contrastive learning to train the model in the pre-training stage. In the meta-training stage, the initial parameters are configured based on the model parameters of the pre-training stage, and a variety of mode training methods such as self-supervision data enhancement, grouping of data with the same label and grouping of data with different labels are adopted. Through the above scheme, less data can be relied on, the feature distance between classes can be increased, and the feature distance within the class can be reduced to achieve an accuracy similar to that of the model obtained by training based on a large amount of training data, that is, the present application can reduce the dependence of deep learning training on training data, and provide a basis for reducing data costs and quickly supporting the realization of new tasks.

[0091] The visual model training device provided in an embodiment of the present application is described below. The visual model training device described below and the visual model training method described above can be referenced to each other.

[0092] Figure 5 Schematic diagram of the structure of a visual model training device disclosed in the embodiment of the present application. Figure 5 As shown, the device may include:

[0093] The training data acquisition unit 11 is used to acquire a pre-training data set, a meta-training first data set and a meta-training second data set; wherein the pre-training data set, the meta-training first data set and the meta-training second data set respectively include: a plurality of remote sensing image pairs annotated with corresponding labels; the remote sensing image pairs in the pre-training data set are composed of two enhanced remote sensing images, and the two enhanced remote sensing images are obtained by subjecting the same original remote sensing image to different data enhancement processes; the remote sensing image pairs in the meta-training first data set and the meta-training second data set are composed of two remote sensing images with the same label, or two remote sensing images with different labels, or two enhanced remote sensing images corresponding to the same original remote sensing image; the label of each remote sensing image pair is composed of the labels of the two remote sensing images constituting the remote sensing image pair, and the labels in the pre-training data set and the meta-training first data set correspond to at least one remote sensing image recognition task, and any one of the at least one remote sensing image recognition tasks is a related task or an unrelated task of the target task; the labels in the meta-training second data set correspond to the target task;

[0094] A pre-training unit 12 is used to perform model pre-training on a pre-configured visual model on the pre-training data set, and update model parameters according to an overall loss to obtain a pre-trained visual model; the overall loss is a weighted loss of contrast loss and cross entropy loss;

[0095] The meta-training unit 13 is used to configure the model parameters of the pre-trained visual model as initial parameters to the basic model to obtain a pre-configured basic model, wherein the pre-configured basic model includes an attention module; perform model training on the pre-configured basic model on the meta-training first data set, and update the model parameters according to the overall loss to obtain the pre-trained basic model; perform model training on the pre-trained basic model on the meta-training second data set, and update the model parameters according to the overall loss to obtain a target visual model for achieving the target task.

[0096] In one or more embodiments provided in the present application, the apparatus may further include a training data configuration unit for configuring the pre-training data set.

[0097] Based on the above, the process of configuring the pre-training data set by the training data configuration unit may include:

[0098] Obtain several original remote sensing images;

[0099] Performing at least two data enhancement processes on the acquired original remote sensing image to obtain at least two enhanced remote sensing images corresponding to the acquired original remote sensing image;

[0100] Grouping two different enhanced remote sensing images corresponding to the same original remote sensing image into one group, obtaining a number of similar image pairs, and configuring labels for the obtained similar image pairs;

[0101] The pre-training data set is generated, and the pre-training data set includes a plurality of similar image pairs annotated with corresponding labels.

[0102] In one or more embodiments provided in the present application, the process in which the pre-training unit 12 performs model pre-training on the pre-configured visual model on the pre-training data set and updates the model parameters according to the overall loss may include:

[0103] Get pre-configured vision models;

[0104] Extracting similar image pairs from the pre-training dataset;

[0105] Calling the two visual models with a shared model structure to process the two enhanced remote sensing images in the extracted similar image pair respectively;

[0106] According to the processing results of the two visual models on the extracted similar image pairs, a first cross entropy loss value and a first contrast loss value are calculated, and the first cross entropy loss value and the first contrast loss value are weightedly added to obtain a first overall loss value;

[0107] The model parameters of the visual model are updated based on the first overall loss value, and the step of extracting similar image pairs from the pre-trained data set is returned to be executed until a preset pre-training stop condition is met, thereby obtaining the pre-trained visual model.

[0108] In one or more embodiments provided in the present application, the apparatus may further include a training data configuration unit, configured to configure the meta-training first data set and the meta-training second data set.

[0109] On the basis of the above, the process of the training data configuration unit configuring each meta-training data set in the meta-training first data set and the meta-training second data set may include:

[0110] Acquire a number of original remote sensing images annotated with corresponding labels; the labels of the acquired original remote sensing images are matched with the tasks corresponding to the meta-training data set;

[0111] After removing the labels marked on the acquired original remote sensing images, performing at least two types of data enhancement processing on the acquired original remote sensing images to obtain at least two enhanced remote sensing images corresponding to the acquired original remote sensing images, and configuring labels for the obtained similar image pairs;

[0112] Two original remote sensing images corresponding to the same task and marked with different labels are grouped together to obtain several pairs of images with different labels;

[0113] Two original remote sensing images corresponding to the same task and marked with the same label are grouped together to obtain several image pairs with the same label;

[0114] The meta-training data set is generated, wherein the meta-training data set includes a plurality of remote sensing image pairs annotated with corresponding labels, wherein the remote sensing image pairs include similar image pairs, different-label image pairs, and same-label image pairs.

[0115] In one or more embodiments provided in the present application, the process in which the meta-training unit 13 performs model training on the preconfigured basic model on the meta-training first data set and updates the model parameters according to the overall loss may include:

[0116] Extracting remote sensing image pairs from the meta-training first data set;

[0117] Calling the two pre-configured basic models of the shared model structure to process two remote sensing images in the extracted remote sensing image pair respectively;

[0118] Calculate a second cross entropy loss value and a second contrast loss value according to the processing results of the extracted remote sensing image pair by the two preconfigured basic models, and perform weighted addition on the second cross entropy loss value and the second contrast loss value to obtain a second overall loss value;

[0119] Based on the second overall loss value, the model parameters of the preconfigured basic model are updated, and the step of extracting remote sensing image pairs from the meta-training first data set is returned to be executed until the preset meta-training first stop condition is met to obtain the pre-trained basic model.

[0120] In one or more embodiments provided in the present application, the process in which the meta-training unit 13 performs model training on the pre-trained basic model on the meta-training second data set and updates the model parameters according to the overall loss may include:

[0121] Extracting remote sensing image pairs from the meta-training second dataset;

[0122] Calling the two pre-trained basic models of the shared model structure to process two remote sensing images in the extracted remote sensing image pair respectively;

[0123] According to the processing results of the extracted remote sensing image pair by the two pre-trained basic models, a third cross entropy loss value and a third contrast loss value are calculated, and the third cross entropy loss value and the third contrast loss value are weightedly added to obtain a third overall loss value;

[0124] Based on the third overall loss value, the model parameters of the pre-trained basic model are updated, and the step of extracting remote sensing image pairs from the meta-training second data set is returned to be executed until a preset meta-training second stop condition is met to obtain the target visual model.

[0125] Optionally, the target task may be any one of a classification task, a positioning task, a detection task and a segmentation task in the field of remote sensing image processing. The segmentation task is taken as an example for explanation below.

[0126] The present application provides a remote sensing image segmentation method, which may include:

[0127] Step S701, obtaining a remote sensing image to be segmented;

[0128] Step S702: calling the target visual model to process the remote sensing image to be segmented, and obtaining a segmentation result of a specified segmentation object in the remote sensing image to be segmented.

[0129] Among them, the target visual model is trained according to any of the above-mentioned visual model training methods, and the target task is the remote sensing image segmentation task of the specified segmentation object, and the related tasks of the target task include: remote sensing image segmentation tasks of other segmentation objects; irrelevant tasks of the target task include at least one of the classification task, positioning task and detection task of the remote sensing image.

[0130] The remote sensing image segmentation device provided in the embodiment of the present application is described below. The remote sensing image segmentation device described below and the remote sensing image segmentation method described above can be referenced to each other.

[0131] The present application provides a remote sensing image segmentation device, which may include:

[0132] An image acquisition unit, used for acquiring remote sensing images to be segmented;

[0133] The image segmentation unit is used to call the target visual model to process the remote sensing image to be segmented, so as to obtain the segmentation result of the designated segmentation object in the remote sensing image to be segmented.

[0134] Among them, the target visual model is trained according to any of the above-mentioned visual model training methods, and the target task is the remote sensing image segmentation task of the specified segmentation object, and the related tasks of the target task include: remote sensing image segmentation tasks of other segmentation objects; irrelevant tasks of the target task include at least one of the classification task, positioning task and detection task of the remote sensing image.

[0135] The visual model training device or remote sensing image segmentation device provided in the embodiment of the present application can be applied to image processing equipment, such as a terminal with data processing capabilities: a computer, a server, etc. Optionally, Figure 6 The hardware structure diagram of the image processing device is shown in FIG. Figure 6 , the hardware structure of the image processing device may include: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4;

[0136] In the embodiment of the present application, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 communicate with each other through the communication bus 4;

[0137] The processor 1 may be a central processing unit CPU, or an application-specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention, etc.;

[0138] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), etc., such as at least one disk memory;

[0139] Among them, the memory is used to store computer programs, and the processor is used to execute the computer programs, so that the image processing device can implement any of the above-mentioned visual model training methods or remote sensing image segmentation methods.

[0140] A storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any visual model training method or remote sensing image segmentation method provided in the embodiment of the present application.

[0141] Also provided in an embodiment of the present application is a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the visual model training methods or remote sensing image segmentation methods provided in the embodiments of the present application.

[0142] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0143] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

[0144] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A visual model training method, characterized in that: include: A pre-training data set, a first meta-training data set and a second meta-training data set are obtained; wherein the pre-training data set, the first meta-training data set and the second meta-training data set respectively include: a plurality of remote sensing image pairs annotated with corresponding labels; the remote sensing image pairs in the pre-training data set are composed of two enhanced remote sensing images, and the two enhanced remote sensing images are obtained by subjecting the same original remote sensing image to different data enhancement processes; the remote sensing image pairs in the first meta-training data set and the second meta-training data set are composed of two remote sensing images with the same label, or two remote sensing images with different labels, or two enhanced remote sensing images corresponding to the same original remote sensing image; the label of each remote sensing image pair is composed of the labels of the two remote sensing images constituting the remote sensing image pair, and the labels in the pre-training data set and the first meta-training data set correspond to at least one remote sensing image recognition task, and any one of the at least one remote sensing image recognition tasks is a related task or an unrelated task of the target task; the labels in the second meta-training data set correspond to the target task; Pre-training a pre-configured visual model on the pre-training data set, and updating model parameters according to the overall loss to obtain a pre-trained visual model; the overall loss is a weighted loss of contrast loss and cross entropy loss; Using the model parameters of the pre-trained visual model as initial parameters, and configuring them to the basic model to obtain a pre-configured basic model, wherein the pre-configured basic model includes an attention module; Performing model training on the pre-configured basic model on the first meta-training data set, and updating model parameters according to the overall loss to obtain a pre-trained basic model; The pre-trained basic model is trained on the meta-training second data set, and the model parameters are updated according to the overall loss to obtain a target visual model for achieving the target task.

2. The visual model training method according to claim 1, characterized in that: The configuration process of the pre-training data set includes: Obtain several original remote sensing images; Performing at least two data enhancement processes on the acquired original remote sensing image to obtain at least two enhanced remote sensing images corresponding to the acquired original remote sensing image; Grouping two different enhanced remote sensing images corresponding to the same original remote sensing image into one group, obtaining a number of similar image pairs, and configuring labels for the obtained similar image pairs; The pre-training data set is generated, and the pre-training data set includes a plurality of similar image pairs annotated with corresponding labels.

3. The visual model training method according to claim 2, characterized in that: Pre-training the pre-configured visual model on the pre-training dataset, and updating the model parameters according to the overall loss, including: Get pre-configured vision models; Extracting similar image pairs from the pre-training dataset; Calling the two visual models with a shared model structure to process the two enhanced remote sensing images in the extracted similar image pair respectively; According to the processing results of the two visual models on the extracted similar image pairs, a first cross entropy loss value and a first contrast loss value are calculated, and the first cross entropy loss value and the first contrast loss value are weightedly added to obtain a first overall loss value; The model parameters of the visual model are updated based on the first overall loss value, and the step of extracting similar image pairs from the pre-trained data set is returned to be executed until a preset pre-training stop condition is met, thereby obtaining the pre-trained visual model.

4. The visual model training method according to any one of claims 1 to 3, characterized in that: The configuration process of each meta-training data set in the meta-training first data set and the meta-training second data set includes: Acquire a number of original remote sensing images annotated with corresponding labels; the labels of the acquired original remote sensing images are matched with the tasks corresponding to the meta-training data set; After removing the labels marked on the acquired original remote sensing images, performing at least two types of data enhancement processing on the acquired original remote sensing images to obtain at least two enhanced remote sensing images corresponding to the acquired original remote sensing images, and configuring labels for the obtained similar image pairs; Two original remote sensing images corresponding to the same task and marked with different labels are grouped together to obtain several pairs of images with different labels; Two original remote sensing images corresponding to the same task and marked with the same label are grouped together to obtain several image pairs with the same label; The meta-training data set is generated, wherein the meta-training data set includes a plurality of remote sensing image pairs annotated with corresponding labels, wherein the remote sensing image pairs include similar image pairs, different-label image pairs, and same-label image pairs.

5. The visual model training method according to claim 4, characterized in that: Performing model training on the preconfigured basic model on the first meta-training data set, and updating model parameters according to the overall loss, including: Extracting remote sensing image pairs from the meta-training first data set; Calling the two pre-configured basic models of the shared model structure to process two remote sensing images in the extracted remote sensing image pair respectively; Calculate a second cross entropy loss value and a second contrast loss value according to the processing results of the extracted remote sensing image pair by the two preconfigured basic models, and perform weighted addition on the second cross entropy loss value and the second contrast loss value to obtain a second overall loss value; Based on the second overall loss value, the model parameters of the preconfigured basic model are updated, and the step of extracting remote sensing image pairs from the meta-training first data set is returned to be executed until the preset meta-training first stop condition is met to obtain the pre-trained basic model.

6. The visual model training method according to claim 4, characterized in that: Performing model training on the pre-trained basic model on the meta-training second data set, and updating model parameters according to the overall loss, including: Extracting remote sensing image pairs from the meta-training second dataset; Calling the two pre-trained basic models of the shared model structure to process two remote sensing images in the extracted remote sensing image pair respectively; According to the processing results of the extracted remote sensing image pair by the two pre-trained basic models, a third cross entropy loss value and a third contrast loss value are calculated, and the third cross entropy loss value and the third contrast loss value are weightedly added to obtain a third overall loss value; Based on the third overall loss value, the model parameters of the pre-trained basic model are updated, and the step of extracting remote sensing image pairs from the meta-training second data set is returned to be executed until a preset meta-training second stop condition is met to obtain the target visual model.

7. A visual model training device, characterized in that: include: A training data acquisition unit is used to acquire a pre-training data set, a meta-training first data set and a meta-training second data set; wherein the pre-training data set, the meta-training first data set and the meta-training second data set respectively include: a plurality of remote sensing image pairs annotated with corresponding labels; the remote sensing image pairs in the pre-training data set are composed of two enhanced remote sensing images, and the two enhanced remote sensing images are obtained by subjecting the same original remote sensing image to different data enhancement processes; the remote sensing image pairs in the meta-training first data set and the meta-training second data set are composed of two remote sensing images with the same label, or two remote sensing images with different labels, or two enhanced remote sensing images corresponding to the same original remote sensing image; the label of each remote sensing image pair is composed of the labels of the two remote sensing images constituting the remote sensing image pair, and the labels in the pre-training data set and the meta-training first data set correspond to at least one remote sensing image recognition task, and any one of the at least one remote sensing image recognition tasks is a related task or an unrelated task of the target task; the labels in the meta-training second data set correspond to the target task; A pre-training unit, used for performing model pre-training on a pre-configured visual model on the pre-training data set, and updating model parameters according to an overall loss to obtain a pre-trained visual model; the overall loss is a weighted loss of a contrast loss and a cross entropy loss; A meta-training unit is used to configure the model parameters of the pre-trained visual model as initial parameters to the basic model to obtain a pre-configured basic model, wherein the pre-configured basic model includes an attention module; perform model training on the pre-configured basic model on the meta-training first data set, and update the model parameters according to the overall loss to obtain the pre-trained basic model; perform model training on the pre-trained basic model on the meta-training second data set, and update the model parameters according to the overall loss to obtain a target visual model for achieving the target task.

8. A remote sensing image segmentation method, characterized in that: The method includes: Obtaining remote sensing images to be segmented; Invoking the target visual model to process the remote sensing image to be segmented, and obtaining a segmentation result of a specified segmentation object in the remote sensing image to be segmented; Wherein, the target visual model is trained according to the visual model training method described in any one of claims 1-6, and the target task is the remote sensing image segmentation task of the specified segmentation object, and the related tasks of the target task include: remote sensing image segmentation tasks of other segmentation objects; irrelevant tasks of the target task include at least one of the classification task, positioning task and detection task of the remote sensing image.

9. An image processing device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the image processing device can implement the visual model training method as described in any one of claims 1 to 6, or implement the remote sensing image segmentation method as described in claim 8.

10. A storage medium, characterized in that: The storage medium carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the visual model training method as described in any one of claims 1 to 6, or the remote sensing image segmentation method as described in claim 8.

Images