Star-ground collaborative labeling fine-tuning method, system and device for data label scarcity

Abstract

The application provides a satellite-ground collaborative labeling fine-tuning method, system and equipment for data label scarcity, which comprises the following steps: a plurality of satellite terminals respectively receive a pre-trained satellite-borne model issued by a ground station; the plurality of satellite terminals utilize unlabeled data to perform unsupervised comparative fine-tuning on the pre-trained satellite-borne model, thereby obtaining fine-tuning parameters of an initial satellite-borne model of this round; the ground station aggregates the fine-tuning parameters of the initial satellite-borne model, thereby obtaining fine-tuning parameters of an aggregated satellite-borne model of this round; after a plurality of rounds of collaborative labeling fine-tuning, the plurality of satellite terminals obtain a first converged satellite-borne model; the plurality of satellite terminals respectively utilize labeled data to perform supervised fine-tuning on the first converged satellite-borne model, thereby obtaining a collaborative labeling fine-tuned satellite-borne model of each satellite terminal; and the collaborative labeling fine-tuned satellite-borne model of each satellite terminal is used for class prediction on image data collected by the satellite terminal, thereby obtaining labeled image data.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of satellite computing, and in particular to a satellite-ground collaborative fine-tuning method, system and device for data label scarcity. BACKGROUND

[0002] In recent years, satellite technology has made significant progress, and commercial off-the-shelf (COTS) hardware has rapidly developed. It has become a development trend for low-orbit satellite constellations to be equipped with intelligent computing capabilities. Mainstream cloud service providers (such as Google and AWS) and satellite companies (such as OrbitsEdge) have also proposed the "space infrastructure as a service" initiative. Through the satellite computing paradigm, on-orbit data can be quickly and intelligently processed on-orbit, improving on-orbit service quality. For example, during Hurricane Harvey and the 2021 Turkey earthquake, Capella Space satellite constellations used intelligent chips to shorten the time for generating disaster emergency maps by 91.6%, quickly providing key and accurate positioning services for rescue personnel, and significantly reducing the scope of disaster and casualties. Therefore, satellite intelligent computing has shown great help for global challenges such as disaster navigation and disaster reduction, and has great potential to support various complex and dynamic on-orbit services.

[0003] Although satellite intelligent computing has shown great value and potential, its system performance still restricts its ability to continuously provide high-quality on-orbit services during its actual deployment and operation. Specifically, due to physical conditions such as satellite size and passive heat dissipation, it is usually difficult to improve system model performance by increasing the size of the deployed model, and system model performance directly affects the accuracy of on-orbit results. Inaccurate results cannot effectively help the provision of on-orbit services. Therefore, considering the use of a federated learning paradigm for collaborative fine-tuning between satellites and the ground provides a promising solution for continuous improvement of the performance of size-constrained on-board models. In particular, each satellite continuously collects a large amount of image data during on-orbit operation every day, usually reaching tens of terabytes per day. Therefore, the satellite-ground federated learning system has the opportunity to learn potential knowledge from the data collected by each satellite and share it, thereby providing an opportunity for the performance evolution of the on-board model of each satellite.

[0004] Some previous technical research has explored how to improve the performance and efficiency of the satellite-ground federated system. Specifically, FedSpace first proposed a satellite-ground federated system and further designed an adaptive buffer to improve system efficiency by balancing the synchronization and asynchronous aggregation of the satellite-ground federated process. Subsequent research further explores how to efficiently improve the performance of the satellite-ground federated system in a resource-constrained on-orbit environment.

[0005] However, although the above research makes a lot of efforts, due to the fact that the data collected in orbit is usually label-scarce, most satellites are difficult to be equipped with professional labeling ability of device model for on-orbit labeling, and it is also difficult to recruit highly specialized data labelers for real-time collaboration, and the lack of data labels brings great difficulties to the supervised fine-tuning process. SUMMARY

[0006] Therefore, the embodiments of the present application provide a star-ground collaborative labeling fine-tuning method, system and device for data label-scarce, so as to overcome the above problems or at least partially solve the above problems.

[0007] The first aspect of the embodiments of the present application provides a star-ground collaborative labeling fine-tuning method for data label-scarce, the method comprising:

[0008] The plurality of satellite terminals respectively receive a pre-trained satellite-borne model issued by the ground station, the pre-trained satellite-borne model being obtained by performing parameter initialization on a to-be-fine-tuned model pre-trained by the ground station, the to-be-fine-tuned model having a function of classifying image data collected by any satellite terminal;

[0009] In each round of collaborative labeling fine-tuning process, the plurality of satellite terminals respectively perform unsupervised contrast fine-tuning on the pre-trained satellite-borne model by using the unlabeled data in the local data set, to obtain the fine-tuning parameters of the initial satellite-borne model of this round;

[0010] The plurality of satellite terminals send the respective fine-tuning parameters of the initial satellite-borne model to the ground station;

[0011] The ground station aggregates the received fine-tuning parameters of the plurality of initial satellite-borne models to obtain the fine-tuning parameters of the aggregated satellite-borne model of this round, and issues the fine-tuning parameters of the aggregated satellite-borne model to the plurality of satellite terminals;

[0012] After the pre-trained satellite-borne model is subjected to multiple rounds of collaborative labeling fine-tuning between the ground station and the plurality of satellite terminals, the plurality of satellite terminals obtain a first converged satellite-borne model, the first converged satellite-borne model being model parameters having a generalization knowledge representation;

[0013] The plurality of satellite terminals respectively perform supervised fine-tuning on the first converged satellite-borne model by using the labeled data in the local data set, to obtain the collaborative labeling fine-tuned satellite-borne model of each satellite terminal, and the collaborative labeling fine-tuned satellite-borne model of each satellite terminal is used for classifying the image data collected by the satellite terminal to obtain labeled image data.

[0014] Optionally, after the plurality of satellite terminals respectively receive the pre-trained satellite-borne model issued by the ground station, the method further comprises:

[0015] The plurality of satellite terminals respectively split the pre-trained satellite-borne model into an encoder network close to an input layer and a decoder network close to an output layer, and freeze parameters of the decoder network;

[0016] In each round of the collaborative labeling fine-tuning process, the plurality of satellite terminals each use unlabeled data in a local data set to perform unsupervised contrastive fine-tuning on the pre-trained satellite-borne model to obtain fine-tuning parameters of an initial satellite-borne model of the round, including: in each round of the collaborative labeling fine-tuning process, the plurality of satellite terminals each use unlabeled data in a local data set to perform unsupervised contrastive fine-tuning on an encoder network in the pre-trained satellite-borne model to obtain fine-tuning parameters of an initial satellite-borne model of the round, the fine-tuning parameters of the initial satellite-borne model being parameters of a fine-tuned encoder network.

[0017] The first converged satellite-borne model includes parameters of a converged encoder network and the frozen parameters of the decoder network.

[0018] Optionally, the plurality of satellite terminals send the fine-tuning parameters of the respective initial satellite-borne models to the ground station, including:

[0019] The central server deployed by the ground station randomly selects a plurality of satellite terminals from a plurality of connectable satellite terminals included in the plurality of satellite terminals as satellite terminals participating in collaborative aggregation;

[0020] The plurality of satellite terminals participating in collaborative aggregation send the fine-tuning parameters of the respective initial satellite-borne models to the ground station.

[0021] Optionally, the ground station aggregates the received fine-tuning parameters of the plurality of initial satellite-borne models to obtain fine-tuning parameters of an aggregated satellite-borne model of the round, including:

[0022] When the number of satellite terminals participating in collaborative aggregation is less than a preset number threshold, the ground station selects, from fine-tuning parameters of a plurality of aggregated satellite-borne models obtained in the previous N rounds of the collaborative labeling fine-tuning process, fine-tuning parameters of an aggregated satellite-borne model that is most similar to the fine-tuning parameters of the plurality of initial satellite-borne models sent by the plurality of satellite terminals participating in collaborative aggregation to the ground station, N > 1.

[0023] The ground station aggregates the received fine-tuning parameters of the plurality of initial satellite-borne models sent by the plurality of satellite terminals participating in collaborative aggregation and the most similar fine-tuning parameters of the aggregated satellite-borne model to obtain fine-tuning parameters of an aggregated satellite-borne model of the round.

[0024] Optionally, the plurality of satellite terminals obtain a first converged satellite-borne model, including:

[0025] The plurality of satellite terminals obtain the parameters of the converged encoder network issued by the ground station after the plurality of satellite terminals perform multiple rounds of collaborative annotation fine-tuning between the satellite-borne model and the ground station and the plurality of satellite terminals.

[0026] The plurality of satellite terminals splice the received parameters of the converged encoder network with the parameters of the frozen decoder network to obtain the first converged satellite-borne model.

[0027] Optionally, before the collaborative annotation fine-tuning process is performed, the method further comprises:

[0028] setting the initialization parameters and the model structure to obtain the pre-trained satellite-borne model;

[0029] setting the first related information participating in the collaborative process, the first related information comprising: the number of the plurality of satellite terminals, the frequency of each of the plurality of satellite terminals participating in the collaborative aggregation, the label scarcity degree of the local data set of each of the plurality of satellite terminals and the label scarcity degree of the local data set, the deviation operator of each of the plurality of satellite terminals and the number of measurement fine-tuning rounds;

[0030] The label scarcity degree is used to represent the proportion of the labeled data in the satellite terminal, and the label deviation degree is used to represent the distribution of the labeled data in the plurality of satellite terminals. The deviation operator represents an operation operator for simulating labeled data with different label deviation degrees on the plurality of satellite terminals. The number of measurement fine-tuning rounds represents the number of times of collaborative aggregation.

[0031] Optionally, after obtaining the satellite-borne model after the collaborative annotation fine-tuning of each of the plurality of satellite terminals, the method further comprises:

[0032] obtaining the real labels of the first test data set, the first test data set comprising first unlabeled data and first labeled data;

[0033] annotating the first unlabeled data by using the satellite-borne model after the collaborative annotation fine-tuning to obtain the first unlabeled data after labeling;

[0034] calculating the annotation accuracy of the satellite-borne model after the collaborative annotation fine-tuning according to the first unlabeled data after labeling and the real labels of the first unlabeled data in the real labels of the first test data set, the annotation accuracy of the satellite-borne model after the collaborative annotation fine-tuning being the proportion of the first unlabeled data annotated correctly in the first unlabeled data after labeling to the total first unlabeled data;

[0035] mixing the first unlabeled data after labeling and the first labeled data to obtain mixed labeled data;

[0036] The mixed labeling data is used for re-fine-tuning the collaborative labeling fine-tuned satellite model, to obtain a first satellite model, the system accuracy of the first satellite model is calculated through a second test data set after the first satellite model is re-fine-tuned through the mixed labeling data; the system accuracy of the first satellite model is; after the first satellite model is re-fine-tuned by using the mixed labeling data, the accuracy that the first satellite model can reach after labeling the second test data set;

[0037] Based on the labeling accuracy of the collaborative labeling fine-tuned satellite model and the system accuracy of the first satellite model, the performance improvement of the collaborative labeling fine-tuned satellite model is evaluated.

[0038] Optionally, before evaluating the performance improvement of the collaborative labeling fine-tuned satellite model, it further includes:

[0039] Setting initialization parameters and model structure to obtain a to-be-fine-tuned satellite model;

[0040] Set the second related information participating in the collaborative process, the second related information includes: the number of multiple satellite terminals, the frequency of each satellite terminal participating in the collaborative process, the label scarcity degree and the label deviation degree of the local training data set of each satellite terminal, the deviation operator of each satellite terminal and the measurement fine-tuning round number;

[0041] The multiple satellite terminals participating in the collaborative process each use the third unlabeled data in the local training data set to perform unsupervised contrast fine-tuning on the to-be-fine-tuned satellite model, to obtain the fine-tuning parameters of the current fine-tuned satellite model;

[0042] The central server of the ground station randomly selects multiple satellite terminals from the multiple satellite terminals participating in the collaborative process as satellite terminals participating in collaborative aggregation, and aggregates the fine-tuning parameters of the multiple current fine-tuned satellite models obtained by the multiple satellite terminals participating in collaborative aggregation through unsupervised contrast fine-tuning, to obtain the fine-tuning parameters of the current aggregated satellite model;

[0043] The ground station sends the fine-tuning parameters of the current aggregated satellite model to the multiple satellite terminals for the next round of collaborative process;

[0044] After multiple rounds of collaborative process, the multiple satellite terminals participating in the collaborative process each obtain a second converged satellite model;

[0045] The multiple satellite terminals participating in the collaborative process each use the labeled data in the local training data set to perform local fine-tuning on the second converged satellite model, to obtain a fine-tuned satellite model;

[0046] The multiple satellite terminals participating in the collaborative process perform class prediction on third unlabeled data in the local training data set by the fine-tuned satellite-borne model, and label the third unlabeled data based on the results of the class prediction to obtain labeled third unlabeled data;

[0047] The label accuracy of the fine-tuned satellite-borne model is determined according to the true labels of the third unlabeled data in the local training data set and the labeled third unlabeled data;

[0048] The multiple satellite terminals participating in the collaborative process each perform supervised fine-tuning on the fine-tuned satellite-borne model using the labeled third unlabeled data in the local training data set and the third labeled data in the local training data set to obtain a final fine-tuned satellite-borne model;

[0049] The multiple satellite terminals participating in the collaborative process each test the accuracy of the final fine-tuned satellite-borne model using a third test data set to obtain the system accuracy of the final fine-tuned satellite-borne model;

[0050] The second relevant information is reset, and the above steps are repeated to obtain a corresponding relationship between the labeling accuracy and the system accuracy of the final fine-tuned satellite-borne model under different label scarcity degrees and different label deviation degrees;

[0051] The performance improvement of the collaborative labeling fine-tuned satellite-borne model is evaluated, including determining the performance improvement degree of the collaborative labeling fine-tuned satellite-borne model based on the corresponding relationship, the labeling accuracy of the collaborative labeling fine-tuned satellite-borne model, and the system accuracy of the collaborative labeling fine-tuned satellite-borne model.

[0052] The second aspect of the embodiments of the application provides a satellite-ground collaborative labeling fine-tuning system for data label scarcity, and the system includes multiple satellite terminals and a ground station:

[0053] The multiple satellite terminals are configured to receive a pre-trained satellite-borne model issued by the ground station, the pre-trained satellite-borne model is obtained by parameter initialization of a to-be-fine-tuned model pre-trained by the ground station, and the to-be-fine-tuned model has a function of performing class prediction on image data collected by any satellite terminal;

[0054] In each round of the collaborative labeling fine-tuning process, the multiple satellite terminals are configured to each perform unsupervised contrast fine-tuning on the pre-trained satellite-borne model using unlabeled data in a local data set to obtain fine-tuning parameters of an initial satellite-borne model of the round;

[0055] The multiple satellite terminals are configured to send the fine-tuning parameters of the respective initial satellite-borne models to the ground station;

[0056] The ground station is configured to aggregate the fine-tuning parameters of the plurality of initial satellite-borne models to obtain fine-tuning parameters of an aggregated satellite-borne model of the current round, and to distribute the fine-tuning parameters of the aggregated satellite-borne model to the plurality of satellite terminals.

[0057] After the pre-trained satellite-borne model is subjected to the multi-round collaborative labeling fine-tuning between the ground station and the plurality of satellite terminals, the plurality of satellite terminals is configured to receive a first converged satellite-borne model, the first converged satellite-borne model being a model parameter having a generalization knowledge representation.

[0058] The plurality of satellite terminals is configured to respectively perform supervised fine-tuning on the first converged satellite-borne model by using labeled data in a local data set to obtain a collaborative labeling fine-tuned satellite-borne model of each of the plurality of satellite terminals, and each satellite terminal's collaborative labeling fine-tuned satellite-borne model is configured to perform category prediction on image data collected by the satellite terminal to obtain labeled image data.

[0059] The third aspect of the embodiments of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method of the first aspect.

[0060] The beneficial effects of the present application are as follows:

[0061] The embodiment of the application provides a satellite-ground collaborative labeling fine-tuning method, system and device for data label scarcity, the method comprising: a plurality of satellite terminals respectively receiving a pre-trained satellite-borne model issued by a ground station, the pre-trained satellite-borne model being obtained by initializing parameters of a to-be-fine-tuned model pre-trained by the ground station, the to-be-fine-tuned model having a function of performing category prediction on image data collected by any satellite terminal; in each round of collaborative labeling fine-tuning process, the plurality of satellite terminals respectively use unlabelled data in a local data set to perform unsupervised contrast fine-tuning on the pre-trained satellite-borne model, to obtain fine-tuning parameters of an initial satellite-borne model of the round; the plurality of satellite terminals send the respective fine-tuning parameters of the initial satellite-borne model to the ground station; the ground station aggregates the received fine-tuning parameters of the plurality of initial satellite-borne models, to obtain fine-tuning parameters of an aggregated satellite-borne model of the round, and issues the fine-tuning parameters of the aggregated satellite-borne model to the plurality of satellite terminals; after the pre-trained satellite-borne model performs multiple rounds of collaborative labeling fine-tuning between the ground station and the plurality of satellite terminals, the plurality of satellite terminals obtain a first converged satellite-borne model, the first converged satellite-borne model being a model parameter having a generalization knowledge representation; the plurality of satellite terminals respectively use labelled data in a local data set to perform supervised fine-tuning on the first converged satellite-borne model, to obtain a collaborative labeling fine-tuned satellite-borne model of each of the plurality of satellite terminals, and the collaborative labeling fine-tuned satellite-borne model of each satellite terminal is used for performing category prediction on image data collected by the satellite terminal, to obtain labelled image data.

[0062] Through the technical solution of the application, the satellite terminal can learn the feature representation in the unlabelled data in the case of label data scarcity, reducing the dependence on label data. The subsequent supervised fine-tuning further improves the performance of the model on specific tasks, and even in the case of label data scarcity, a high labeling accuracy can be achieved. This combination of unsupervised contrast fine-tuning and supervised fine-tuning enables the model to better extract feature representations from image data and optimize with a small amount of labelled data, thereby significantly improving the labeling accuracy. Moreover, the multi-round collaborative fine-tuning and parameter aggregation enable the model to integrate the learning results of different satellite terminals, enhancing the generalization ability of the model. By sharing the feature representation of the data, the model can better adapt to the data characteristics of different satellites. In addition, through the satellite-ground collaborative labeling fine-tuning, the model can adapt to the particularity of the satellite data. In summary, the application solves the technical problem of satellite data label scarcity through this satellite-ground collaborative labeling fine-tuning method, improves the labeling ability and overall performance of the model, and meets the needs of satellite terminals in actual applications. BRIEF DESCRIPTION OF DRAWINGS

[0063] The accompanying drawings, which form a part of the present application, are intended to provide further understanding of the present application and are incorporated herein for a purpose of explanations and are not for a purpose of limitations of the present application.

[0064] In order to more clearly illustrate the technical solutions of the present application, the drawings required to be used in the description of the present application will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative labor under the premise of the drawings.

[0065] Figure 1 is a flow chart of a satellite-ground collaborative labeling fine-tuning method for data label scarcity according to an embodiment of the present application;

[0066] Figure 2 is a whole framework diagram of a satellite-ground collaborative labeling fine-tuning method for data label scarcity according to an embodiment of the present application;

[0067] Figure 3 is a schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0068] It should be noted that the embodiments and the features in the embodiments in the present application can be combined with each other without conflict.

[0069] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative labor are within the scope of protection of the present application.

[0070] Reference Figure 1 , Figure 1 is a flow chart of a satellite-ground collaborative labeling fine-tuning method for data label scarcity according to an embodiment of the present application, Figure 2 is a whole framework diagram of a satellite-ground collaborative labeling fine-tuning method for data label scarcity according to an embodiment of the present application, specifically, the embodiment provides a satellite-ground collaborative labeling fine-tuning method for data label scarcity, the method comprises steps S11 to S16:

[0071] Step S11, a plurality of satellite terminals respectively receive a pre-trained satellite-borne model issued by a ground station, the pre-trained satellite-borne model is obtained by parameter initialization of a to-be-fine-tuned model pre-trained by the ground station, and the to-be-fine-tuned model has a function of classifying prediction on image data collected by any satellite terminal.

[0072] As shown in Figure 2 The method is applied to a satellite-ground federation system including multiple satellite terminals and a ground station. In an initial stage of a cooperative labeling fine-tuning process between the multiple satellite terminals and the ground station, the multiple satellite terminals respectively receive model parameters of a pre-trained satellite-borne model issued by the ground station. The pre-trained satellite-borne model is a model to be fine-tuned that is pre-trained by the ground station and is obtained by initializing a specific neural network model selected by a central server of the ground station.

[0073] The pre-trained satellite-borne model has a function of making a category prediction on image data collected by any satellite terminal. The purpose of the pre-trained model is to provide a satellite terminal with a model that is preliminarily trained, so as to improve the accuracy of a category prediction on image data by the model after fine-tuning of the pre-trained satellite-borne model and to make the fine-tuned model better adapt to characteristics of image data collected by each satellite terminal.

[0074] In step S12, in each round of the cooperative labeling fine-tuning process, the multiple satellite terminals respectively perform unsupervised contrastive fine-tuning on the pre-trained satellite-borne model by using unlabeled data in a local data set, to obtain fine-tuning parameters of an initial satellite-borne model in the round.

[0075] In the embodiment, the multiple satellite terminals and the ground station will perform multiple rounds of the cooperative labeling fine-tuning process. In each round of the cooperative labeling fine-tuning process, the multiple satellite terminals respectively perform unsupervised contrastive fine-tuning on the pre-trained satellite-borne model by using unlabeled data in a local data set. The local data set for each satellite terminal is a collection of a large amount of image data collected by the satellite terminal. The image data in the local data set is divided into two categories, i.e., unlabeled data and labeled data. The labeled data refers to image data containing a pre-accurately labeled category, and the unlabeled data refers to image data not containing a pre-accurately labeled category. As an advanced learning method, unsupervised contrastive fine-tuning enables the pre-trained satellite-borne model to distinguish the similarity and difference between image data in the local data set by contrastive learning.

[0076] Specifically, in each round of the cooperative labeling fine-tuning process, the satellite terminal performs a data augmentation operation (such as random cropping, rotation, etc.) on the unlabeled image to generate a positive sample pair, and randomly samples a negative sample pair from other images. By using a contrastive loss function (such as an InfoNCE loss), the pre-trained satellite-borne model learns to make the features of the positive sample pair closer and the features of the negative sample pair farther.

[0077] This process enables the pre-trained satellite-borne model to extract feature representations from image data without labels, resulting in the initial satellite-borne model's fine-tuning parameters for this round.

[0078] The unsupervised contrastive fine-tuning method can make full use of unlabeled data to perform local learning on satellite terminals and update the local representation information of satellite-borne data (i.e., image data collected by satellite terminals), thereby improving the feature extraction capability of the pre-trained satellite-borne model for the satellite-borne data of the satellite terminal, and laying a foundation for subsequent classification prediction and other labeling tasks.

[0079] Step S13, the multiple satellite terminals send the fine-tuning parameters of the respective initial satellite-borne models to the ground station.

[0080] In this embodiment, after completing the unsupervised contrastive fine-tuning in step S12, the multiple satellite terminals send the fine-tuning parameters of the respective initial satellite-borne models to the ground station. This process is a key link in satellite-ground collaboration. The satellite terminals transmit the fine-tuning parameters of the initial satellite-borne models to the ground station as the parameters of the updated pre-trained satellite-borne model through the establishment of a communication link with the ground station. The fine-tuning parameters of the initial satellite-borne model contain the feature representations learned by the satellite terminals from the unlabeled data. By sending these fine-tuning parameters to the ground station, knowledge sharing between different satellite terminals can be achieved, providing data support for the generation of the aggregated satellite-borne model.

[0081] Step S14, the ground station aggregates the received fine-tuning parameters of the multiple initial satellite-borne models to obtain the fine-tuning parameters of the aggregated satellite-borne model for this round, and distributes the fine-tuning parameters of the aggregated satellite-borne model to the multiple satellite terminals.

[0082] In this embodiment, the ground station aggregates the received fine-tuning parameters of the multiple initial satellite-borne models to obtain the fine-tuning parameters of the aggregated satellite-borne model for this round, and distributes the obtained fine-tuning parameters of the aggregated satellite-borne model to the multiple satellite terminals. The aggregation process can use the average or weighted average method to fuse the fine-tuning parameters of the multiple initial satellite-borne models from different satellite terminals.

[0083] Specifically, by sharing the fine-tuning parameters of each satellite terminal, the aggregation process learns the local latent feature representations extracted from the unlabeled data by each satellite terminal, thereby obtaining the fine-tuning parameters of the aggregated satellite-borne model to improve the generalization capability of the initial pre-trained satellite-borne model. Further, the ground station distributes the fine-tuning parameters of the aggregated satellite-borne model to all satellite terminals for the next round of collaborative labeling fine-tuning. In this way, not only can the information loss caused by insufficient number of satellites be compensated, but also the overall performance of the model can be enhanced through collaborative learning.

[0084] Step S15, after the pre-trained satellite-borne model performs multiple rounds of collaborative annotation fine-tuning between the ground station and the plurality of satellite terminals, the plurality of satellite terminals obtains a first converged satellite-borne model, and the first converged satellite-borne model is a model parameter with a generalization knowledge representation.

[0085] In this embodiment, the plurality of satellite terminals and the ground station will perform a multiple-round collaborative annotation fine-tuning process, each round will obtain the fine-tuning parameters of the plurality of initial satellite-borne models corresponding to the plurality of satellite terminals, and then be transmitted to the ground station, and the fine-tuning parameters of the aggregated satellite-borne model are aggregated by the ground station, and each satellite terminal performs the next round of unsupervised contrastive fine-tuning on the received aggregated satellite-borne model to obtain the fine-tuning parameters of the new initial satellite-borne model... and so on. After multiple rounds of collaborative annotation fine-tuning, the plurality of satellite terminals receives the first converged satellite-borne model sent by the ground station. The first converged satellite-borne model refers to the model parameters that gradually tend to be stable after multiple rounds of unsupervised contrastive fine-tuning and parameter aggregation, and the model parameters have a generalization knowledge representation. The model at this stage can better adapt to the data characteristics of different satellites and has strong feature extraction capability and generalization capability. Through multiple rounds of collaborative annotation fine-tuning, the first converged satellite-borne model can share knowledge among different satellite terminals, thereby optimizing the performance in the global range. Finally, the converged model received by the satellite terminal provides a solid foundation for subsequent supervised fine-tuning.

[0086] Step S16, each of the plurality of satellite terminals uses the labeled data in the local data set to perform supervised fine-tuning on the first converged satellite-borne model, and obtains a collaborative annotation fine-tuned satellite-borne model of each of the plurality of satellite terminals. The collaborative annotation fine-tuned satellite-borne model of each satellite terminal is used for class prediction on the image data collected by the satellite terminal to obtain labeled image data.

[0087] In this embodiment, after each satellite terminal receives the first converged satellite-borne model sent by the ground station, each of the plurality of satellite terminals uses the labeled data in the local data set to perform supervised fine-tuning on the first converged satellite-borne model. Supervised fine-tuning is a process of further optimizing the first converged satellite-borne model using labeled data, which is a small amount of annotated image data. Through supervised fine-tuning, the satellite terminal combines the feature extraction capability of the converged model with the local labeled data, further adjusts the model parameters, so that the model can more accurately predict the class of image data, and the model can achieve higher performance.

[0088] And, since the local dataset of each satellite terminal is different, the labeled data in the local dataset of each satellite terminal is also different, and each satellite terminal uses different labeled data to further supervise the supervised fine-tuning of the first converged satellite model, so that the model finally obtained by each satellite terminal is also different, that is, the collaborative labeling fine-tuned satellite model finally obtained by each satellite terminal is different, and this personalized model can learn the feature representation of the labeled data of the satellite it is in, thereby adapting to the local labeling task of the satellite terminal.

[0089] Finally, the collaborative labeling fine-tuned satellite model obtained by each satellite terminal can accurately label the locally collected image data, that is, labeled image data can be obtained, without the need to recruit highly specialized data labelers for real-time collaboration, solving the problem of scarcity of satellite terminal label data, and at the same time, since the model has better ability to adapt to the specific task requirements of the satellite terminal it is in, the accuracy of labeling local image data can be improved, in addition, these image data carrying accurate labels can be used for model training, data analysis, and other tasks of the satellite terminal.

[0090] For example, in the labeling task of disaster map generation, the collaborative labeling fine-tuned satellite model of each satellite terminal can be used to specifically identify disasters in the task execution area of the satellite terminal it is in and label the collected disaster images to obtain a disaster map. Specifically, in the application scenario of disaster map generation, the fine-tuning process can use a small amount of labeled disaster images (such as flood inundation areas, fire disaster areas, etc.) and a large amount of unlabeled image data as the local dataset. Through the combination of unsupervised contrast fine-tuning and supervised fine-tuning, the satellite model can learn the feature representation of the disaster area and make high-precision class predictions on the image data collected by the satellite terminal after fine-tuning. For example, when a flood disaster occurs, the fine-tuned satellite model can quickly identify the flood inundation area and label it on the map to generate a detailed disaster map. This high-precision labeling capability significantly improves the efficiency of disaster response, helping rescue personnel quickly locate disaster areas, optimizing rescue routes, and reducing disaster losses. This process not only demonstrates the strong adaptability of the present scheme in the case of data label scarcity, but also highlights its efficiency and accuracy in disaster emergency response.

[0091] Through the technical solutions of the above embodiments, the satellite terminal can learn the feature representation in the labeled data in the case of labeled data scarcity, reducing the dependence on labeled data. Subsequent supervised fine-tuning further improves the performance of the model on specific tasks, and even in the case of labeled data scarcity, a high labeling accuracy can be achieved. This combination of unsupervised contrastive fine-tuning and supervised fine-tuning enables the model to better extract feature representations from image data and optimize them using a small amount of labeled data, thereby significantly improving the labeling accuracy. Furthermore, the multi-round collaborative fine-tuning and parameter aggregation enable the model to integrate the learning results of different satellite terminals, enhancing the generalization ability of the model. Through the sharing of knowledge representations, the model can better adapt to the data characteristics of different satellites. In addition, through the star-ground collaborative labeling fine-tuning, the model can adapt to the particularity of the satellite data. In summary, the present application effectively solves the technical problem of labeled data scarcity in satellite data, improves the labeling ability and overall performance of the model, and meets the needs of satellite terminals in practical applications.

[0092] Optionally, in combination with the above embodiments, an embodiment of the present application also provides another star-ground collaborative labeling fine-tuning method for data label scarcity, in which, after the step S11 of "the plurality of satellite terminals receiving the pre-trained satellite-borne model issued by the ground station", the step S12 further includes step S12-1:

[0093] Step S21, the plurality of satellite terminals respectively split the pre-trained satellite-borne model into an encoder network close to the input layer and a decoder network close to the output layer, and freeze the parameters of the decoder network.

[0094] In this embodiment, first, considering that in satellite image processing, the quality of feature representation of data directly determines the performance of the model, and the encoder can convert complex image data into more compact and robust feature representations through multi-layer feature extraction, these feature representations can capture key information and patterns in image data, thereby learning the basic features of each image data, such as edges, textures and shapes, etc., which can provide strong support for labeling tasks. Homogeneous, also considering that in the process of star-ground collaborative labeling fine-tuning, the communication resources between the satellite terminal and the ground station are limited, and the transmission amount of model parameters will directly affect the communication, based on these two points, the present application proposes the technical concept of splitting the pre-trained satellite-borne model into an encoder network containing learning features close to the input layer and a decoder network close to the output layer for decoding the output of the downstream task, and only fine-tuning the encoder network during the unsupervised contrastive fine-tuning process using unlabeled data.

[0095] The decoder network does not participate in the unsupervised contrastive fine-tuning process, and thus the parameters of the decoder network are frozen, meaning that the parameters of the decoder network of all satellite terminals do not change during the backpropagation update in the entire unsupervised contrastive fine-tuning process.

[0096] The step S12 includes a step S12-1 of, in each round of the collaborative labeling fine-tuning process, using the unlabeled data in the local data set to perform unsupervised contrastive fine-tuning on the encoder network in the pre-trained satellite-borne model, to obtain the fine-tuning parameters of the initial satellite-borne model in the round, the fine-tuning parameters of the initial satellite-borne model being the parameters of the fine-tuned encoder network.

[0097] In this embodiment, in the process of each round of collaborative labeling fine-tuning, the encoder network can extract feature representations in the expressionless data without labels, and obtain the fine-tuning parameters of the initial satellite-borne model in the round, i.e., the parameters of the fine-tuned encoder network. Through fine-tuning of the encoder network, the model can learn more robust feature representations on unlabeled data, providing a better foundation for subsequent labeling tasks.

[0098] The first converged satellite-borne model includes the parameters of the converged encoder network and the parameters of the frozen decoder network.

[0099] In this embodiment, after multiple rounds of collaborative labeling fine-tuning between the satellite terminals and the ground station, the satellite terminals receive the parameters of the converged encoder network, and obtain the first converged satellite-borne model based on the parameters of the converged encoder network and the parameters of the frozen decoder network.

[0100] That is, in the unsupervised contrastive fine-tuning phase, although only the parameters of the encoder network are updated, the first converged satellite-borne model ultimately obtained still includes the complete encoder network and decoder network. In the subsequent supervised fine-tuning phase (step S16), the satellite terminals will use the local labeled data to further optimize the entire first converged satellite-borne model (including the encoder and the decoder) to improve the performance of the satellite-borne model in specific class prediction and other labeling tasks.

[0101] Through the technical solutions of the above embodiments, the pre-trained satellite-borne model is split into an encoder and a decoder network, and the parameters of the decoder network are frozen, so that the satellite terminal can focus on improving the extraction capability of the feature representation of the encoder network in the unsupervised contrast fine-tuning phase. This design enables the model to make full use of the unlabeled data to learn more robust feature representations, thereby enabling the satellite-borne model to perform better in subsequent labeling tasks. Moreover, freezing the parameters of the decoder network reduces the computational overhead in the unsupervised contrast fine-tuning phase, because only the parameters of the encoder network need to be updated. Meanwhile, during the parameter transmission and aggregation process, transmitting only the parameters of the encoder network also reduces the communication load and improves the efficiency of the system.

[0102] Optionally, in combination with the above embodiments, an embodiment of the present application also provides another satellite-ground collaborative labeling fine-tuning method for data label scarcity, in which the step S13 of "sending the fine-tuning parameters of the initial satellite-borne model of each satellite terminal to the ground station" specifically includes steps S13-1 to S13-2:

[0103] Step S13-1: The central server deployed by the ground station randomly selects a plurality of satellite terminals from the plurality of satellite terminals that can establish a connection to participate in the collaborative aggregation.

[0104] In this embodiment, during the process of collaborative labeling fine-tuning, it is considered that the communication between the satellite and the ground station is limited by the satellite orbit and the coverage range of the ground station, that is, the satellite has a long revisit period and can only establish a communication connection with the ground station when the satellite orbit passes over the ground station, so the communication resources between the satellite and the ground station are limited, and the long revisit period exacerbates the difficulty of the collaborative aggregation process of the model parameters.

[0105] Based on this, during the process of each round of collaborative labeling fine-tuning, although each satellite terminal will perform local unsupervised contrast fine-tuning, which satellite terminals need to transmit the fine-tuning parameters of the initial satellite-borne model obtained after fine-tuning is determined by the ground station. Specifically, the central server deployed by the ground station needs to randomly select a plurality of satellite terminals from the satellite terminals that can establish a connection with the ground station to participate in the collaborative aggregation of this round. The fine-tuning parameters of the initial satellite-borne model of these selected satellite terminals will be sent to the ground station within the communication window and participate in the aggregation process of the ground station, so as to ensure the flexibility of the aggregation process and maximize the use of available communication resources in the case of limited number of satellites.

[0106] Step S13-2: The plurality of satellite terminals participating in the collaborative aggregation send the fine-tuning parameters of the initial satellite-borne model of each satellite terminal to the ground station.

[0107] In this embodiment, after randomly selecting multiple satellite terminals participating in collaborative aggregation, these satellite terminals send the fine-tuning parameters of their respective initial on-board models to the ground station. These fine-tuning parameters are the encoder network parameters obtained by the satellite terminals after unsupervised contrastive fine-tuning on local unlabeled data. Sending these fine-tuning parameters allows the ground station to aggregate the learning results of different satellite terminals and generate more optimized global model parameters, i.e., the fine-tuning parameters of the aggregated on-board model in this round.

[0108] Through the technical solutions of the above embodiments, a random selection mechanism and a flexible communication strategy are introduced, further optimizing the process of satellite-ground collaborative annotation fine-tuning. Not only can it effectively deal with the limitations of satellite communication, but also can achieve efficient model updating under limited communication resources. By randomly selecting currently connectable satellite terminals to participate in aggregation, the central server can maximize the use of available resources in each communication, while reducing communication overhead and improving the overall performance of the system. This strategy is particularly suitable for scenarios where satellite data labels are scarce and communication is limited, and can significantly improve the efficiency and effectiveness of satellite-ground collaborative fine-tuning.

[0109] Alternatively, in combination with the above embodiments, an embodiment of the present application also provides another satellite-ground collaborative annotation fine-tuning method for data label scarcity, in which the step S14 of "the ground station aggregates the received fine-tuning parameters of multiple initial on-board models to obtain the fine-tuning parameters of the aggregated on-board model in this round" specifically includes steps S14-1 to S14-2:

[0110] Step S14-1, when the number of satellite terminals participating in collaborative aggregation is less than a preset number threshold, the ground station selects the fine-tuning parameters of the aggregated on-board model most similar to the fine-tuning parameters of the multiple initial on-board models sent by the multiple satellite terminals participating in collaborative aggregation from the fine-tuning parameters of the multiple aggregated on-board models obtained in the previous N rounds of collaborative annotation fine-tuning process, N≥1.

[0111] In this embodiment, during the collaborative annotation fine-tuning process, when the number of satellite terminals participating in collaborative aggregation is less than a preset number threshold, the central server of the ground station selects the fine-tuning parameters of the aggregated on-board model most similar to the fine-tuning parameters of the initial on-board model sent by the current satellite terminal participating in collaborative aggregation from the fine-tuning parameters of the multiple aggregated on-board models obtained in the previous N rounds of collaborative annotation fine-tuning process (N≥1). Thus, the lack of diversity and richness of update information (i.e., fewer fine-tuning parameters of initial on-board models) due to insufficient number of participating satellites is compensated for, and the information richness and diversity of the current aggregation process are enhanced by introducing historical aggregated parameters (i.e., the most similar fine-tuning parameters of the aggregated on-board model in history).

[0112] Specifically, the central server of the ground station calculates the similarity (for example, by cosine similarity calculation) between the fine-tuning parameters of the current received initial satellite-borne model and the fine-tuning parameters of the plurality of aggregated satellite-borne models obtained in the previous N rounds of collaborative labeling fine-tuning, and selects the most similar parameters as compensation. This method not only makes full use of historical information, but also optimizes the update of the aggregated satellite-borne model under the condition that the number of satellites participating in collaborative aggregation is small due to the long revisit period of the satellite.

[0113] The preset number threshold can be set according to actual conditions, such as the length of the communication window between the satellite terminal and the ground station, the satellite orbit characteristics, the task requirements of the collaborative labeling fine-tuning process, the performance target of the final desired satellite-borne model, and / or historical data and experience.

[0114] Step S14-2, the ground station aggregates the fine-tuning parameters of the plurality of initial satellite-borne models sent by the plurality of satellite terminals participating in collaborative aggregation and the fine-tuning parameters of the most similar aggregated satellite-borne model to obtain the fine-tuning parameters of the aggregated satellite-borne model of this round.

[0115] Through the technical solutions of the above embodiments, the fine-tuning parameters of the historical aggregated satellite-borne model are introduced as a compensation mechanism, further optimizing the collaborative labeling fine-tuning process. Not only can it enhance the information richness and diversity of the aggregation process, but also can optimize the update of the global model under the condition that the number of satellites is limited, thereby realizing more efficient model updating in fewer communication times and improving the overall performance of the satellite-ground collaborative labeling fine-tuning system facing data label scarcity.

[0116] Optionally, in combination with the above embodiments, an embodiment of the present application also provides another satellite-ground collaborative labeling fine-tuning method facing data label scarcity, in which the step S15 "the plurality of satellite terminals obtain a first converged satellite-borne model" specifically includes steps S15-1 to S15-2:

[0117] Step S15-1, after the satellite-borne model is subjected to multiple rounds of collaborative labeling fine-tuning between the ground station and the plurality of satellite terminals, the plurality of satellite terminals obtain the parameters of the converged encoder network issued by the ground station.

[0118] In this embodiment, during the collaborative labeling fine-tuning process, after multiple rounds of collaborative labeling fine-tuning, the plurality of satellite terminals will receive the parameters of the converged encoder network obtained through multiple rounds of collaborative aggregation from the ground station, indicating that the fine-tuning optimization of the satellite-borne model in the global range is completed.

[0119] Specifically, the satellite terminals gradually optimize the parameters of the encoder network in the multiple rounds of unsupervised contrast fine-tuning and parameter aggregation process, so that the encoder network can extract more robust and more generalized feature representations to more accurately perform class prediction and other labeling tasks. Finally, the ground station will issue the converged parameters of the encoder network to each satellite terminal, and the converged parameters of the encoder network represent the feature extraction capability of the globally optimized on-board model, providing a high-quality feature basis for subsequent labeling tasks.

[0120] Step S15-2, the plurality of satellite terminals splices the received converged parameters of the encoder network with the frozen parameters of the decoder network to obtain the first converged on-board model.

[0121] In this embodiment, after receiving the converged parameters of the encoder network, the plurality of satellite terminals splices these parameters with the locally frozen parameters of the decoder network to obtain a complete first converged on-board model. The parameters of the decoder network remain frozen during the previous fine-tuning process, so the parameters of the initialization or pre-training stage are directly used during splicing, thereby forming a complete on-board model that can be used for actual labeling tasks. In this way, the satellite terminal can locally use the optimized model to accurately predict the class of image data, thereby obtaining labeled image data.

[0122] Optionally, in combination with the above embodiment, an embodiment of the present application also provides another star-ground collaborative labeling fine-tuning method for data label scarcity, in which the method further includes steps S31 and S32 before the collaborative labeling fine-tuning process:

[0123] Step S31, setting initialization parameters and model structure to obtain the pre-trained on-board model.

[0124] In this embodiment, before the collaborative labeling fine-tuning, the initialization parameters need to be set and the appropriate model structure (such as manually selecting a specific model) needs to be selected to obtain the pre-trained on-board model.

[0125] The initialization parameters include the initial weights and biases of the model, which are randomly initialized. Then, a suitable model structure, such as a lightweight convolutional neural network (CNN) or a Transformer, is selected to obtain a pre-trained model. The pre-trained on-board model is the basis for the entire collaborative fine-tuning process, providing a starting point for subsequent unsupervised contrast fine-tuning and supervised fine-tuning. By reasonably setting the initialization parameters and selecting the model structure, the model can still effectively learn and optimize in the case of label scarcity.

[0126] Step S32, set the first related information participating in the collaborative process, the first related information includes: the number of the plurality of satellite terminals, the frequency of each satellite terminal participating in collaborative aggregation, the label scarcity of the local data set of each satellite terminal and the label scarcity of the local data set, the deviation operator of each satellite terminal and the number of measurement fine-tuning rounds.

[0127] Wherein, the label scarcity is used to represent the proportion of labeled data in the satellite terminal, and the label deviation degree is used to represent the distribution of labeled data in the plurality of satellite terminals; the deviation operator represents the operation operator of the labeled data for simulating different label deviation degrees on the plurality of satellite terminals; and the number of measurement fine-tuning rounds represents the number of times of collaborative aggregation.

[0128] One of the purposes of the present scheme is to evaluate the ability of each satellite terminal to maintain the accuracy of the on-board model under different local data sets, considering that the data label scarcity and the label deviation degree of the satellite have a great influence on the final fine-tuning accuracy. Therefore, the label scarcity and the label deviation degree of the data label of the satellite terminal are defined in this paper.

[0129] Specifically, in the present embodiment, before the collaborative labeling fine-tuning starts, the first related information participating in the collaborative process also needs to be set, and these first related information includes:

[0130] 1. The number of satellite terminals: indicating how many satellites participate in the process of collaborative labeling fine-tuning, and the number of satellites will affect the diversity of data and the speed of model updating.

[0131] 2. The frequency of each satellite terminal participating in collaborative aggregation: by setting the frequency of each satellite participating in collaborative aggregation, for example, aggregating once every 6 hours or every 12 hours, the frequency can be adjusted according to the communication window between the satellite terminal and the ground station and / or the requirements of the labeling task.

[0132] 3. Label scarcity of local data set: define the proportion of labeled data in each satellite terminal. For example, the label scarcity of satellite terminal A is 10%, which means that 10% of its local data set is labeled. This indicator reflects the data labeling situation of the image data contained in the local data set of the satellite terminal, and has an important influence on the fine-tuning strategy of the model.

[0133] 4. Label scarcity of local data set: indicating the distribution of labeled data among different satellite terminals. For example, some satellite terminals may have more labeled data, while others may have almost none. Different label deviation degrees affect the balance of the model in the global optimization process.

[0134] 5. Deviation operator: An operation operator for simulating different degrees of label deviation of the labeled data on multiple satellite terminals. These operators can help more realistically simulate the actual situation of satellite data, thereby optimizing the robustness of the model.

[0135] 6. Measure the number of fine-tuning rounds: Set the number of times of collaborative aggregation, that is, how many iterations the entire fine-tuning process needs to perform, and the setting of measuring the number of fine-tuning rounds needs to be adjusted according to the convergence of the model and the task requirements.

[0136] For example, before performing collaborative annotation fine-tuning, a series of initialization settings need to be performed. First, the ground station needs to set the initialization parameters and select a suitable model structure to generate a pre-trained satellite model. For example, a lightweight CNN model can be selected and its parameters are randomly initialized. Next, the first relevant information participating in the collaborative process needs to be set, including the number of satellite terminals, the frequency of each satellite terminal participating in collaborative aggregation, the degree of label scarcity, the degree of label deviation, the deviation operator, and the number of measurement fine-tuning rounds. For example, assuming that there are 5 satellites in the satellite constellation, the label scarcity degrees of each satellite are 10%, 5%, 15%, 10%, and 5%, respectively, the label deviation degree is simulated by the deviation operator, the frequency of each satellite terminal participating in collaborative aggregation is 6 hours / time, 7 hours / time, 10 hours / time, 2 hours / time, and 5 hours / time, respectively, and the number of measurement fine-tuning rounds is 10. Then each satellite terminal corresponds to a satellite, and each satellite participates in collaborative aggregation every time it reaches the corresponding time, and the entire collaborative annotation fine-tuning process is performed for 10 iterations. Through these settings, the pre-trained satellite model can be better learned and optimized in the case of label scarcity, thereby improving the efficiency and effectiveness of collaborative annotation fine-tuning and the overall performance of the satellite model.

[0137] Through the above embodiment, before performing collaborative annotation fine-tuning, reasonable initialization parameters and appropriate model structures can be set to provide an efficient starting point for the collaborative annotation fine-tuning process. The pre-trained satellite model can better learn and optimize in the case of label scarcity, thereby improving the efficiency and effectiveness of collaborative annotation fine-tuning and the overall performance of the satellite model.

[0138] Optionally, in combination with the above embodiment, an embodiment of the present application also provides another satellite-ground collaborative annotation fine-tuning method for data label scarcity, in which, after the step S16 of "obtaining the collaborative annotation fine-tuned satellite model of each of the plurality of satellite terminals", the method further comprises steps S41 to S47:

[0139] Step S41, obtaining the true label of the first test data set, wherein the first test data set comprises first unlabeled data and first labeled data.

[0140] In this embodiment, after obtaining the spaceborne model fine-tuned by multiple rounds of collaborative annotation, in order to evaluate the annotation performance of the model, first, the pre-prepared first test data set is used for evaluation, the first test data set contains multiple image data, which are divided into two types of first unlabeled data and first labeled data, the first unlabeled data refers to the unlabeled data in the first test data set, and the first labeled data refers to the labeled data in the first test data set.

[0141] The first test data set is used to verify the annotation accuracy of the spaceborne model fine-tuned by collaborative annotation. The true label refers to the correct class of each image data in the first test data set, which is pre-annotated by artificial annotation or obtained by other reliable methods.

[0142] For example, the first test data set contains 100 labeled image data (first labeled data) and 900 unlabeled image data (first unlabeled data), and the true classes of these image data are known for subsequent performance evaluation.

[0143] Step S42, annotating the first unlabeled data by the spaceborne model fine-tuned by collaborative annotation to obtain the first unlabeled data after labeling.

[0144] In this embodiment, the spaceborne model fine-tuned by collaborative annotation is used to perform the annotation task of class prediction on the first unlabeled data in the first test data set. In this process, the spaceborne model fine-tuned by collaborative annotation will predict the class label for each first unlabeled data according to its learned feature representation and classification ability. For example, assuming that there are 900 unlabeled images in the first test data set, the model will generate predicted labels for the 900 images, thereby obtaining the first unlabeled data after labeling.

[0145] Step S43, calculating the annotation accuracy of the spaceborne model fine-tuned by collaborative annotation according to the first unlabeled data after labeling and the true labels of the first unlabeled data in the true labels of the first test data set, the annotation accuracy of the spaceborne model fine-tuned by collaborative annotation is: the proportion of the first unlabeled data annotated correctly in the first unlabeled data after labeling in all first unlabeled data.

[0146] In this embodiment, the labeling accuracy of the co-labeled fine-tuned spaceborne model is calculated according to the first unlabeled data after labeling and the true labels of the first unlabeled data in the first test data set. The labeling accuracy refers to the proportion of the data labeled correctly in the total unlabeled data. For example, if the model correctly labels 810 out of 900 unlabeled images, the labeling accuracy is 810 / 900 = 90%. This index reflects the labeling performance of the co-labeled fine-tuned spaceborne model on unlabeled data, and is an important reference for evaluating the generalization ability of the model.

[0147] Step S44, mixing the first unlabeled data after labeling and the first labeled data to obtain mixed labeled data.

[0148] In this embodiment, the first unlabeled data after labeling and the first labeled data in the first test data set are mixed to obtain mixed labeled data. The purpose of this step is to combine the first unlabeled data labeled by the model with the existing first labeled data to form a more abundant labeled data set. For example, 900 labeled unlabeled images are mixed with 100 labeled images to obtain a mixed labeled data set containing 1000 images. The mixed data set can be used to further optimize the model.

[0149] Step S45, re-fine-tuning the co-labeled fine-tuned spaceborne model through the mixed labeled data to obtain a first spaceborne model, wherein the first spaceborne model is the co-labeled fine-tuned spaceborne model re-fine-tuned through the mixed labeled data.

[0150] In this embodiment, the co-labeled fine-tuned spaceborne model is re-fine-tuned using mixed labeled data, and the obtained model is called the first spaceborne model. In this process, the performance of the model is further optimized by introducing the first unlabeled data labeled by the co-labeled fine-tuned spaceborne model. The mixed labeled data not only contains the original first labeled data, but also adds the first unlabeled data labeled by the model, thereby providing more learning samples for the model. By fine-tuning on the mixed labeled data, the co-labeled fine-tuned spaceborne model can learn more rich feature representations and improve its labeling ability.

[0151] Step S46, calculating the system accuracy of the first spaceborne model through a second test data set; the system accuracy of the first spaceborne model is the correct rate that the first spaceborne model can achieve after labeling the second test data set; the first spaceborne model is re-fine-tuned using mixed labeled data; the second test data set contains second unlabeled data and second labeled data.

[0152] In this embodiment, another test data set, i.e., second test data, is further introduced, and the system accuracy of the first spaceborne model is calculated through the second test data set. The second test data set contains second unlabeled data and second labeled data, which is used to evaluate the performance of the first spaceborne model on unseen data. The system accuracy refers to the correct rate that the first spaceborne model can achieve after labeling the second test data set. For example, if there are 1000 images in the second test data set, and the model correctly labels 920 of them, then the system accuracy is 92%. This indicator reflects the overall performance of the model on new data, and is a key indicator for evaluating the generalization ability and practical application value of the model.

[0153] In step S47, the performance improvement of the co-annotation fine-tuned spaceborne model is evaluated based on the labeling accuracy of the co-annotation fine-tuned spaceborne model and the system accuracy of the first spaceborne model.

[0154] In this embodiment, the performance improvement of the co-annotation fine-tuned spaceborne model is evaluated based on the labeling accuracy of the co-annotation fine-tuned spaceborne model and the system accuracy of the first spaceborne model. Here, the performance improvement refers to the improvement of the labeling accuracy of image data. By analyzing the labeling accuracy and the system accuracy, the improvement effect of the model performance can be evaluated. If the system accuracy is significantly higher than the labeling accuracy, it means that the fine-tuning process effectively improves the performance of the model, making it more suitable for actual labeling tasks.

[0155] Through the technical solutions of the above embodiments, for the co-annotation fine-tuned spaceborne model obtained, the labeling ability of the model on unlabeled data can be intuitively evaluated by calculating the labeling accuracy. A high labeling accuracy indicates that the model can effectively learn from unlabeled data and improve its generalization ability. Then, the model can be fine-tuned again using mixed labeled data, which can further optimize the performance of the model. The mixed labeled data not only contains the original labeled data, but also adds the unlabeled data labeled by the model, thereby providing more learning samples for the model and improving its labeling ability. Furthermore, by calculating the system accuracy on the second test data set, the overall performance of the model on unseen data can be evaluated. A high system accuracy indicates that the model has strong generalization ability and can maintain high labeling accuracy on new data. Finally, by analyzing the labeling accuracy and the system accuracy, the improvement effect of the performance of the co-annotation fine-tuned spaceborne model can be intuitively evaluated.

[0156] Optionally, in combination with the above-mentioned embodiments, an embodiment of the present application further provides another satellite-ground collaborative labeling fine-tuning method for data label scarcity, in which, before the step S47 of "evaluating the performance improvement of the collaborative labeling fine-tuned satellite-borne model", the method further comprises steps S51 to S513, and the step S47 specifically comprises a step S47-1:

[0157] In step S51, initialization parameters and model structures are set to obtain a to-be-fine-tuned satellite-borne model.

[0158] In this embodiment, before evaluating the performance improvement of the collaborative labeling fine-tuned satellite-borne model, a satellite-borne data label scarcity simulation measurement scheme needs to be formulated to measure the ability of the satellite-borne model to maintain labeling accuracy under different label data, therefore, initialization parameters need to be set and a suitable model structure needs to be selected to obtain a to-be-fine-tuned satellite-borne model.

[0159] The initialization parameters include the initial weights and biases of the model, which are usually randomly initialized or obtained from a pre-trained model. A suitable model structure, a lightweight convolutional neural network (CNN) or a Transformer, is selected.

[0160] In step S52, second related information participating in the collaborative process is set, and the second related information includes: the number of multiple satellite terminals, the frequency of each satellite terminal participating in the collaborative process, the label scarcity degree and the label deviation degree of the local training data set of each satellite terminal, the deviation operator of each satellite terminal and the measurement fine-tuning round number.

[0161] Before the collaborative process, the second related information needs to be set, which can refer to the setting process of the first related information.

[0162] In step S53, the multiple satellite terminals participating in the collaborative process each use third unlabeled data in the local training data set to perform unsupervised contrastive fine-tuning on the to-be-fine-tuned satellite-borne model to obtain fine-tuning parameters of the current fine-tuned satellite-borne model.

[0163] In this embodiment, the multiple satellite terminals participating in the collaborative process each use third unlabeled data in the local training data set to perform unsupervised contrastive fine-tuning on the satellite-borne model that has not been fine-tuned by the collaborative labeling. Through unsupervised contrastive learning, the model can extract feature representations from the point unlabeled data without labels, and obtain fine-tuning parameters of the current fine-tuned satellite-borne model.

[0164] Step S54, the central server of the ground station randomly selects a plurality of satellite terminals from the plurality of satellite terminals participating in the collaborative process as satellite terminals participating in collaborative aggregation, and aggregates the fine-tuning parameters of the plurality of current fine-tuned satellite models obtained through unsupervised contrast fine-tuning to obtain fine-tuning parameters of a current aggregated satellite model.

[0165] In this embodiment, the central server of the ground station randomly selects a plurality of satellite terminals from the plurality of satellite terminals participating in the collaborative process as satellite terminals participating in collaborative aggregation, and aggregates the fine-tuning parameters of the plurality of current fine-tuned satellite models obtained through unsupervised contrast fine-tuning to obtain fine-tuning parameters of a current aggregated satellite model. The aggregation process usually uses a simple average or weighted average method to fuse parameters from different sources. In this way, the central server can integrate the learning achievements of multiple satellite terminals to generate more optimized global model parameters.

[0166] Step S55, the ground station sends the fine-tuning parameters of the current aggregated satellite model to the plurality of satellite terminals for the next round of collaborative process.

[0167] In this embodiment, the ground station sends the fine-tuning parameters of the current aggregated satellite model to the plurality of satellite terminals for the next round of collaborative process, ensuring that different satellite terminals can further fine-tune based on the latest global model parameters.

[0168] Step S56, after a plurality of rounds of collaborative process, the plurality of satellite terminals participating in the collaborative process each obtain a second converged satellite model.

[0169] In this embodiment, after a plurality of rounds of collaborative process, the plurality of satellite terminals participating in the collaborative process each obtain a second converged satellite model. The second converged satellite model is a model obtained after multiple rounds of unsupervised contrast fine-tuning and parameter aggregation, and its model parameters have stabilized, with good feature extraction ability and generalization ability.

[0170] Step S57, each of the plurality of satellite terminals participating in the collaborative process locally fine-tunes the second converged satellite model using labeled data in the local training data set to obtain a fine-tuned satellite model.

[0171] In this embodiment, each of the plurality of satellite terminals participating in the collaborative process locally fine-tunes the second converged satellite model using the third labeled data in the local training data set to obtain a fine-tuned satellite model. The purpose of local fine-tuning is to further optimize the performance of the model using a small amount of labeled data, so that the satellite model can better adapt to specific labeling tasks in the satellite terminal.

[0172] Step S58, the plurality of satellite terminals participating in the collaborative process perform class prediction on the third unlabeled data in the local training data set through the fine-tuned satellite-borne model, and label the third unlabeled data based on the results of the class prediction to obtain labeled third unlabeled data.

[0173] In this embodiment, the purpose of the prediction process of the fine-tuned satellite-borne model on the unlabeled data is to generate labeled data for subsequent calculation of labeling accuracy.

[0174] Step S59, determine the labeling accuracy of the fine-tuned satellite-borne model according to the true labels of the third unlabeled data in the local training data set and the labeled third unlabeled data.

[0175] In this embodiment, the labeling accuracy of the fine-tuned satellite-borne model is determined according to the true labels of the third unlabeled data in the local training data set and the labeled third unlabeled data. The labeling accuracy of the fine-tuned satellite-borne model refers to the proportion of the third unlabeled data that is labeled correctly among all the third unlabeled data.

[0176] Step S510, the plurality of satellite terminals participating in the collaborative process each use the labeled third unlabeled data in the local training data set and the third labeled data in the local training data set to supervise the fine-tuning of the fine-tuned satellite-borne model to obtain a final fine-tuned satellite-borne model.

[0177] In this embodiment, the use of labeled unlabeled data and labeled data to form hybrid labeled data further optimizes the performance of the model, making it better adapt to specific labeling tasks.

[0178] Step S511, the plurality of satellite terminals participating in the collaborative process each use the third test data set to test the accuracy of the final fine-tuned satellite-borne model to obtain the system accuracy of the final fine-tuned satellite-borne model.

[0179] In this embodiment, the plurality of satellite terminals participating in the collaborative process each use the local third test data set to test the accuracy of the final fine-tuned satellite-borne model to obtain the system accuracy of the final fine-tuned satellite-borne model. The system accuracy of the final fine-tuned satellite-borne model refers to the accuracy that the final fine-tuned satellite-borne model can achieve after labeling the third test data set, reflecting the comprehensive performance of the final fine-tuned satellite-borne model on the mixed labeled data under the current second related information setting after the collaborative process.

[0180] Step S512, reset the label scarcity and the label deviation in the second related information, and repeat the above steps to obtain the corresponding relationship between the labeling accuracy and the system accuracy of the final fine-tuned satellite-borne model under different label scarcity and different label deviation.

[0181] In this embodiment, the label scarcity and the label deviation in the second related information participating in the collaborative process are reset, and the processes of steps S53 to S511 are repeated to obtain the corresponding relationship between the labeling accuracy and the system accuracy of the final fine-tuned satellite-borne model under different label scarcity and different label deviation. Through multiple tests, the performance change of the satellite-borne model after the collaborative process under different label scarcity and label deviation can be analyzed, which provides data support for the performance improvement of the subsequent collaborative labeling fine-tuning process optimization model.

[0182] The performance improvement of the collaborative labeling fine-tuned satellite-borne model in step S47 includes: step S47-1, determining the performance improvement degree of the collaborative labeling fine-tuned satellite-borne model based on the corresponding relationship, the labeling accuracy of the collaborative labeling fine-tuned satellite-borne model and the system accuracy of the collaborative labeling fine-tuned satellite-borne model.

[0183] In this embodiment, the performance improvement degree of the collaborative labeling fine-tuned satellite-borne model is determined based on the above corresponding relationship, the labeling accuracy and the system accuracy of the collaborative labeling fine-tuned satellite-borne model. By comparing the labeling accuracy and the system accuracy under different conditions, the performance improvement effect of the model through the collaborative labeling fine-tuning process in the foregoing can be intuitively evaluated, and the accuracy of the model in actual class prediction and other labeling tasks is ensured to be higher.

[0184] Through the above various embodiments, the core invention points and advantages of the present application are:

[0185] 1. A fine-tuning scheme for a satellite-borne model based on an encoder-decoder combined scheme based on a small amount of label guidance is designed. By freezing the parameters of the decoder network of the satellite terminal model and only updating the parameters of the encoder network of the satellite-borne model, about 20% of the fine-tuning time is saved, the system overhead required for fine-tuning is reduced, and the operation efficiency of the satellite-ground collaborative system is greatly improved.

[0186] 2. In the satellite-ground collaborative aggregation process, only the model parameters of the encoder network are shared, i.e., the satellite terminal satellite-borne data representation is learned, and the decoder network does not need to participate in the coordination aggregation process, thereby saving more than 10% of the data transmission amount in the transmission process and reducing the communication load overhead of the satellite-ground link.

[0187] 3. The data label scarcity and label deviation of the satellite terminal in the on-orbit environment are simulated, and the influence of the label scarcity and label deviation on the performance of the satellite model is further explored. The performance improvement of the satellite model after the subsequent cooperative labeling fine-tuning is evaluated through the two indicators. Based on the same inventive concept, another embodiment of the present application also provides a satellite-ground cooperative labeling fine-tuning system for satellite data label scarcity, which comprises a plurality of satellite terminals and a ground station:

[0188] The plurality of satellite terminals are configured to receive the pre-trained satellite model issued by the ground station, wherein the pre-trained satellite model is obtained by initializing the parameters of the to-be-fine-tuned model pre-trained by the ground station, and the to-be-fine-tuned model has the function of classifying the image data collected by any satellite terminal;

[0189] In each round of cooperative labeling fine-tuning, the plurality of satellite terminals are configured to respectively use the unlabeled data in the local data set to perform unsupervised contrast fine-tuning on the pre-trained satellite model to obtain the fine-tuning parameters of the initial satellite model in this round;

[0190] The plurality of satellite terminals are configured to send the fine-tuning parameters of the respective initial satellite models to the ground station;

[0191] The ground station is configured to aggregate the fine-tuning parameters of the plurality of initial satellite models received to obtain the fine-tuning parameters of the aggregated satellite model in this round, and issue the fine-tuning parameters of the aggregated satellite model to the plurality of satellite terminals;

[0192] After the pre-trained satellite model is subjected to multiple rounds of cooperative labeling fine-tuning between the ground station and the plurality of satellite terminals, the plurality of satellite terminals are configured to receive a first converged satellite model, wherein the first converged satellite model is a model parameter with generalization knowledge representation;

[0193] The plurality of satellite terminals are configured to respectively use the labeled data in the local data set to perform supervised fine-tuning on the first converged satellite model to obtain the cooperative labeling fine-tuned satellite model of each satellite terminal, and the cooperative labeling fine-tuned satellite model of each satellite terminal is used for classifying the image data collected by the satellite terminal to obtain labeled image data.

[0194] Optionally, it further comprises:

[0195] The plurality of satellite terminals are configured to, after receiving the pre-trained satellite model issued by the ground station, respectively split the pre-trained satellite model into an encoder network close to the input layer and a decoder network close to the output layer, and freeze the parameters of the decoder network;

[0196] The plurality of satellite terminals are configured to respectively utilize the unlabeled data in the local data set to perform unsupervised contrastive fine-tuning on the pre-trained satellite-borne model in each round of the collaborative labeling fine-tuning process to obtain fine-tuning parameters of an initial satellite-borne model of the round, including: in each round of the collaborative labeling fine-tuning process, the plurality of satellite terminals are respectively configured to utilize the unlabeled data in the local data set to perform unsupervised contrastive fine-tuning on an encoder network in the pre-trained satellite-borne model to obtain the fine-tuning parameters of the initial satellite-borne model of the round, the fine-tuning parameters of the initial satellite-borne model being parameters of the fine-tuned encoder network.

[0197] The first converged satellite-borne model includes parameters of a converged encoder network and parameters of the frozen decoder network.

[0198] Optionally, the plurality of satellite terminals are configured to send the fine-tuning parameters of the respective initial satellite-borne models to the ground station, including:

[0199] The ground station is configured to randomly select a plurality of satellite terminals from a plurality of connectable satellite terminals included in the plurality of satellite terminals as satellite terminals participating in collaborative aggregation by a deployed central server.

[0200] The plurality of satellite terminals participating in collaborative aggregation are configured to send the fine-tuning parameters of the respective initial satellite-borne models to the ground station.

[0201] Optionally, the ground station aggregates the received fine-tuning parameters of the plurality of initial satellite-borne models to obtain fine-tuning parameters of an aggregated satellite-borne model of the round, including:

[0202] When the number of satellite terminals participating in collaborative aggregation is less than a preset number threshold, the ground station is configured to select, from fine-tuning parameters of a plurality of aggregated satellite-borne models obtained in the previous N rounds of the collaborative labeling fine-tuning process, fine-tuning parameters of an aggregated satellite-borne model that is most similar to the fine-tuning parameters of the plurality of initial satellite-borne models sent by the plurality of satellite terminals participating in collaborative aggregation to the ground station, N≥1.

[0203] The ground station is configured to aggregate the received fine-tuning parameters of the plurality of initial satellite-borne models sent by the plurality of satellite terminals participating in collaborative aggregation and the most similar fine-tuning parameters of the aggregated satellite-borne model to obtain fine-tuning parameters of an aggregated satellite-borne model of the round.

[0204] The plurality of satellite terminals are configured to obtain a first converged satellite-borne model, including:

[0205] The plurality of satellite terminals are configured to obtain the parameters of the converged encoder network issued by the ground station after the satellite-borne model is subjected to the multiple rounds of collaborative labeling fine-tuning between the ground station and the plurality of satellite terminals.

[0206] The plurality of satellite terminals are configured to splice the received converged encoder network parameters and the frozen decoder network parameters to obtain the first converged satellite-borne model.

[0207] The system further comprises:

[0208] The ground station is configured to set initialization parameters and model structures to obtain the pre-trained satellite-borne model before performing the collaborative labeling fine-tuning process.

[0209] The ground station is configured to set first related information participating in the collaborative process, the first related information comprising: the number of the plurality of satellite terminals, the frequency of each of the plurality of satellite terminals participating in the collaborative aggregation, the label scarcity degree of the local data set of each of the plurality of satellite terminals and the label scarcity degree of the local data set, the deviation operator of each of the plurality of satellite terminals and the number of measurement fine-tuning rounds.

[0210] The label scarcity degree is used to represent the proportion of labeled data in the satellite terminal, and the label deviation degree is used to represent the distribution of labeled data in the plurality of satellite terminals; the deviation operator represents an operation operator for simulating labeled data with different label deviation degrees on the plurality of satellite terminals; and the number of measurement fine-tuning rounds represents the number of collaborative aggregations.

[0211] Optionally, the system further comprises:

[0212] The plurality of satellite terminals are configured to obtain real labels of a first test data set after obtaining the satellite-borne model fine-tuned by the collaborative labeling of each of the plurality of satellite terminals, the first test data set comprising first unlabeled data and first labeled data.

[0213] The plurality of satellite terminals are configured to label the first unlabeled data by using the satellite-borne model fine-tuned by the collaborative labeling to obtain labeled first unlabeled data.

[0214] Each of the plurality of satellite terminals is configured to calculate a labeling accuracy of the satellite-borne model fine-tuned by the collaborative labeling according to the labeled first unlabeled data and the real labels of the first unlabeled data in the real labels of the first test data set, the labeling accuracy of the satellite-borne model fine-tuned by the collaborative labeling being a proportion of labeled correct first unlabeled data in all first unlabeled data.

[0215] Each of the plurality of satellite terminals is configured to mix the labeled first unlabeled data and the first labeled data to obtain mixed labeled data.

[0216] Each satellite terminal of the plurality of satellite terminals is configured to fine-tune the fine-tuned spaceborne model through the mixed labeling data to obtain a first spaceborne model, the first spaceborne model being a system accuracy of the first spaceborne model calculated through a second test data set after the first spaceborne model is fine-tuned through the mixed labeling data; and the system accuracy of the first spaceborne model being a correct rate that the first spaceborne model can reach after the first spaceborne model is fine-tuned through the mixed labeling data and labels the second test data set.

[0217] The ground station is configured to evaluate performance improvement of the fine-tuned spaceborne model through collaborative labeling based on the labeling accuracy of the fine-tuned spaceborne model through collaborative labeling and the system accuracy of the first spaceborne model.

[0218] The system further comprises:

[0219] The ground station is configured to set an initialization parameter and a model structure to obtain a spaceborne model to be fine-tuned before evaluating the performance improvement of the fine-tuned spaceborne model through collaborative labeling.

[0220] The ground station is configured to set second related information participating in the collaborative process, the second related information comprising: a number of the plurality of satellite terminals, a frequency of each satellite terminal participating in the collaborative process, a label scarcity degree and a label deviation degree of a local training data set of each satellite terminal, a deviation operator of each satellite terminal, and a number of measurement fine-tuning rounds.

[0221] The plurality of satellite terminals participating in the collaborative process are configured to each utilize third unlabeled data in a local training data set to unsupervisedly fine-tune the spaceborne model to be fine-tuned to obtain a fine-tuning parameter of a current fine-tuned spaceborne model.

[0222] The central server of the ground station is configured to randomly select a plurality of satellite terminals from the plurality of satellite terminals participating in the collaborative process as satellite terminals participating in collaborative aggregation, and aggregate the fine-tuning parameters of the plurality of current fine-tuned spaceborne models obtained through unsupervised fine-tuning of the plurality of satellite terminals participating in the collaborative aggregation to obtain a fine-tuning parameter of a current aggregated spaceborne model.

[0223] The ground station is configured to send the fine-tuning parameter of the current aggregated spaceborne model to the plurality of satellite terminals to perform a next round of collaborative process.

[0224] The plurality of satellite terminals participating in the collaborative process are configured to obtain a second converged spaceborne model after performing a plurality of rounds of collaborative process.

[0225] The plurality of satellite terminals participating in the collaborative process are configured to respectively fine-tune the second converged satellite model using labeled data in the local training dataset to obtain a fine-tuned satellite model.

[0226] The plurality of satellite terminals participating in the collaborative process are configured to perform category prediction on third unlabeled data in the local training dataset by using the fine-tuned satellite model, and label the third unlabeled data based on the category prediction result to obtain labeled third unlabeled data.

[0227] The plurality of satellite terminals participating in the collaborative process are configured to determine the labeling accuracy of the fine-tuned satellite model according to the true labels of the third unlabeled data in the local training dataset and the labeled third unlabeled data.

[0228] The plurality of satellite terminals participating in the collaborative process are configured to respectively perform supervised fine-tuning on the fine-tuned satellite model using the labeled third unlabeled data in the local training dataset and the third labeled data in the local training dataset to obtain a final fine-tuned satellite model.

[0229] The plurality of satellite terminals participating in the collaborative process are configured to respectively test the accuracy of the final fine-tuned satellite model using a third test dataset to obtain a system accuracy of the final fine-tuned satellite model.

[0230] The ground station is configured to reset the second related information and repeat the above steps to obtain a corresponding relationship between the labeling accuracy and the system accuracy of the final fine-tuned satellite model under different label scarcity degrees and different label deviation degrees.

[0231] The ground station is configured to evaluate the performance improvement of the collaborative labeling fine-tuned satellite model, including that the ground station is configured to determine the performance improvement degree of the collaborative labeling fine-tuned satellite model based on the corresponding relationship, the labeling accuracy of the collaborative labeling fine-tuned satellite model, and the system accuracy of the collaborative labeling fine-tuned satellite model.

[0232] Based on the same inventive concept, another embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the satellite-ground collaborative labeling fine-tuning method for satellite data label scarcity as described in any of the above embodiments.

[0233] The electronic device refers to Figure 3 , Figure 3 is a schematic diagram of the electronic device provided by an embodiment of the present application. As shown in Figure 3As shown, the electronic device 300 includes a memory 310 and a processor 320, the memory 310 and the processor 320 are connected by a bus in communication, the memory 310 stores a computer program, the computer program is executable on the processor 320, and then the steps in the method disclosed in the above embodiments of the present application are implemented.

[0234] For the system, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant part is referred to the part of the method embodiment.

[0235] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, device, or computer program product. Therefore, the embodiments of the present application can adopt a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can adopt the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes. The embodiments of the present application are described with reference to flowcharts and / or block diagrams of the method, terminal device (system), and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks can be realized by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the computer or other programmable data processing terminal device produce a device for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that carries out the functions specified in one or more flows and / or blocks. These computer program instructions can also be stored in a computer-readable memory that can cause the computer or other programmable data processing terminal device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction apparatus, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that carries out the functions specified in one or more flows and / or blocks. These computer program instructions can also be stored in a computer-readable memory that can cause the computer or other programmable data processing terminal device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction apparatus, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1The functions processes of the various elements were determined in some embodiments of the application with reference to the specified functions of the block (s) in the computer system, as well as the processes for operating these block (s). While preferred embodiments of the methods of the application have been described, those skilled in the art will be able to make additional variations and modifications without departing from the scope of the application. Therefore, the appended claims are intended to cover all such variations and modifications that come within the scope of the application.

[0236] Finally, it should be noted that, in the present document, relational terms such as first and second, and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0237] The above provides a satellite data label scarce satellite-ground collaborative labeling fine-tuning method, system and device, and the principle and implementation manner of the present application are described by specific examples in the present document. The above description of the embodiments is only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range can be changed, and the content of the present specification should not be understood as a limitation of the present application.

Claims

1. A method for fine-tuning satellite-ground collaborative annotation for data label scarcity, characterized in that, The method includes: Multiple satellite terminals receive pre-trained satellite-borne models from the ground station. The pre-trained satellite-borne models are obtained by initializing the parameters of the model to be fine-tuned that was pre-trained by the ground station. The model to be fine-tuned has the function of predicting the category of image data collected by any satellite terminal. In each round of collaborative labeling fine-tuning, the multiple satellite terminals each use unlabeled data in their local datasets to perform unsupervised comparative fine-tuning of the pre-trained satellite-borne model, thereby obtaining the fine-tuning parameters of the initial satellite-borne model for this round. The multiple satellite terminals send the fine-tuning parameters of their respective initial onboard models to the ground station; The ground station aggregates the fine-tuning parameters of the multiple initial satellite-borne models received to obtain the fine-tuning parameters of the aggregated satellite-borne model for this round, and then sends the fine-tuning parameters of the aggregated satellite-borne model to the multiple satellite terminals. After the pre-trained spaceborne model undergoes multiple rounds of collaborative annotation and fine-tuning between the ground station and multiple satellite terminals, the multiple satellite terminals obtain a first converged spaceborne model, which consists of model parameters with generalized knowledge representation. Each of the multiple satellite terminals uses labeled data from its local dataset to perform supervised fine-tuning of the first converged satellite-borne model, resulting in a collaboratively labeled fine-tuned satellite-borne model for each of the multiple satellite terminals. The collaboratively labeled fine-tuned satellite-borne model for each satellite terminal is used to predict the category of the image data collected by that satellite terminal in order to obtain labeled image data. This includes, after multiple satellite terminals receive the pre-trained spaceborne model from the ground station, the following: The multiple satellite terminals respectively split the pre-trained onboard model into an encoder network near the input layer and a decoder network near the output layer, and freeze the parameters of the decoder network; In each round of collaborative annotation fine-tuning, the multiple satellite terminals each use unlabeled data in their local datasets to perform unsupervised comparative fine-tuning on the pre-trained satellite-borne model to obtain the fine-tuning parameters of the initial satellite-borne model for this round. This includes: in each round of collaborative annotation fine-tuning, the multiple satellite terminals each use unlabeled data in their local datasets to perform unsupervised comparative fine-tuning on the encoder network in the pre-trained satellite-borne model to obtain the fine-tuning parameters of the initial satellite-borne model for this round. The fine-tuning parameters of the initial satellite-borne model are the parameters of the fine-tuned encoder network. The first converged onboard model includes the parameters of the converged encoder network and the parameters of the frozen decoder network.

2. The satellite-ground collaborative annotation fine-tuning method for data label scarcity as described in claim 1, characterized in that, The multiple satellite terminals send their respective initial onboard model fine-tuning parameters to the ground station, including: The central server deployed at the ground station randomly selects multiple satellite terminals from the multiple satellite terminals that can establish connections as satellite terminals to participate in collaborative aggregation. The multiple satellite terminals participating in the collaborative aggregation send their respective initial onboard model fine-tuning parameters to the ground station.

3. The satellite-ground collaborative annotation fine-tuning method for scarce data labels as described in claim 2, characterized in that, The ground station aggregates the fine-tuning parameters of multiple initial spaceborne models received to obtain the fine-tuning parameters of the aggregated spaceborne model for this round, including: When the number of satellite terminals participating in collaborative aggregation is less than a preset threshold, the ground station selects the fine-tuning parameters of the aggregated satellite-borne model that are most similar to the fine-tuning parameters of the multiple initial satellite-borne models sent to the ground station by the multiple satellite terminals participating in collaborative aggregation from the fine-tuning parameters of the multiple aggregated satellite-borne models obtained from the previous N rounds of collaborative annotation and fine-tuning process, where N≥1; The ground station aggregates the fine-tuning parameters of the multiple initial onboard models sent by the multiple satellite terminals participating in the collaborative aggregation, and the fine-tuning parameters of the most similar aggregated onboard model, to obtain the fine-tuning parameters of the aggregated onboard model for this round.

4. The satellite-ground collaborative annotation fine-tuning method for data tag scarcity as described in claim 1, characterized in that, The multiple satellite terminals obtained the first converged onboard model, including: After the onboard model undergoes multiple rounds of collaborative annotation and fine-tuning between the ground station and the multiple satellite terminals, the parameters of the converged encoder network issued by the ground station are obtained. The multiple satellite terminals concatenate the parameters of the received converged encoder network with the parameters of the frozen decoder network to obtain the first converged onboard model.

5. The satellite-ground collaborative annotation fine-tuning method for data tag scarcity as described in claim 1, characterized in that, Before the collaborative annotation fine-tuning process, the following is also included: Set the initialization parameters and model structure to obtain the pre-trained spaceborne model; The first relevant information for participating in the collaborative process is set, which includes: the number of the multiple satellite terminals, the frequency of each of the multiple satellite terminals participating in collaborative aggregation, the label scarcity of the local dataset of each of the multiple satellite terminals, the deviation operator and the number of measurement fine-tuning rounds of each of the multiple satellite terminals; The tag scarcity level is used to represent the proportion of tagged data in the satellite terminal, and the tag deviation level is used to represent the distribution of tagged data across multiple satellite terminals; the deviation operator is an operation operator used to simulate tagged data with different tag deviation levels on multiple satellite terminals; and the measurement fine-tuning rounds represent the number of times collaborative aggregation is performed.

6. The satellite-ground collaborative annotation fine-tuning method for data tag scarcity as described in claim 1, characterized in that, After obtaining the collaboratively labeled and finely tuned onboard models of the multiple satellite terminals, the following is also included: Obtain the true labels of the first test dataset, which contains first unlabeled data and first labeled data; The first unlabeled data is labeled using the spaceborne model finely adjusted by the collaborative annotation, resulting in labeled first unlabeled data. Based on the first unlabeled data after labeling and the real labels of the first unlabeled data in the real labels of the first unlabeled data in the first test dataset, the labeling accuracy of the spaceborne model after collaborative labeling fine-tuning is calculated. The labeling accuracy of the spaceborne model after collaborative labeling fine-tuning is: the proportion of the first unlabeled data with correct labeling in the first unlabeled data after labeling to all the first unlabeled data. The first unlabeled data and the first labeled data are mixed to obtain mixed labeled data; Using the hybrid labeled data, the co-labeled fine-tuned spaceborne model is further fine-tuned to obtain a first spaceborne model. The first spaceborne model is the system accuracy of the first spaceborne model after further fine-tuning using the hybrid labeled data and then labeled with a second test dataset. The system accuracy of the first spaceborne model is: the accuracy that the first spaceborne model can achieve after further fine-tuning using the hybrid labeled data and labeling the second test dataset. The performance improvement of the co-labeled fine-tuned spaceborne model is evaluated based on the annotation accuracy of the spaceborne model after fine-tuning and the system accuracy of the first spaceborne model.

7. The satellite-ground collaborative annotation fine-tuning method for scarce data labels as described in claim 6, characterized in that, Before evaluating the performance improvement of the co-annotation-fine-tuned spaceborne model, the following steps are also included: Set the initialization parameters and model structure to obtain the spaceborne model to be fine-tuned; Set a second set of relevant information for participating in the collaborative process. The second set of relevant information includes: the number of multiple satellite terminals, the frequency of each satellite terminal participating in the collaborative process, the scarcity of labels in the local training dataset of each satellite terminal and the degree of label deviation in the local training dataset, the deviation operator of each satellite terminal and the number of measurement fine-tuning rounds. The multiple satellite terminals participating in the collaborative process each use the third unlabeled data in their local training dataset to perform unsupervised comparative fine-tuning of the onboard model to be fine-tuned, thereby obtaining the fine-tuning parameters of the current onboard model. The central server of the ground station randomly selects multiple satellite terminals from the multiple satellite terminals participating in the collaborative process as satellite terminals participating in collaborative aggregation, and aggregates the fine-tuning parameters of multiple currently fine-tuned onboard models obtained by unsupervised comparison fine-tuning of the selected multiple satellite terminals participating in collaborative aggregation to obtain the fine-tuning parameters of the currently aggregated onboard model. The ground system sends the fine-tuning parameters of the current aggregated onboard model to multiple satellite terminals for the next round of collaborative processing. After multiple rounds of collaborative processes, each of the satellite terminals participating in the collaborative process obtains a second converged onboard model. The multiple satellite terminals participating in the collaborative process each use labeled data in their local training datasets to locally fine-tune the second convergent onboard model, thereby obtaining a fine-tuned onboard model. The multiple satellite terminals participating in the collaborative process use the fine-tuned onboard model to predict the category of the third unlabeled data in the local training dataset, and label the third unlabeled data based on the category prediction results to obtain the labeled third unlabeled data. The labeling accuracy of the fine-tuned spaceborne model is determined based on the true label of the third unlabeled data in the local training dataset and the labeled third unlabeled data. The multiple satellite terminals participating in the collaborative process each use the third unlabeled data after labeling in their local training dataset and the third labeled data in their local training dataset to perform supervised fine-tuning of the fine-tuned onboard model, thereby obtaining the final fine-tuned onboard model. The multiple satellite terminals participating in the collaborative process each use a third test dataset to test the accuracy of the final fine-tuned onboard model, thereby obtaining the system accuracy of the final fine-tuned onboard model. Reset the second relevant information and repeat the above steps to obtain the correspondence between the labeling accuracy and system accuracy of the final fine-tuned spaceborne model under different label scarcity and different label deviation. The evaluation of the performance improvement of the collaboratively labeled fine-tuned spaceborne model includes: determining the degree of performance improvement of the collaboratively labeled fine-tuned spaceborne model based on the correspondence, the labeling accuracy of the collaboratively labeled fine-tuned spaceborne model, and the system accuracy of the collaboratively labeled fine-tuned spaceborne model.

8. A satellite-ground collaborative labeling fine-tuning system for satellite data tag scarcity, characterized in that, The system includes multiple satellite terminals and ground stations: The multiple satellite terminals are used to receive pre-trained satellite models sent by the ground station. The pre-trained satellite models are obtained by initializing the parameters of the model to be fine-tuned pre-trained by the ground station. The model to be fine-tuned has the function of predicting the category of image data collected by any satellite terminal. In each round of collaborative labeling fine-tuning, the multiple satellite terminals are used to perform unsupervised comparative fine-tuning of the pre-trained satellite model using unlabeled data in their local datasets, so as to obtain the fine-tuning parameters of the initial satellite model for this round. The multiple satellite terminals are used to send the fine-tuning parameters of their respective initial onboard models to the ground station; The ground station is used to aggregate the fine-tuning parameters of multiple initial satellite models received to obtain the fine-tuning parameters of the aggregated satellite model for this round, and to send the fine-tuning parameters of the aggregated satellite model to the multiple satellite terminals. After the pre-trained spaceborne model undergoes multiple rounds of collaborative annotation and fine-tuning between the ground station and multiple satellite terminals, the multiple satellite terminals are used to receive the first converged spaceborne model, which is a model parameter with generalized knowledge representation. The multiple satellite terminals are used to perform supervised fine-tuning of the first converged satellite model using labeled data in their local datasets, so as to obtain the satellite model after collaborative labeling fine-tuning of each satellite terminal. The satellite model after collaborative labeling fine-tuning of each satellite terminal is used to predict the category of the image data collected by the satellite terminal in order to obtain labeled image data. The system further includes: the plurality of satellite terminals, which, after receiving the pre-trained satellite-borne model from the ground station, respectively split the pre-trained satellite-borne model into an encoder network near the input layer and a decoder network near the output layer, and freeze the parameters of the decoder network; The multiple satellite terminals are used to perform unsupervised comparative fine-tuning of the pre-trained satellite-borne model using unlabeled data in their local datasets during each round of collaborative labeling fine-tuning, to obtain the fine-tuning parameters of the initial satellite-borne model for this round. This includes: during each round of collaborative labeling fine-tuning, the multiple satellite terminals use unlabeled data in their local datasets to perform unsupervised comparative fine-tuning of the encoder network in the pre-trained satellite-borne model, to obtain the fine-tuning parameters of the initial satellite-borne model for this round. The fine-tuning parameters of the initial satellite-borne model are the parameters of the fine-tuned encoder network. The first converged onboard model includes the parameters of the converged encoder network and the parameters of the frozen decoder network.

9. An electronic device, characterized in that, The system includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the satellite-ground collaborative labeling fine-tuning method for data tag scarcity as described in any one of claims 1-7.

Images