Data label scarcity-oriented satellite-ground collaborative annotation fine tuning method, system and equipment

Abstract

The invention provides a satellite-ground collaborative annotation fine tuning method, system and device for data label scarcity, and the method comprises the steps: a plurality of satellite terminals respectively receive pre-trained satellite-borne models issued by a ground station, the plurality of satellite terminals carry out the unsupervised comparison fine tuning of the pre-trained satellite-borne models through the unlabeled data, and the unsupervised comparison fine tuning of the pre-trained satellite-borne models is carried out; the ground station aggregates the fine tuning parameters of the initial satellite-borne model to obtain fine tuning parameters of the aggregated satellite-borne model of the current round, after multiple rounds of collaborative annotation fine tuning, a plurality of satellite terminals obtain a first convergence satellite-borne model, and the plurality of satellite terminals respectively utilize the labeled data to obtain a second convergence satellite-borne model; supervised fine tuning is carried out on the first convergence satellite-borne model to obtain respective satellite-borne models of the plurality of satellite terminals after collaborative annotation fine tuning, and the satellite-borne model of each satellite terminal after collaborative annotation fine tuning is used for carrying out category prediction on the image data collected by the satellite terminal to obtain annotated image data.

Description

Technical Field

[0001] This application relates to the field of satellite computing technology, and particularly to a space-ground collaborative annotation fine-tuning method, system and device for scarce data tags. Background Art

[0002] In recent years, with the remarkable progress of satellite technology and the rapid development of commercial off-the-shelf (COTS) hardware, it has become a trend for low-earth orbit satellite constellations to be equipped with intelligent computing capabilities. Mainstream cloud service providers (such as Google and AWS) and satellite companies (such as OrbitsEdge) have successively proposed the "space infrastructure as a service" initiative. Through the satellite computing paradigm, on-orbit data can be quickly and intelligently processed on orbit to improve the quality of on-orbit services. For example, during Hurricane Harvey and the 2021 earthquake in Turkey, the Capella Space satellite constellation used the intelligent computing on the satellite to shorten the disaster emergency map generation time by 91.6% by equipping intelligent chips, quickly providing key and accurate positioning services for rescue workers, and significantly reducing the scope of the disaster and casualties. Therefore, satellite intelligent computing has demonstrated its great help in global challenges such as disaster navigation and disaster reduction, and has great potential to support various complex and dynamic on-orbit services.

[0003] Although satellite intelligent computing has shown great value and potential, in its actual deployment and operation process, its system performance still restricts its ability to continuously provide high-quality on-orbit services. Specifically, limited by physical conditions such as satellite size and passive heat dissipation, it is usually difficult to improve the system model performance by increasing the size of the deployed model, and the system model performance directly affects the accuracy of on-orbit results. Inaccurate results are difficult to effectively assist in the provision of on-orbit services. Therefore, considering the use of the federated learning paradigm for collaborative fine-tuning between space and ground provides a promising solution for continuously improving the performance of spaceborne lightweight models with limited size. In particular, during the continuous on-orbit operation of each satellite every day, a large amount of image data is collected, usually reaching dozens of terabytes per day. Therefore, the space-ground federated learning system has the opportunity to learn potential knowledge from these data collected by each satellite and share it, thus providing an opportunity for the performance evolution of the spaceborne models of each satellite.

[0004] Some previous technical studies have explored how to improve the performance and efficiency of space-ground federated systems. Specifically, FedSpace first proposed a space-ground federated system and further designed an adaptive buffer to improve system efficiency by balancing the synchronous and asynchronous aggregation of the space-ground federated process. Subsequent studies have further explored how to efficiently improve the performance of space-ground federated systems in resource-constrained on-orbit environments.

[0005] However, despite the many efforts made in the above research, since the data collected in orbit is usually scarce in labels, most satellites are neither equipped with equipment models with professional annotation capabilities for on-orbit annotation nor can they recruit highly specialized data annotators for real-time collaboration. This lack of data labels poses great difficulties to the supervised fine-tuning process. Summary of the Invention

[0006] In view of this, embodiments of the present application provide a space-ground collaborative annotation fine-tuning method, system and device for scarce data labels, so as to overcome the above problems or at least partially solve the above problems.

[0007] The first aspect of the embodiments of the present application provides a space-ground collaborative annotation fine-tuning method for scarce data labels, and the method includes: Multiple satellite terminals respectively receive a pre-trained on-board model sent by a ground station. The pre-trained on-board model is obtained after initializing the parameters of a to-be-fine-tuned model pre-trained by the ground station. The to-be-fine-tuned model has the function of predicting the category of image data collected by any satellite terminal; In each round of collaborative annotation fine-tuning process, the multiple satellite terminals respectively use the unlabeled data in their local data sets to perform unsupervised contrast fine-tuning on the pre-trained on-board model to obtain the fine-tuning parameters of the initial on-board model in this round; The multiple satellite terminals send the fine-tuning parameters of their respective initial on-board models to the ground station; The ground station aggregates the fine-tuning parameters of the multiple initial on-board models received to obtain the fine-tuning parameters of the aggregated on-board model in this round, and sends the fine-tuning parameters of the aggregated on-board model to the multiple satellite terminals; After the pre-trained on-board model undergoes multiple rounds of collaborative annotation fine-tuning between the ground station and the multiple satellite terminals, the multiple satellite terminals obtain a first converged on-board model. The first converged on-board model is the model parameters with generalized knowledge representation; The multiple satellite terminals respectively use the labeled data in their local data sets to perform supervised fine-tuning on the first converged on-board model to obtain the on-board models after collaborative annotation fine-tuning of the multiple satellite terminals. The on-board model after collaborative annotation fine-tuning of each satellite terminal is used to predict the category of the image data collected by the satellite terminal to obtain labeled image data.

[0008] Optionally, after the multiple satellite terminals respectively receive the pre-trained on-board model sent by the ground station, it further includes: The multiple satellite terminals respectively split the pre-trained on-board model into an encoder network close to the input layer and a decoder network close to the output layer, and freeze the parameters of the decoder network; In each round of collaborative annotation fine-tuning process, each of the multiple satellite terminals uses the unlabeled data in its local dataset to perform unsupervised contrastive fine-tuning on the pre-trained on-board model, obtaining the fine-tuning parameters of the initial on-board model for this round, including: In each round of collaborative annotation fine-tuning process, each of the multiple satellite terminals uses the unlabeled data in its local dataset to perform unsupervised contrastive fine-tuning on the encoder network in the pre-trained on-board model, obtaining the fine-tuning parameters of the initial on-board model for this round, and the fine-tuning parameters of the initial on-board model are the parameters of the fine-tuned encoder network; Among them, the first converged on-board model includes the parameters of the converged encoder network and the parameters of the frozen decoder network.

[0009] Optionally, each of the multiple satellite terminals sends the fine-tuning parameters of its initial on-board model to the ground station, including: The central server deployed at the ground station randomly selects multiple satellite terminals from the multiple connectable satellite terminals included in the multiple satellite terminals as the satellite terminals participating in collaborative aggregation; The multiple satellite terminals participating in collaborative aggregation send the fine-tuning parameters of their respective initial on-board models to the ground station.

[0010] Optionally, the ground station aggregates the fine-tuning parameters of the multiple initial on-board models received to obtain the fine-tuning parameters of the aggregated on-board model for this round, including: When the number of satellite terminals participating in collaborative aggregation is less than the preset quantity threshold, the ground station selects, from the fine-tuning parameters of the multiple aggregated on-board models obtained in the previous N rounds of collaborative annotation fine-tuning processes, the fine-tuning parameters of the aggregated on-board model that are most similar to the fine-tuning parameters of the multiple initial on-board models sent by the multiple satellite terminals participating in collaborative aggregation to the ground station, where N≥1; The ground station aggregates based on the fine-tuning parameters of the multiple initial on-board models sent by the multiple satellite terminals participating in collaborative aggregation received and the fine-tuning parameters of the most similar aggregated on-board model to obtain the fine-tuning parameters of the aggregated on-board model for this round.

[0011] Optionally, each of the multiple satellite terminals obtains the first converged on-board model, including: After the on-board model undergoes multiple rounds of collaborative annotation fine-tuning between the ground station and the multiple satellite terminals, each of the multiple satellite terminals obtains the parameters of the converged encoder network sent by the ground station; Each of the multiple satellite terminals splices the received parameters of the converged encoder network with the parameters of the frozen decoder network to obtain the first converged on-board model.

[0012] Optionally, before the collaborative annotation fine-tuning process, it further includes: Set initialization parameters and the model structure to obtain the pre-trained on-board model; Set the first relevant information participating in the collaboration process, where the first relevant information includes: the number of the multiple satellite terminals, the respective frequencies of the multiple satellite terminals participating in collaborative aggregation, the label scarcity degrees of the respective local data sets of the multiple satellite terminals and the label scarcity degree of the local data sets, the respective deviation operators of the multiple satellite terminals and the measurement fine-tuning rounds; Among them, the label scarcity degree is used to represent the proportion of labeled data in the satellite terminal, and the label deviation degree is used to represent the distribution of labeled data among the multiple satellite terminals; the deviation operator represents an operation operator for simulating labeled data with different label deviation degrees on multiple satellite terminals; the measurement fine-tuning rounds represent the number of times of collaborative aggregation.

[0013] Optionally, after obtaining the on-board models of the multiple satellite terminals after collaborative annotation and fine-tuning, it further includes: Obtain the true labels of the first test data set, where the first test data set contains first unlabeled data and first labeled data; Label the first unlabeled data through the on-board model after collaborative annotation and fine-tuning to obtain the labeled first unlabeled data; According to the true labels of the first unlabeled data in the labeled first unlabeled data and the true labels of the first test data set, calculate the annotation accuracy rate of the on-board model after collaborative annotation and fine-tuning, where the annotation accuracy rate of the on-board model after collaborative annotation and fine-tuning is: the proportion of the correctly labeled first unlabeled data in the labeled first unlabeled data to all the first unlabeled data; Mix the labeled first unlabeled data and the first labeled data to obtain mixed labeled data; Fine-tune the on-board model after collaborative annotation and fine-tuning again through the mixed labeled data to obtain a first on-board model. The first on-board model is to calculate the system accuracy rate of the first on-board model through a second test data set after fine-tuning again through the mixed labeled data; the system accuracy rate of the first on-board model is: the correct rate that can be achieved after the first on-board model is labeled with the second test data set after fine-tuning the first on-board model again with the mixed labeled data; Evaluate the performance improvement of the on-board model after collaborative annotation and fine-tuning based on the annotation accuracy rate of the on-board model after collaborative annotation and fine-tuning and the system accuracy rate of the first on-board model.

[0014] Optionally, before evaluating the performance improvement of the on-board model after collaborative annotation and fine-tuning, it further includes: Set the initialization parameters and model structure to obtain the satellite-borne model to be fine-tuned; Set the second relevant information participating in the collaborative process, where the second relevant information includes: the number of multiple satellite terminals, the frequency of each satellite terminal participating in the collaborative process, the label scarcity degree and label deviation degree of the local training dataset of each satellite terminal, the deviation operator of each satellite terminal, and the measured fine-tuning rounds; Each of the multiple satellite terminals participating in the collaborative process uses the third unlabeled data in the local training dataset to perform unsupervised contrast fine-tuning on the satellite-borne model to be fine-tuned, and obtains the fine-tuning parameters of the current fine-tuned satellite-borne model; The central server of the ground station randomly selects multiple satellite terminals from the multiple satellite terminals participating in the collaborative process as the satellite terminals participating in collaborative aggregation, and aggregates the fine-tuning parameters of the multiple current fine-tuned satellite-borne models obtained by the selected multiple satellite terminals participating in collaborative aggregation through unsupervised contrast fine-tuning to obtain the fine-tuning parameters of the current aggregated satellite-borne model; The ground sends the fine-tuning parameters of the current aggregated satellite-borne model to multiple satellite terminals for the next round of collaborative process; After multiple rounds of collaborative processes, each of the multiple satellite terminals participating in the collaborative process obtains a second converged satellite-borne model; Each of the multiple satellite terminals participating in the collaborative process uses the labeled data in the local training dataset to perform local fine-tuning on the second converged satellite-borne model to obtain a fine-tuned satellite-borne model; Each of the multiple satellite terminals participating in the collaborative process uses the fine-tuned satellite-borne model to perform class prediction on the third unlabeled data in the local training dataset, and based on the result of the class prediction, annotates the third unlabeled data to obtain the labeled third unlabeled data; Determine the annotation accuracy rate of the fine-tuned satellite-borne model according to the true labels of the third unlabeled data in the local training dataset and the labeled third unlabeled data; Each of the multiple satellite terminals participating in the collaborative process uses the labeled third unlabeled data in the local training dataset and the third labeled data in the local training dataset to perform supervised fine-tuning on the fine-tuned satellite-borne model to obtain a final fine-tuned satellite-borne model; Each of the multiple satellite terminals participating in the collaborative process uses the third test dataset to test the accuracy rate of the final fine-tuned satellite-borne model to obtain the system accuracy rate of the final fine-tuned satellite-borne model; Reset the second relevant information and repeat the above steps to obtain the corresponding relationship between the annotation accuracy rate and the system accuracy rate of the final fine-tuned satellite-borne model under different label scarcity degrees and different label deviation degrees; Evaluating the performance improvement of the spaceborne model after the collaborative annotation fine-tuning includes: determining the degree of performance improvement of the spaceborne model after the collaborative annotation fine-tuning based on the corresponding relationship, the annotation accuracy of the spaceborne model after the collaborative annotation fine-tuning, and the system accuracy of the spaceborne model after the collaborative annotation fine-tuning.

[0015] In a second aspect of the embodiments of the present application, a space-ground collaborative annotation fine-tuning system for scarce data labels is provided. The system includes multiple satellite terminals and a ground station: The multiple satellite terminals are each configured to receive a pre-trained spaceborne model sent by the ground station. The pre-trained spaceborne model is obtained after parameter initialization of a to-be-fine-tuned model pre-trained by the ground station. The to-be-fine-tuned model has a function of predicting the category of image data collected by any satellite terminal. In each round of collaborative annotation fine-tuning process, the multiple satellite terminals are each configured to use the unlabeled data in their local datasets to perform unsupervised contrast fine-tuning on the pre-trained spaceborne model to obtain the fine-tuning parameters of the initial spaceborne model for this round. The multiple satellite terminals are each configured to send the fine-tuning parameters of their respective initial spaceborne models to the ground station. The ground station is configured to aggregate the fine-tuning parameters of the multiple received initial spaceborne models to obtain the fine-tuning parameters of the aggregated spaceborne model for this round, and send the fine-tuning parameters of the aggregated spaceborne model to the multiple satellite terminals. After the pre-trained spaceborne model undergoes multiple rounds of collaborative annotation fine-tuning between the ground station and the multiple satellite terminals, the multiple satellite terminals are each configured to receive a first converged spaceborne model. The first converged spaceborne model is a model parameter with generalized knowledge representation. The multiple satellite terminals are each configured to use the labeled data in their local datasets to perform supervised fine-tuning on the first converged spaceborne model to obtain the spaceborne models after collaborative annotation fine-tuning for their respective satellite terminals. The spaceborne model after collaborative annotation fine-tuning for each satellite terminal is used to predict the category of the image data collected by that satellite terminal to obtain labeled image data. The system includes: In a third aspect of the embodiments of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored on the memory. The processor executes the computer program to implement the method as described in the first aspect.

[0016] Advantages of the present application: An embodiment of the present application provides a space-ground collaborative annotation fine-tuning method, system and device for scarce data labels. The method includes: multiple satellite terminals respectively receive a pre-trained on-board model sent by a ground station. The pre-trained on-board model is obtained after parameter initialization of a to-be-fine-tuned model pre-trained by the ground station. The to-be-fine-tuned model has the function of predicting the category of image data collected by any satellite terminal; in each round of collaborative annotation fine-tuning process, the multiple satellite terminals respectively use the unlabeled data in their local datasets to perform unsupervised contrast fine-tuning on the pre-trained on-board model to obtain the fine-tuning parameters of the initial on-board model in this round; the multiple satellite terminals send the fine-tuning parameters of their respective initial on-board models to the ground station; the ground station aggregates the fine-tuning parameters of the multiple initial on-board models received to obtain the fine-tuning parameters of the aggregated on-board model in this round, and sends the fine-tuning parameters of the aggregated on-board model to the multiple satellite terminals; after multiple rounds of collaborative annotation fine-tuning of the pre-trained on-board model between the ground station and the multiple satellite terminals, the multiple satellite terminals obtain a first converged on-board model. The first converged on-board model is a model parameter with generalized knowledge representation; the multiple satellite terminals respectively use the labeled data in their local datasets to perform supervised fine-tuning on the first converged on-board model to obtain the on-board models after collaborative annotation fine-tuning of the multiple satellite terminals respectively. The on-board model after collaborative annotation fine-tuning of each satellite terminal is used to predict the category of image data collected by the satellite terminal to obtain labeled image data.

[0017] Through the technical solution of the present application, satellite terminals can use unlabeled data to learn the feature representation therein in the case of scarce labeled data, reducing the dependence on labeled data. The subsequent supervised fine-tuning further improves the performance of the model in specific tasks, and can achieve a high annotation accuracy even in the case of scarce labeled data. This way of combining unsupervised contrast fine-tuning and supervised fine-tuning enables the model to better extract the feature representation in image data and use a small amount of labeled data for optimization, thus significantly improving the annotation accuracy. Moreover, adopting multiple rounds of collaborative fine-tuning and parameter aggregation enables the model to comprehensively integrate the learning results of different satellite terminals, enhancing the generalization ability of the model. By sharing the feature representation of data, the model can better adapt to the data characteristics of different satellites. In addition, through space-ground collaborative annotation fine-tuning, the model can adapt to the particularity of the satellite data where it is located. To sum up, through this way of space-ground collaborative annotation fine-tuning, the present application effectively solves the technical problem of scarce satellite data labels, improves the annotation ability and overall performance of the model, and meets the requirements of satellite terminals in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings that form a part of this application are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation of this application.

[0019] To more clearly illustrate the technical solutions of this application, the accompanying drawings required for the description of this application will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of this application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0020] Figure 1 is a flowchart of a satellite-ground collaborative annotation fine-tuning method for scarce data labels shown in an embodiment of this application; Figure 2 is an overall framework diagram of a satellite-ground collaborative annotation fine-tuning method for scarce data labels shown in an embodiment of this application; Figure 3 is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed implementation manners

[0021] It should be noted that, without conflict, the embodiments in this application and the features in the embodiments can be combined with each other.

[0022] Next, the technical solutions in the embodiments of this application will be clearly and completely described with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are some, but not all, of the embodiments of this application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without creative efforts fall within the scope of protection of this application.

[0023] Refer to Figure 1 , Figure 1 is a flowchart of a satellite-ground collaborative annotation fine-tuning method for scarce data labels shown in an embodiment of this application, Figure 2 is an overall framework diagram of a satellite-ground collaborative annotation fine-tuning method for scarce data labels shown in an embodiment of this application. Specifically, this embodiment provides a satellite-ground collaborative annotation fine-tuning method for scarce data labels, and this method includes steps S11 to S16: Step S11, multiple satellite terminals respectively receive the pre-trained on-board model sent by the ground station. The pre-trained on-board model is obtained after parameter initialization of the to-be-fine-tuned model pre-trained by the ground station. The to-be-fine-tuned model has the function of predicting the category of image data collected by any satellite terminal.

[0024] Such as Figure 2As shown, this method is applied to a satellite-ground federated system, which includes multiple satellite terminals and a ground station. In this embodiment, in the initial stage of the collaborative annotation fine-tuning process between the multiple satellite terminals and the ground station, each of the multiple satellite terminals receives the model parameters of the pre-trained on-board model sent by the ground station. The pre-trained on-board model is a model to be fine-tuned pre-trained by the ground station, and is obtained after parameter initialization of a specific neural network model selected by the central server of the ground station.

[0025] The pre-trained on-board model has the function of predicting the category of image data collected by any satellite terminal. The purpose of the pre-trained model is to provide a satellite terminal with a preliminarily trained model, which can improve the accuracy of the model in predicting the category of image data after subsequent fine-tuning of the pre-trained on-board model. At the same time, the fine-tuned model can better adapt to the characteristics of the image data collected by each satellite terminal.

[0026] Step S12, in each round of the collaborative annotation fine-tuning process, each of the multiple satellite terminals uses the unlabeled data in its local dataset to perform unsupervised contrast fine-tuning on the pre-trained on-board model, and obtains the fine-tuning parameters of the initial on-board model in this round.

[0027] In this embodiment, the multiple satellite terminals and the ground station will perform multiple rounds of collaborative annotation fine-tuning processes. In each round of the collaborative annotation fine-tuning process, each of the multiple satellite terminals will use the unlabeled data in its local dataset to perform unsupervised contrast fine-tuning on the pre-trained on-board model. The local dataset targeted by each satellite is a collection of a large amount of image data collected by the satellite terminal. The image data in the local dataset is divided into two categories: unlabeled data and labeled data. Labeled data refers to image data containing pre-accurately labeled categories, and unlabeled data is image data that does not contain pre-accurately labeled categories; unsupervised contrast fine-tuning, as an advanced learning method, enables the pre-trained on-board model to distinguish the similarities and differences between the image data in the local dataset through contrast learning.

[0028] Specifically, in each round of the collaborative annotation fine-tuning process, the satellite terminal will perform data augmentation operations (such as random cropping, rotation, etc.) on the unlabeled images to generate positive sample pairs, and randomly sample from other images to generate negative sample pairs. Through a contrast loss function (such as the InfoNCE loss), the pre-trained on-board model will learn to make the features of the positive sample pairs closer and the features of the negative sample pairs farther away.

[0029] This process enables the pre-trained on-board model to extract the feature representation in the image data without labels, thereby obtaining the fine-tuning parameters of the initial on-board model in this round. The initial on-board model refers to the model obtained by performing unsupervised contrast fine-tuning on the pre-trained on-board model using unlabeled data.

[0030] The unsupervised contrastive fine-tuning method can make full use of unlabeled data for local learning at the satellite terminal and update the representation information of the on-board data (i.e., the image data collected by the satellite terminal) locally, improving the feature extraction ability of the pre-trained on-board model for the on-board data of the satellite terminal where it is located, and laying a foundation for subsequent annotation tasks such as class prediction.

[0031] Step S13: The multiple satellite terminals send the fine-tuning parameters of their respective initial on-board models to the ground station.

[0032] In this embodiment, after completing the unsupervised contrastive fine-tuning in step S12, the multiple satellite terminals send the fine-tuning parameters of their respective obtained initial on-board models to the ground station. This process is a key link in satellite-ground collaboration. The satellite terminal establishes a communication link with the ground station and transmits the fine-tuning parameters of the initial on-board model as the parameters of the updated pre-trained on-board model to the ground station. The fine-tuning parameters of the initial on-board model contain the feature representations learned by the satellite terminal from unlabeled data. By sending these fine-tuning parameters to the ground station, knowledge sharing between different satellite terminals can be achieved, providing data support for the generation of the aggregated on-board model.

[0033] Step S14: The ground station aggregates the fine-tuning parameters of the multiple received initial on-board models to obtain the fine-tuning parameters of the aggregated on-board model for this round, and sends the fine-tuning parameters of the aggregated on-board model to the multiple satellite terminals.

[0034] In this embodiment, the ground station aggregates the fine-tuning parameters of the multiple received initial on-board models to obtain the fine-tuning parameters of the aggregated on-board model for this round, and sends the obtained fine-tuning parameters of the aggregated on-board model to the multiple satellite terminals. Among them, the aggregation process can adopt the method of averaging or weighted averaging to fuse the fine-tuning parameters of the multiple initial on-board models of different satellite terminals.

[0035] Specifically, by sharing the fine-tuning parameters of each satellite terminal, during the aggregation process, learn the local latent feature representations extracted from unlabeled data by each satellite terminal, so as to obtain the fine-tuning parameters of the aggregated on-board model, and improve the generalization ability of the initial pre-trained on-board model. Further, the ground station sends the fine-tuning parameters of the aggregated on-board model to all satellite terminals for use in the next round of collaborative annotation fine-tuning. In this way, not only can the information loss caused by insufficient satellite quantity be compensated, but also the overall performance of the model can be enhanced through collaborative learning.

[0036] Step S15, after the pre-trained on-board model undergoes multiple rounds of collaborative annotation fine-tuning between the ground station and multiple satellite terminals, the multiple satellite terminals obtain a first converged on-board model, and the first converged on-board model is model parameters with generalized knowledge representation.

[0037] In this embodiment, multiple satellite terminals and the ground station will perform multiple rounds of collaborative annotation fine-tuning processes. In each round, fine-tuning parameters of multiple initial on-board models corresponding to the multiple satellite terminals will be obtained and then sent to the ground station. The ground station aggregates them to obtain fine-tuning parameters of the aggregated on-board model. Each satellite terminal then performs the next round of unsupervised contrast fine-tuning on the received aggregated on-board model to obtain new fine-tuning parameters of the initial on-board model... and so on. After multiple rounds of collaborative annotation fine-tuning processes, the multiple satellite terminals receive the first converged on-board model sent by the ground station. The first converged on-board model refers to model parameters whose model parameters gradually tend to be stable and have generalized knowledge representation after multiple rounds of unsupervised contrast fine-tuning and parameter aggregation. The model at this stage can already better adapt to the data characteristics of different satellites and has strong feature extraction ability and generalization ability. Through multiple rounds of collaborative annotation fine-tuning, the first converged on-board model can share knowledge among different satellite terminals, thereby optimizing performance globally. Finally, the converged model received by the satellite terminal provides a solid foundation for subsequent supervised fine-tuning.

[0038] Step S16, each of the multiple satellite terminals uses the labeled data in its local dataset to perform supervised fine-tuning on the first converged on-board model, obtaining the on-board models after collaborative annotation fine-tuning for each of the multiple satellite terminals. The on-board model after collaborative annotation fine-tuning for each satellite terminal is used to perform class prediction on the image data collected by that satellite terminal to obtain labeled image data.

[0039] In this embodiment, after each satellite terminal receives the first converged on-board model sent by the ground station, each of the multiple satellite terminals uses the labeled data in its local dataset to perform supervised fine-tuning on the first converged on-board model. Supervised fine-tuning is a process of further optimizing the first converged on-board model using this small amount of labeled image data, i.e., the labeled data. Through supervised fine-tuning, the satellite terminal combines the feature extraction ability of the converged model with the local labeled data, further adjusts the model parameters, enabling it to more accurately perform class prediction on image data and enabling the model to achieve higher performance.

[0040] Moreover, since the local datasets of each satellite terminal are different, the labeled data in the local datasets of each satellite terminal are also different. Each satellite terminal uses different labeled data to further perform supervised fine-tuning on the first convergent on-board model, so that the models finally obtained by each satellite terminal are also different, that is, the on-board models after collaborative annotation fine-tuning finally obtained by each satellite terminal are different. Such personalized models can learn the feature representations of the labeled data of the satellite where they are located, so as to adapt to the local annotation tasks of the satellite terminal where they are located.

[0041] Finally, the on-board models after collaborative annotation fine-tuning obtained by each satellite terminal can accurately annotate the image data collected locally, that is, labeled image data can be obtained, without the need to recruit highly professional data annotators for real-time collaboration, solving the problem of scarce labeled data for satellite terminals. At the same time, since the model has the ability to better adapt to the specific task requirements of the satellite terminal where it is located, the accuracy of the annotation of local image data can be improved. In addition, these image data carrying accurate annotations can be used for tasks such as model training and data analysis of satellite terminals.

[0042] For example, in the annotation task of disaster map generation, the on-board models after collaborative annotation fine-tuning of each satellite terminal can be specifically used to identify disasters in the task execution area of the satellite terminal where they are located and annotate the collected disaster images to obtain a disaster map. Specifically, in the application scenario of disaster map generation, the fine-tuning process can use a small number of already labeled disaster images (such as flood inundation areas, fire affected areas, etc.) and a large number of unlabeled image data as the local dataset. Through a combination of unsupervised contrast fine-tuning and supervised fine-tuning, the on-board model can learn the feature representations of disaster areas and perform high-precision class prediction on the image data collected by the satellite terminal after fine-tuning. For example, when a flood disaster occurs, the fine-tuned on-board model can quickly identify the flood inundation area and mark it on the map to generate a detailed disaster map. This high-precision annotation ability significantly improves the efficiency of disaster response, helps rescue personnel quickly locate the affected area, optimize the rescue route, and reduce disaster losses. This process not only demonstrates the strong adaptability of this solution in the case of scarce data labels, but also highlights its high efficiency and accuracy in disaster emergency response.

[0043] Through the technical solutions of the above embodiments, the satellite terminal can utilize unlabeled data to learn the feature representations therein in the case of scarce labeled data, reducing the dependence on labeled data. Subsequently, the supervised fine-tuning further improves the performance of the model on specific tasks, and a high annotation accuracy can be achieved even in the case of scarce labeled data. This method combining unsupervised contrast fine-tuning and supervised fine-tuning enables the model to better extract the feature representations in the image data and optimize using a small amount of labeled data, thus significantly improving the annotation accuracy. Moreover, adopting multiple rounds of collaborative fine-tuning and parameter aggregation enables the model to integrate the learning results of different satellite terminals, enhancing the generalization ability of the model. By sharing knowledge representations, the model can better adapt to the data characteristics of different satellites. In addition, through satellite-ground collaborative annotation fine-tuning, the model can adapt to the particularity of the satellite data where it is located. In summary, through this method of satellite-ground collaborative annotation fine-tuning, the present application effectively solves the technical problem of scarce satellite data labels, improves the annotation ability and overall performance of the model, and meets the requirements of satellite terminals in practical applications.

[0044] Optionally, in combination with the above embodiments, an embodiment of the present application further provides another satellite-ground collaborative annotation fine-tuning method for scarce data labels. In this method, after "multiple satellite terminals respectively receive the pre-trained on-board model sent by the ground station" in step S11, step S21 is further included, and step S12 includes step S12-1: Step S21, the multiple satellite terminals respectively split the pre-trained on-board model into an encoder network near the input layer and a decoder network near the output layer, and freeze the parameters of the decoder network.

[0045] In this embodiment, first, considering that in satellite image processing, the quality of the feature representations of the data directly determines the performance of the model, and the encoder can convert complex image data into more compact and robust feature representations through multi-layer feature extraction. These feature representations can capture the key information and patterns in the image data, thereby learning the basic features of each image data, such as edges, textures, and shapes, in a large amount of unlabeled data, which can provide strong support for the annotation task. Similarly, considering that in the process of satellite-ground collaborative annotation fine-tuning, the communication resources between the satellite terminal and the ground station are limited, and the transmission volume of model parameters will directly affect the communication cost. Based on these two considerations, the present application proposes a technical concept of splitting the pre-trained on-board model into two parts, an encoder network near the input layer for learning features and a decoder network near the output layer for outputting to downstream tasks, and only fine-tuning the encoder network during the unsupervised contrast fine-tuning using unlabeled data.

[0046] The decoder network does not participate in the unsupervised contrast fine-tuning process. Therefore, the parameters of the decoder network are frozen, which means that during the entire unsupervised contrast fine-tuning process, the parameters of the decoder networks of all satellite terminals will not change during the backpropagation update.

[0047] Step S12 includes: Step S12-1, in each round of collaborative annotation fine-tuning process, each of the multiple satellite terminals uses the unlabeled data in its local dataset to perform unsupervised contrast fine-tuning on the encoder network in the pre-trained on-board model, obtaining the fine-tuning parameters of the initial on-board model for this round. The fine-tuning parameters of the initial on-board model are the parameters of the fine-tuned encoder network.

[0048] In this embodiment, during each round of collaborative annotation fine-tuning, the encoder network can extract the feature representations in the expressionless data without labels and obtain the fine-tuning parameters of the initial on-board model for this round, that is, the parameters of the fine-tuned encoder network. Through the fine-tuning of the encoder network, the model can learn more robust feature representations on unlabeled data, providing a better basis for subsequent annotation tasks.

[0049] Among them, the first converged on-board model includes the parameters of the converged encoder network and the parameters of the frozen decoder network.

[0050] In this embodiment, after multiple rounds of collaborative annotation fine-tuning between the satellite terminal and the ground station, the satellite terminal will receive the parameters of the converged encoder network and obtain the first converged on-board model based on the parameters of the converged encoder network and the parameters of the frozen decoder network.

[0051] That is to say, in the unsupervised contrast fine-tuning stage, although only the parameters of the encoder network are updated, the finally obtained first converged on-board model still includes a complete encoder network and decoder network. In the subsequent supervised fine-tuning stage (Step S16), the satellite terminal will use the local labeled data to further optimize the entire first converged on-board model (including the encoder and decoder) to improve the performance of the on-board model in annotation tasks such as specific category prediction.

[0052] Through the technical solutions of the above embodiments, the pre-trained on-board model is split into an encoder and a decoder network, and the parameters of the decoder network are frozen, enabling the satellite terminal to focus on enhancing the extraction ability of the feature representation of the encoder network during the unsupervised contrast fine-tuning phase. This design allows the model to fully utilize unlabeled data to learn more robust feature representations, thereby enabling the on-board model to perform better in subsequent annotation tasks. Moreover, freezing the parameters of the decoder network reduces the computational overhead during the unsupervised contrast fine-tuning phase because only the parameters of the encoder network need to be updated. Meanwhile, during the parameter upload and aggregation process, only transmitting the parameters of the encoder network also reduces the communication load and improves the efficiency of the system.

[0053] Optionally, in combination with the above embodiments, an embodiment of the present application further provides another space-ground collaborative annotation fine-tuning method for scarce data labels. In this method, the step of "multiple satellite terminals send the fine-tuning parameters of their respective initial on-board models to the ground station" in step S13 specifically includes steps S13-1 to S13-2: Step S13-1, the central server deployed at the ground station randomly selects multiple satellite terminals from the multiple connectable satellite terminals included in the multiple satellite terminals as the satellite terminals participating in collaborative aggregation.

[0054] In this embodiment, during the process of collaborative annotation fine-tuning, considering that the communication between the satellite and the ground station is restricted by the satellite orbit and the ground station coverage area, that is, the satellite has a long revisit period and can only establish a communication connection with the ground station when the satellite orbit passes over the ground station. Therefore, the communication resources between the space and the ground are limited, and the long revisit period exacerbates the difficulty of the collaborative aggregation process of model parameters.

[0055] Based on this, during each round of collaborative annotation fine-tuning, although each satellite terminal will perform local unsupervised contrast fine-tuning, which satellite terminals need to transmit the fine-tuning parameters of the initial on-board model obtained after fine-tuning is determined by the ground station. Specifically, the central server deployed at the ground station needs to randomly select multiple satellite terminals from the connectable satellite terminals included in the multiple satellite terminals as the objects participating in the collaborative aggregation in this round. The fine-tuning parameters of the initial on-board models of these selected satellite terminals will be sent to the ground station within the communication window and participate in the aggregation process at the ground station, thereby ensuring the flexibility of the aggregation process and maximizing the utilization of available communication resources in the case of limited satellite numbers.

[0056] Step S13-2, the multiple satellite terminals participating in collaborative aggregation send the fine-tuning parameters of their respective initial on-board models to the ground station.

[0057] In this embodiment, after randomly selecting multiple satellite terminals participating in collaborative aggregation, these satellite terminals send the fine-tuning parameters of their respective initial on-board models to the ground station. These fine-tuning parameters are the encoder network parameters obtained after the satellite terminals perform unsupervised contrastive fine-tuning on the local unlabeled data. Sending these fine-tuning parameters enables the ground station to aggregate the learning results of different satellite terminals and generate more optimized global model parameters, that is, the fine-tuning parameters of the aggregated on-board model for this round.

[0058] Through the technical solution of the above embodiment, a random selection mechanism and a flexible communication strategy are introduced, further optimizing the process of space-ground collaborative annotation fine-tuning. It can not only effectively cope with the limitations of satellite communication but also achieve efficient model updates under limited communication resources. By randomly selecting currently connectable satellite terminals to participate in aggregation, the central server can maximize the utilization of available resources in each communication, while reducing communication overhead and improving the overall performance of the system. This strategy is particularly suitable for scenarios where satellite data labels are scarce and communication is restricted, and can significantly improve the efficiency and effect of space-ground collaborative fine-tuning.

[0059] Optionally, in combination with the above embodiment, an embodiment of the present application also provides another space-ground collaborative annotation fine-tuning method for scarce data labels. In this method, the step of "the ground station aggregates the fine-tuning parameters of the multiple initial on-board models received to obtain the fine-tuning parameters of the aggregated on-board model for this round" in step S14 specifically includes steps S14-1 to S14-2: Step S14-1, when the number of satellite terminals participating in collaborative aggregation is less than a preset quantity threshold, the ground station selects, from the fine-tuning parameters of the multiple aggregated on-board models obtained in the previous N rounds of collaborative annotation fine-tuning processes, the fine-tuning parameters of the aggregated on-board model that are most similar to the fine-tuning parameters of the multiple initial on-board models sent by the multiple satellite terminals participating in collaborative aggregation, where N≥1.

[0060] In this embodiment, during the process of collaborative annotation fine-tuning, when the number of satellite terminals participating in collaborative aggregation is less than the preset quantity threshold, the central server of the ground station will select, from the fine-tuning parameters of the multiple aggregated on-board models obtained in the previous N rounds of collaborative annotation fine-tuning processes, the fine-tuning parameters of the aggregated on-board model that are most similar to the fine-tuning parameters of the initial on-board model sent by the currently participating satellite terminals in collaborative aggregation (N≥1). Thus, it makes up for the lack of diversity and richness of update information caused by insufficient number of participating satellites (referring to the small number of fine-tuning parameters of the initial on-board model), and enhances the information richness and diversity of the current aggregation process by introducing historical aggregation parameters (referring to the most similar historical fine-tuning parameters of the aggregated on-board model).

[0061] Specifically, the central server of the ground station calculates the similarity (e.g., calculated by cosine similarity) between the fine-tuning parameters of the currently received initial on-board model and the fine-tuning parameters of multiple aggregated on-board models obtained during the previous N rounds of collaborative annotation fine-tuning processes, and selects the most similar parameters as compensation. This method can not only make full use of historical information but also optimize the update of the aggregated on-board model in the case of a small number of satellites participating in collaborative aggregation under the limitation of the long revisit period of the satellite.

[0062] Among them, the preset quantity threshold can be set according to the actual situation, such as according to the time length of the communication window between the satellite terminal and the ground station, the satellite orbit characteristics, the task requirements of the collaborative annotation fine-tuning process, the performance target of the finally expected on-board model, and / or historical data and experience.

[0063] Step S14-2: The ground station aggregates based on the fine-tuning parameters of the multiple initial on-board models sent by the multiple satellite terminals participating in collaborative aggregation received and the fine-tuning parameters of the most similar aggregated on-board model to obtain the fine-tuning parameters of the aggregated on-board model for this round.

[0064] Through the technical solution of the above embodiments, introducing the fine-tuning parameters of the historical aggregated on-board model as a compensation mechanism further optimizes the process of collaborative annotation fine-tuning. It can not only enhance the information richness and diversity of the aggregation process but also optimize the update of the global model in the case of limited satellite numbers, thereby achieving more efficient model updates with fewer communication times and improving the overall performance of the space-ground collaborative annotation fine-tuning system facing scarce data labels.

[0065] Optionally, in combination with the above embodiments, an embodiment of the present application also provides another space-ground collaborative annotation fine-tuning method for scarce data labels. In this method, the step of "multiple satellite terminals obtain the first converged on-board model" in step S15 specifically includes steps S15-1 to S15-2: Step S15-1: After the on-board model undergoes multiple rounds of collaborative annotation fine-tuning between the ground station and multiple satellite terminals, the multiple satellite terminals obtain the parameters of the converged encoder network sent by the ground station.

[0066] In this embodiment, during the process of collaborative annotation fine-tuning, after multiple rounds of collaborative annotation fine-tuning, the multiple satellite terminals will receive the parameters of the converged encoder network obtained through multiple rounds of collaborative aggregation from the ground station, indicating that the fine-tuning optimization of the on-board model within the global scope is completed.

[0067] Specifically, during multiple rounds of unsupervised contrast fine-tuning and parameter aggregation processes, the satellite terminals gradually optimize the parameters of the encoder network, enabling it to extract more robust and generalized feature representations for more accurate annotation tasks such as class prediction. Finally, the ground station distributes the parameters of the converged encoder network to each satellite terminal. The parameters of the converged encoder network represent the feature extraction ability of the on-board model after global optimization, providing a high-quality feature basis for subsequent annotation tasks.

[0068] Step S15-2, the multiple satellite terminals splice the received parameters of the converged encoder network with the parameters of the frozen decoder network to obtain the first converged on-board model.

[0069] In this embodiment, after receiving the parameters of the converged encoder network, the multiple satellite terminals splice these parameters with the locally frozen decoder network parameters to obtain a complete first converged on-board model. The parameters of the decoder network remain frozen during the previous fine-tuning process. Therefore, the parameters in the initialization or pre-training stage are directly used during splicing, thus forming a complete on-board model that can be used for actual annotation tasks. In this way, the satellite terminals can accurately predict the class of image data using the optimized model locally, thereby obtaining annotated image data.

[0070] Optionally, in combination with the above embodiments, an embodiment of the present application also provides another space-ground collaborative annotation fine-tuning method for scarce data labels. In this method, before the collaborative annotation fine-tuning process, it further includes step S31 and step S32: Step S31, set the initialization parameters and model structure to obtain the pre-trained on-board model.

[0071] In this embodiment, before the collaborative annotation fine-tuning, it is first necessary to set the initialization parameters and select a suitable model structure (such as manually selecting a specific model) to obtain the pre-trained on-board model.

[0072] Among them, the initialization parameters include the initial weights and biases of the model, and these parameters are randomly initialized. Then, select a suitable model structure, such as a lightweight convolutional neural network CNN or Transformer, to obtain the pre-trained model. The pre-trained on-board model is the basis of the entire collaborative fine-tuning process, providing a starting point for subsequent unsupervised contrast fine-tuning and supervised fine-tuning. By reasonably setting the initialization parameters and selecting the model structure, it can be ensured that the model can still effectively learn and optimize in the case of scarce labels.

[0073] Step S32: Set the first relevant information for participating in the collaboration process. The first relevant information includes: the number of the multiple satellite terminals, the frequency of each of the multiple satellite terminals participating in collaborative aggregation, the label scarcity degree of each local data set of the multiple satellite terminals and the label scarcity degree of the local data set, the deviation operator of each of the multiple satellite terminals, and the number of measurement fine-tuning rounds.

[0074] Among them, the label scarcity degree is used to represent the proportion of labeled data in the satellite terminal, and the label deviation degree is used to represent the distribution of labeled data among the multiple satellite terminals; the deviation operator represents an operation operator for simulating labeled data with different label deviation degrees on multiple satellite terminals; the number of measurement fine-tuning rounds represents the number of times of collaborative aggregation.

[0075] One of the purposes of this solution is to evaluate the ability of each satellite terminal to maintain the accuracy of the on-board model under different local data sets. Considering that the different label scarcity degrees and label deviation degrees of satellite data have a great impact on the final fine-tuning accuracy rate, based on this, this article defines the label scarcity degree and label deviation degree of the data labels of satellite terminals.

[0076] Specifically, in this embodiment, before the collaborative annotation fine-tuning starts, it is also necessary to set the first relevant information for participating in the collaboration process. These first relevant information include: 1. The number of satellite terminals: It represents how many satellites are determined to participate in the collaborative annotation fine-tuning process. The number of satellites will affect the data diversity and the speed of model update.

[0077] 2. The frequency of each satellite terminal participating in collaborative aggregation: By setting the frequency of each satellite participating in collaborative aggregation, for example, aggregating once every 6 hours or every 12 hours, the frequency selection can be adjusted according to the communication window between the satellite terminal and the ground station and / or the requirements of the annotation task.

[0078] 3. The label scarcity degree of the local data set: Define the proportion of labeled data of each satellite terminal. For example, the label scarcity degree of satellite terminal A is 10%, indicating that 10% of its local data set is labeled. This indicator reflects the data annotation situation of the image data contained in the local data set of the satellite terminal and has an important impact on the fine-tuning strategy of the model.

[0079] 4. The label scarcity degree of the local data set: It represents the distribution of labeled data among different satellite terminals. For example, some satellite terminals may have more labeled data, while some satellites may have almost none. Different label deviation degrees affect the balance of the model in the global optimization process.

[0080] 5. Deviation operator: An operator for operating on labeled data that simulates different label deviation degrees on multiple satellite terminals. These operators can help simulate the actual situation of satellite data more realistically, thereby optimizing the robustness of the model.

[0081] 6. Measure the number of fine-tuning rounds: Set the number of times of collaborative aggregation, that is, how many rounds of iteration are required for the entire fine-tuning process. The setting of the measured number of fine-tuning rounds needs to be adjusted according to the convergence situation of the model and task requirements.

[0082] Exemplarily, a series of initialization settings need to be performed before collaborative annotation fine-tuning. First, the ground station needs to set initialization parameters and select a suitable model structure to generate a pre-trained on-board model. For example, a lightweight CNN model can be selected and its parameters can be randomly initialized. Next, the first relevant information participating in the collaborative process needs to be set, including the number of satellite terminals, the respective frequencies of each of the multiple satellite terminals participating in collaborative aggregation, the label scarcity degree, the label deviation degree, the deviation operator, and the measured number of fine-tuning rounds. For example, assume that there are 5 satellites in the satellite constellation, and the label scarcity degrees of each satellite are 10%, 5%, 15%, 10%, and 5% respectively. The label deviation degree is simulated by the deviation operator. The respective frequencies of each of the multiple satellite terminals participating in collaborative aggregation are 6 hours / time, 7 hours / time, 10 hours / time, 2 hours / time, and 5 hours / time, and the measured number of fine-tuning rounds is 10. Then each satellite terminal corresponds to a satellite. Each satellite participates in a collaborative aggregation every time it reaches the corresponding time, and the entire collaborative annotation fine-tuning process performs 10 rounds of iteration. Through these settings, in the case of label scarcity, unlabeled data can be fully utilized for optimization, and the performance of the pre-trained on-board model can be improved through multiple rounds of collaborative aggregation.

[0083] Through the above embodiments, before performing collaborative annotation fine-tuning, by reasonably setting initialization parameters and selecting a suitable model structure, an efficient starting point can be provided for the process of collaborative annotation fine-tuning. The pre-trained on-board model can better learn and optimize in the case of label scarcity, thereby improving the efficiency and effect of collaborative annotation fine-tuning, as well as the overall performance of the on-board model.

[0084] Optionally, in combination with the above embodiments, an embodiment of the present application further provides another space-ground collaborative annotation fine-tuning method for data label scarcity. In this method, after "obtaining the on-board models of each of the multiple satellite terminals after collaborative annotation fine-tuning" in step S16, it further includes steps S41 to S47: Step S41, obtain the true labels of the first test data set, where the first test data set includes first unlabeled data and first labeled data.

[0085] In this embodiment, after obtaining the spaceborne model after multiple rounds of collaborative annotation fine-tuning, in order to evaluate the annotation performance of the model, it is first necessary to use a pre-prepared first test dataset for evaluation. The first test dataset contains multiple image data, which are divided into two types: first unlabeled data and first labeled data. The first unlabeled data refers to the unlabeled data in the first test dataset, and the first labeled data refers to the labeled data in the first test dataset.

[0086] The first test dataset is used to verify the annotation accuracy of the spaceborne model after collaborative annotation fine-tuning. The true label refers to the pre-annotated correct category of each image data in the first test dataset, and the true label is obtained by manual annotation or other reliable methods.

[0087] For example, the first test dataset contains 100 labeled image data (first labeled data) and 900 unlabeled image data (first unlabeled data). The true categories of these image data are known and are used for subsequent performance evaluation.

[0088] Step S42: Use the spaceborne model after collaborative annotation fine-tuning to annotate the first unlabeled data, and obtain the labeled first unlabeled data.

[0089] In this embodiment, use the spaceborne model after collaborative annotation fine-tuning to perform the annotation task of class prediction on the first unlabeled data in the first test dataset. In this process, the spaceborne model after collaborative annotation fine-tuning will predict the class label for each first unlabeled data according to the feature representation and classification ability it has learned. For example, assume that there are 900 unlabeled images in the first test dataset, and the model will generate prediction labels for these 900 images, thereby obtaining the labeled first unlabeled data.

[0090] Step S43: Calculate the annotation accuracy of the spaceborne model after collaborative annotation fine-tuning according to the true label of the first unlabeled data in the labeled first unlabeled data and the first test dataset. The annotation accuracy of the spaceborne model after collaborative annotation fine-tuning is: the proportion of the first unlabeled data with correct annotation in the labeled first unlabeled data to all the first unlabeled data.

[0091] In this embodiment, according to the first unlabeled data after labeling and the true labels of the first unlabeled data in the first test dataset, the labeling accuracy of the spaceborne model after collaborative annotation fine-tuning is calculated. The labeling accuracy refers to the proportion of the correctly labeled data among all the unlabeled data after labeling. For example, if the model correctly labels 810 out of 900 unlabeled images, the labeling accuracy is 810 / 900 = 90%. This metric reflects the labeling performance of the spaceborne model after collaborative annotation fine-tuning on unlabeled data and is an important reference for evaluating the generalization ability of the model.

[0092] Step S44: Mix the first unlabeled data after labeling and the first labeled data to obtain mixed labeled data.

[0093] In this embodiment, the first unlabeled data after labeling and the first labeled data in the first test dataset are mixed to obtain mixed labeled data. The purpose of this step is to combine the first unlabeled data already labeled by the model with the existing first labeled data to form a richer labeled dataset. For example, 900 unlabeled images after labeling are mixed with 100 labeled images to obtain a mixed labeled dataset containing 1000 images. The mixed dataset can be used to further optimize the model.

[0094] Step S45: Use the mixed labeled data to further fine-tune the spaceborne model after collaborative annotation fine-tuning to obtain a first spaceborne model, where the first spaceborne model is the spaceborne model after collaborative annotation fine-tuning further fine-tuned with the mixed labeled data.

[0095] In this embodiment, the spaceborne model after collaborative annotation fine-tuning is further fine-tuned using the mixed labeled data, and the resulting model is called the first spaceborne model. In this process, by introducing the first unlabeled data labeled by the spaceborne model after collaborative annotation fine-tuning, the performance of the model is further optimized. The mixed labeled data not only contains the original first labeled data but also adds the first unlabeled data labeled by the model, thus providing more learning samples for the model. By fine-tuning on the mixed labeled data, the spaceborne model after collaborative annotation fine-tuning can learn more rich feature representations and improve its labeling ability.

[0096] Step S46: Calculate the system accuracy of the first spaceborne model through the second test dataset; the system accuracy of the first spaceborne model is; after using the mixed labeled data to further fine-tune the first spaceborne model, the accuracy rate that the first spaceborne model can achieve after labeling the second test dataset; the second test dataset contains second unlabeled data and second labeled data.

[0097] In this embodiment, another test data set, i.e., the second test data, is further introduced, and the system accuracy rate of the first spaceborne model is calculated through the second test data set. The second test data set includes second unlabeled data and second labeled data, and is used to evaluate the performance of the first spaceborne model on unseen data. The system accuracy rate refers to the correct rate that can be achieved after the first spaceborne model annotates the second test data set. For example, if there are 1000 images in the second test data set and the model correctly annotates 920 of them, the system accuracy rate is 92%. This indicator reflects the overall performance of the model on new data and is a key indicator for evaluating the generalization ability and practical application value of the model.

[0098] Step S47, based on the annotation accuracy rate of the spaceborne model after collaborative annotation fine-tuning and the system accuracy rate of the first spaceborne model, evaluate the performance improvement of the spaceborne model after collaborative annotation fine-tuning.

[0099] In this embodiment, based on the annotation accuracy rate of the spaceborne model after collaborative annotation fine-tuning and the system accuracy rate of the first spaceborne model, evaluate the performance improvement of the spaceborne model after collaborative annotation fine-tuning. Here, the performance improvement refers to the improvement of the accuracy rate of annotating image data. By analyzing the annotation accuracy rate and the system accuracy rate, the improvement effect of the model performance can be evaluated. If the system accuracy rate is significantly higher than the annotation accuracy rate, it indicates that the fine-tuning process effectively improves the performance of the model and makes it more suitable for actual annotation tasks.

[0100] Through the technical solutions of the above embodiments, for the obtained spaceborne model after collaborative annotation fine-tuning, by calculating the annotation accuracy rate, the annotation ability of the model on unlabeled data can be intuitively evaluated. A high annotation accuracy rate indicates that the model can effectively utilize unlabeled data for learning and improve its generalization ability. Then, using the mixed labeled data for further fine-tuning can further optimize the performance of the model. The mixed labeled data not only includes the original labeled data but also adds the unlabeled data annotated by the model, thus providing more learning samples for the model and improving its annotation ability. Further, by calculating the system accuracy rate on the second test data set, the overall performance of the model on unseen data can be evaluated. A high system accuracy rate indicates that the model has strong generalization ability and can maintain a high annotation accuracy rate on new data. Finally, by analyzing the annotation accuracy rate and the system accuracy rate, the improvement effect of the performance of the spaceborne model after collaborative annotation fine-tuning can be intuitively evaluated.

[0101] Optionally, in combination with the above embodiments, an embodiment of the present application further provides another space-ground collaborative annotation fine-tuning method for scarce data labels. In this method, before "evaluating the performance improvement of the spaceborne model after collaborative annotation fine-tuning" in step S47, steps S51 to S513 are further included. Step S47 specifically includes step S47-1: Step S51, set the initialization parameters and model structure to obtain the spaceborne model to be fine-tuned.

[0102] In this embodiment, before evaluating the performance improvement of the spaceborne model after collaborative annotation fine-tuning, it is necessary to formulate a scarce simulation measurement scheme for spaceborne data labels to measure the ability of the spaceborne model to maintain annotation accuracy under different label data conditions. Therefore, it is first necessary to set the initialization parameters and select a suitable model structure to obtain the spaceborne model to be fine-tuned.

[0103] The initialization parameters include the initial weights and biases of the model, which are usually randomly initialized or obtained from a pre-trained model. Select a suitable model structure, a lightweight convolutional neural network CNN or Transformer.

[0104] Step S52, set the second relevant information participating in the collaborative process. The second relevant information includes: the number of multiple satellite terminals, the frequency of each satellite terminal participating in the collaborative process, the label scarcity degree and label deviation degree of the local training data set of each satellite terminal, the deviation operator of each satellite terminal, and the measurement fine-tuning rounds.

[0105] Before the collaborative process, it is necessary to set the second relevant information, which can refer to the setting process of the first relevant information.

[0106] Step S53, each of the multiple satellite terminals participating in the collaborative process uses the third unlabeled data in the local training data set to perform unsupervised contrast fine-tuning on the spaceborne model to be fine-tuned, and obtains the fine-tuning parameters of the current fine-tuned spaceborne model.

[0107] In this embodiment, each of the multiple satellite terminals participating in the collaborative process uses the third unlabeled data in the local training data set to perform unsupervised contrast fine-tuning on the spaceborne model that has not been fine-tuned by collaborative annotation. Through the way of unsupervised contrast learning, the model can extract the feature representation in the unlabeled data without labels and obtain the fine-tuning parameters of the current fine-tuned spaceborne model.

[0108] Step S54, the central server of the ground station randomly selects multiple satellite terminals from the multiple satellite terminals participating in the collaboration process as the satellite terminals participating in collaborative aggregation, and aggregates the fine-tuning parameters of the multiple current fine-tuned on-board models obtained by unsupervised contrast fine-tuning of the selected multiple satellite terminals participating in collaborative aggregation to obtain the fine-tuning parameters of the current aggregated on-board model.

[0109] In this embodiment, the central server of the ground station randomly selects multiple satellite terminals from the multiple satellite terminals participating in the collaboration process as the satellite terminals participating in collaborative aggregation, and aggregates the fine-tuning parameters of the multiple current fine-tuned on-board models obtained by unsupervised contrast fine-tuning of the selected multiple satellite terminals participating in collaborative aggregation to obtain the fine-tuning parameters of the current aggregated on-board model. The aggregation process usually adopts methods such as simple averaging or weighted averaging to fuse parameters from different sources. In this way, the central server can integrate the learning results of multiple satellite terminals to generate more optimized global model parameters.

[0110] Step S55, the ground sends the fine-tuning parameters of the current aggregated on-board model to multiple satellite terminals for the next round of collaboration process.

[0111] In this embodiment, the ground station sends the fine-tuning parameters of the current aggregated on-board model to multiple satellite terminals for the next round of collaboration process, ensuring that different satellite terminals can perform further fine-tuning based on the latest global model parameters.

[0112] Step S56, after multiple rounds of collaboration process, each of the multiple satellite terminals participating in the collaboration process obtains a second converged on-board model.

[0113] In this embodiment, after multiple rounds of collaboration process, each of the multiple satellite terminals participating in the collaboration process obtains a second converged on-board model. The second converged on-board model is a model obtained after multiple rounds of unsupervised contrast fine-tuning and parameter aggregation, and its model parameters have tended to be stable, with good feature extraction ability and generalization ability.

[0114] Step S57, each of the multiple satellite terminals participating in the collaboration process uses the labeled data in the local training dataset to perform local fine-tuning on the second converged on-board model to obtain a fine-tuned on-board model.

[0115] In this embodiment, each of the multiple satellite terminals participating in the collaboration process uses the third labeled data in the local training dataset to perform local fine-tuning on the second converged on-board model to obtain a fine-tuned on-board model. The purpose of local fine-tuning is to further optimize the performance of the model using a small amount of labeled data, so that the on-board model can better adapt to the specific annotation tasks in the satellite terminal.

[0116] Step S58. The multiple satellite terminals participating in the collaboration process perform class prediction on the third unlabeled data in the local training dataset by fine-tuning the on-board model, and label the third unlabeled data based on the class prediction results to obtain the labeled third unlabeled data.

[0117] In this embodiment, the purpose of the prediction process of the fine-tuned on-board model for unlabeled data is to generate labeled data for subsequent calculation of the labeling accuracy.

[0118] Step S59. Determine the labeling accuracy of the fine-tuned on-board model according to the true labels of the third unlabeled data in the local training dataset and the labeled third unlabeled data.

[0119] In this embodiment, the labeling accuracy of the fine-tuned on-board model is determined according to the true labels of the third unlabeled data in the local training dataset and the labeled third unlabeled data. The labeling accuracy of the fine-tuned on-board model is the proportion of the correctly labeled third unlabeled data in all the third unlabeled data.

[0120] Step S510. The multiple satellite terminals participating in the collaboration process respectively use the labeled third unlabeled data in the local training dataset and the third labeled data in the local training dataset to perform supervised fine-tuning on the fine-tuned on-board model to obtain the final fine-tuned on-board model.

[0121] In this embodiment, the performance of the model is further optimized by using the labeled unlabeled data and the labeled data to form mixed labeled data, so that it can better adapt to the specific labeling task.

[0122] Step S511. The multiple satellite terminals participating in the collaboration process respectively use the third test dataset to test the accuracy of the final fine-tuned on-board model to obtain the system accuracy of the final fine-tuned on-board model.

[0123] In this embodiment, the multiple satellite terminals participating in the collaboration process respectively use the local third test dataset to test the accuracy of the final fine-tuned on-board model to obtain the system accuracy of the final fine-tuned on-board model. The system accuracy of the final fine-tuned on-board model refers to the correct rate that can be achieved after the final fine-tuned on-board model labels the third test dataset, so as to reflect the comprehensive performance of the final fine-tuned on-board model obtained through the collaboration process on the mixed labeled data under the current setting of the second relevant information.

[0124] Step S512: Reset the tag scarcity degree and tag deviation degree in the second relevant information, and repeat the above steps to obtain the corresponding relationship between the annotation accuracy and the system accuracy of the finally fine-tuned on-board model under different tag scarcity degrees and different tag deviation degrees.

[0125] In this embodiment, reset the tag scarcity degree and tag deviation degree in the second relevant information participating in the collaboration process, and repeat the process from step S53 to step S511 above to obtain the corresponding relationship between the annotation accuracy and the system accuracy of the finally fine-tuned on-board model under different tag scarcity degrees and different tag deviation degrees. Through multiple tests, the performance change of the on-board model after the collaboration process under different tag scarcity degrees and tag deviation degrees can be analyzed, providing data support for the performance improvement of the subsequent collaborative annotation fine-tuning process optimization model.

[0126] "Evaluating the performance improvement of the on-board model after collaborative annotation fine-tuning" in step S47 includes: step S47-1, determining the performance improvement degree of the on-board model after collaborative annotation fine-tuning based on the corresponding relationship, the annotation accuracy of the on-board model after collaborative annotation fine-tuning, and the system accuracy of the on-board model after collaborative annotation fine-tuning.

[0127] In this embodiment, based on the above corresponding relationship, the annotation accuracy and system accuracy of the on-board model after collaborative annotation fine-tuning, determine the performance improvement degree of the on-board model after collaborative annotation fine-tuning. By comparing the annotation accuracy and system accuracy under different conditions, the improvement effect of the model performance through the collaborative annotation fine-tuning process in the previous text can be intuitively evaluated, ensuring higher accuracy of the model in actual annotation tasks such as category prediction.

[0128] Through the above various embodiments, the core invention points and advantages of this application are as follows: 1. Designed a fine-tuning scheme for an on-board model based on the combination of an encoder and a decoder guided by a small number of tags. By freezing the parameters of the decoder network of the on-board model at the satellite terminal and only updating the parameters of the encoder network of the on-board model, it saves about 20% of the fine-tuning time, reduces the required system overhead for fine-tuning, and greatly improves the operation efficiency of the space-ground collaboration system.

[0129] 2. By only sharing the model parameters of the encoder network during the space-ground collaboration aggregation process, the method can learn the on-board data representations of each satellite terminal, and the decoder network does not need to participate in the coordination aggregation process, thus saving more than 10% of the data transmission volume during the transmission process and reducing the communication load overhead of the space-ground link.

[0130] 3. A full simulation was conducted on the scarcity of data labels and the degree of label deviation in the on-orbit environment of the satellite terminal, and the impact of label scarcity and label deviation on the performance of the onboard model was further explored, and the performance improvement of the onboard model obtained by subsequent collaborative labeling fine-tuning was evaluated by these two indicators. Based on the same inventive concept, another embodiment of the present application also provides a satellite-ground collaborative labeling fine-tuning system for satellite data label scarcity, the system comprising multiple satellite terminals and ground stations: The multiple satellite terminals are used to respectively receive pre-trained satellite models sent by the ground station, wherein the pre-trained satellite models are obtained after parameter initialization of the model to be fine-tuned pre-trained by the ground station, and the model to be fine-tuned has the function of performing category prediction on image data collected by any satellite terminal; In each round of collaborative labeling fine-tuning, the multiple satellite terminals are used to perform unsupervised comparative fine-tuning on the pre-trained satellite model using unlabeled data in the local data set to obtain fine-tuning parameters of the initial satellite model of this round; The multiple satellite terminals are used to send the fine-tuning parameters of their respective initial onboard models to the ground station; The ground station is used to aggregate the fine-tuning parameters of the received multiple initial onboard models to obtain the fine-tuning parameters of the aggregated onboard model of this round, and send the fine-tuning parameters of the aggregated onboard model to the multiple satellite terminals; After the pre-trained onboard model is fine-tuned through multiple rounds of collaborative annotation between the ground station and the plurality of satellite terminals, the plurality of satellite terminals are configured to receive a first converged onboard model, wherein the first converged onboard model is a model parameter having a generalized knowledge representation; The multiple satellite terminals are used to perform supervised fine-tuning on the first converged satellite-borne model using the labeled data in the local data set, respectively, to obtain the collaboratively labeled fine-tuned satellite-borne models of the multiple satellite terminals, and the collaboratively labeled fine-tuned satellite-borne model of each satellite terminal is used to perform category prediction on the image data collected by the satellite terminal, so as to obtain labeled image data.

[0131] Optionally, it also includes: The multiple satellite terminals are used to split the pre-trained satellite models into an encoder network close to the input layer and a decoder network close to the output layer after the multiple satellite terminals respectively receive the pre-trained satellite models sent by the ground station, and freeze the parameters of the decoder network; The multiple satellite terminals are used to respectively utilize the unlabeled data in the local datasets during each round of collaborative annotation fine-tuning to perform unsupervised contrastive fine-tuning on the pre-trained on-board model, so as to obtain the fine-tuning parameters of the initial on-board model in this round, including: during each round of collaborative annotation fine-tuning, the multiple satellite terminals respectively utilize the unlabeled data in the local datasets to perform unsupervised contrastive fine-tuning on the encoder network in the pre-trained on-board model, so as to obtain the fine-tuning parameters of the initial on-board model in this round, and the fine-tuning parameters of the initial on-board model are the parameters of the fine-tuned encoder network; Among them, the first converged on-board model includes the parameters of the converged encoder network and the parameters of the frozen decoder network.

[0132] Optionally, the multiple satellite terminals are used to send the fine-tuning parameters of their respective initial on-board models to the ground station, including: The ground station is used to randomly select multiple satellite terminals as the satellite terminals participating in collaborative aggregation from the multiple satellite terminals that can establish connections included in the multiple satellite terminals through the deployed central server; The multiple satellite terminals participating in collaborative aggregation are used to send the fine-tuning parameters of their respective initial on-board models to the ground station.

[0133] Optionally, the ground station aggregates the fine-tuning parameters of the multiple initial on-board models received to obtain the fine-tuning parameters of the aggregated on-board model in this round, including: The ground station is used to, when the number of satellite terminals participating in collaborative aggregation is less than the preset quantity threshold, select the fine-tuning parameters of the aggregated on-board model that are most similar to the fine-tuning parameters of the multiple initial on-board models sent by the multiple satellite terminals participating in collaborative aggregation to the ground station from the fine-tuning parameters of the multiple aggregated on-board models obtained in the previous N rounds of collaborative annotation fine-tuning processes, where N≥1; The ground station is used to perform aggregation based on the fine-tuning parameters of the multiple initial on-board models sent by the multiple satellite terminals participating in collaborative aggregation received and the fine-tuning parameters of the most similar aggregated on-board model to obtain the fine-tuning parameters of the aggregated on-board model in this round.

[0134] The multiple satellite terminals are used to obtain the first converged on-board model, including: The multiple satellite terminals are used to obtain the parameters of the converged encoder network sent by the ground station after the on-board model undergoes multiple rounds of collaborative annotation fine-tuning between the ground station and the multiple satellite terminals; The multiple satellite terminals are used to splice the received parameters of the converged encoder network with the parameters of the frozen decoder network to obtain the first converged on-board model.

[0135] The system further includes: The ground station is configured to set initialization parameters and a model structure before the collaborative annotation fine-tuning process to obtain the pre-trained on-board model; The ground station is configured to set first relevant information participating in the collaborative process, where the first relevant information includes: the number of the multiple satellite terminals, the respective frequencies of the multiple satellite terminals participating in collaborative aggregation, the label scarcity degrees of the respective local data sets of the multiple satellite terminals and the label scarcity degree of the local data sets, the respective deviation operators of the multiple satellite terminals, and the measurement fine-tuning rounds; Wherein, the label scarcity degree is used to represent the proportion of labeled data in the satellite terminal, and the label deviation degree is used to represent the distribution of the labeled data among the multiple satellite terminals; the deviation operator represents an operation operator for simulating labeled data with different label deviation degrees on the multiple satellite terminals; the measurement fine-tuning rounds represent the number of times of collaborative aggregation.

[0136] Optionally, the system further includes: The multiple satellite terminals are configured to obtain the true labels of the first test data set after obtaining the on-board models of the multiple satellite terminals after collaborative annotation fine-tuning, where the first test data set includes first unlabeled data and first labeled data; The multiple satellite terminals are configured to label the first unlabeled data through the on-board model after collaborative annotation fine-tuning to obtain the labeled first unlabeled data; Each satellite terminal of the multiple satellite terminals is configured to calculate the annotation accuracy of the on-board model after collaborative annotation fine-tuning according to the true labels of the first unlabeled data in the labeled first unlabeled data and the first test data set, where the annotation accuracy of the on-board model after collaborative annotation fine-tuning is: the proportion of the first unlabeled data with correct labels in the labeled first unlabeled data to all the first unlabeled data; Each satellite terminal of the multiple satellite terminals is configured to mix the labeled first unlabeled data and the first labeled data to obtain mixed labeled data; Each satellite terminal of the multiple satellite terminals is configured to further fine-tune the on-board model after collaborative annotation fine-tuning through the mixed labeled data to obtain a first on-board model, where the first on-board model is to calculate the system accuracy of the first on-board model through a second test data set after further fine-tuning through the mixed labeled data; the system accuracy of the first on-board model is: the correct rate that can be achieved after the first on-board model annotates the second test data set after further fine-tuning the first on-board model with the mixed labeled data; The ground station is used to evaluate the performance improvement of the spaceborne model after collaborative annotation fine-tuning based on the annotation accuracy of the spaceborne model after collaborative annotation fine-tuning and the system accuracy of the first spaceborne model.

[0137] The system further includes: The ground station is used to set initialization parameters and a model structure before evaluating the performance improvement of the spaceborne model after collaborative annotation fine-tuning, so as to obtain a spaceborne model to be fine-tuned. The ground station is used to set second relevant information participating in the collaborative process, where the second relevant information includes: the number of multiple satellite terminals, the frequency of each satellite terminal participating in the collaborative process, the label scarcity degree and label deviation degree of the local training data set of each satellite terminal, the deviation operator of each satellite terminal, and the measurement fine-tuning rounds. The multiple satellite terminals participating in the collaborative process are used to respectively use the third unlabeled data in the local training data set to perform unsupervised contrast fine-tuning on the spaceborne model to be fine-tuned, so as to obtain the fine-tuning parameters of the current fine-tuned spaceborne model. The central server of the ground station is used to randomly select multiple satellite terminals from the multiple satellite terminals participating in the collaborative process as satellite terminals participating in collaborative aggregation, and aggregate the fine-tuning parameters of the multiple current fine-tuned spaceborne models obtained by the selected multiple satellite terminals participating in collaborative aggregation through unsupervised contrast fine-tuning, so as to obtain the fine-tuning parameters of the current aggregated spaceborne model. The ground is used to send the fine-tuning parameters of the current aggregated spaceborne model to multiple satellite terminals for the next round of collaborative process. The multiple satellite terminals participating in the collaborative process are used to obtain a second converged spaceborne model after multiple rounds of collaborative process. The multiple satellite terminals participating in the collaborative process are used to respectively use the labeled data in the local training data set to perform local fine-tuning on the second converged spaceborne model, so as to obtain a fine-tuned spaceborne model. The multiple satellite terminals participating in the collaborative process are used to perform class prediction on the third unlabeled data in the local training data set through the fine-tuned spaceborne model, and based on the result of the class prediction, annotate the third unlabeled data to obtain the labeled third unlabeled data. The multiple satellite terminals participating in the collaborative process are used to determine the annotation accuracy of the fine-tuned spaceborne model according to the true label of the third unlabeled data in the local training data set and the labeled third unlabeled data. The multiple satellite terminals participating in the collaborative process are used to respectively perform supervised fine-tuning on the fine-tuned on-board model by using the third unlabeled data labeled in the local training dataset and the third labeled data in the local training dataset, so as to obtain the final fine-tuned on-board model; The multiple satellite terminals participating in the collaborative process are used to respectively test the accuracy rate of the final fine-tuned on-board model by using the third test dataset, so as to obtain the system accuracy rate of the final fine-tuned on-board model; The ground station is used to reset the second relevant information and repeat the above steps to obtain the corresponding relationship between the annotation accuracy rate and the system accuracy rate of the final fine-tuned on-board model under different label scarcity degrees and different label deviation degrees; The ground station is used to evaluate the performance improvement of the on-board model after collaborative annotation fine-tuning, including: the ground station is used to determine the performance improvement degree of the on-board model after collaborative annotation fine-tuning based on the corresponding relationship, the annotation accuracy rate of the on-board model after collaborative annotation fine-tuning, and the system accuracy rate of the on-board model after collaborative annotation fine-tuning.

[0138] Based on the same inventive concept, another embodiment of the present application further provides an electronic device, including a memory, a processor, and a computer program stored on the memory. Wherein, the processor executes the computer program to implement the satellite-ground collaborative annotation fine-tuning method for satellite data label scarcity as described in any one of the above embodiments.

[0139] Among them, the electronic device refers to Figure 3 , Figure 3 is a schematic diagram of an electronic device provided in an embodiment of the present application. As Figure 3 shown, the electronic device 300 includes: a memory 310 and a processor 320. The memory 310 and the processor 320 are communicatively connected through a bus. A computer program is stored in the memory 310, and the computer program can run on the processor 320, thereby implementing the steps in the method disclosed in the above embodiments of the present application.

[0140] For the system, since it is basically similar to the method embodiment, the description is relatively simple. For the relevant parts, refer to the partial description of the method embodiment.

[0141] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, apparatus, or computer program product. Therefore, the embodiments of the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code. The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing terminal devices generate a device for implementing the functions specified in the process Figure 1 one process or multiple processes and / or blocks Figure 1 or multiple blocks. These computer program instructions can also be stored in a computer-readable memory capable of guiding a computer or other programmable data processing terminal devices to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the functions specified in the process Figure 1 one process or multiple processes and / or blocks Figure 1 or multiple blocks. These computer program instructions can also be loaded onto a computer or other programmable data processing terminal devices, such that a series of operation steps are executed on the computer or other programmable terminal devices to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable terminal devices provide steps for implementing the functions specified in the process Figure 1 one process or multiple processes and / or blocks Figure 1 or multiple blocks. Although the preferred embodiments of the present application have been described, those skilled in the art can make additional changes and modifications to these embodiments once they learn the basic creative concepts. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0142] Finally, it should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or terminal device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent in such process, method, article or terminal device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or terminal device comprising the element.

[0143] The above has introduced in detail a space-ground collaborative annotation fine-tuning method, system and device for satellite data with scarce labels provided by this application. Specific examples are used in this text to elaborate on the principle and implementation manner of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application; at the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to this application.

Claims

1. A space-ground collaborative annotation fine-tuning method for scarce data labels, characterized in that, The method includes: Multiple satellite terminals respectively receive a pre-trained on-board model sent by a ground station. The pre-trained on-board model is obtained after initializing the parameters of a to-be-fine-tuned model pre-trained by the ground station. The to-be-fine-tuned model has the function of predicting the category of image data collected by any satellite terminal. In each round of collaborative annotation fine-tuning process, the multiple satellite terminals respectively use the unlabeled data in their local datasets to perform unsupervised contrast fine-tuning on the pre-trained on-board model to obtain the fine-tuning parameters of the initial on-board model in this round. The multiple satellite terminals send the fine-tuning parameters of their respective initial on-board models to the ground station. The ground station aggregates the fine-tuning parameters of the multiple initial on-board models received to obtain the fine-tuning parameters of the aggregated on-board model in this round, and sends the fine-tuning parameters of the aggregated on-board model to the multiple satellite terminals. After the pre-trained on-board model undergoes multiple rounds of collaborative annotation fine-tuning between the ground station and the multiple satellite terminals, the multiple satellite terminals obtain a first converged on-board model. The first converged on-board model is the model parameters with generalized knowledge representation. The multiple satellite terminals respectively use the labeled data in their local datasets to perform supervised fine-tuning on the first converged on-board model to obtain the on-board models after collaborative annotation fine-tuning for each of the multiple satellite terminals. The on-board model after collaborative annotation fine-tuning for each satellite terminal is used to predict the category of the image data collected by this satellite terminal to obtain labeled image data.

2. The satellite-ground collaborative annotation fine-tuning method for scarce data labels according to claim 1, wherein After the multiple satellite terminals respectively receive the pre-trained on-board model sent by the ground station, it further includes: The multiple satellite terminals respectively split the pre-trained on-board model into an encoder network close to the input layer and a decoder network close to the output layer, and freeze the parameters of the decoder network. In each round of collaborative annotation fine-tuning process, the multiple satellite terminals respectively use the unlabeled data in their local datasets to perform unsupervised contrast fine-tuning on the pre-trained on-board model to obtain the fine-tuning parameters of the initial on-board model in this round, including: in each round of collaborative annotation fine-tuning process, the multiple satellite terminals respectively use the unlabeled data in their local datasets to perform unsupervised contrast fine-tuning on the encoder network in the pre-trained on-board model to obtain the fine-tuning parameters of the initial on-board model in this round. The fine-tuning parameters of the initial on-board model are the parameters of the fine-tuned encoder network. Among them, the first converged on-board model includes the parameters of the converged encoder network and the parameters of the frozen decoder network.

3. The satellite-ground collaborative annotation fine-tuning method for scarce data labels according to claim 1, characterized in that The multiple satellite terminals send the fine-tuning parameters of their respective initial on-board models to the ground station, including: The central server deployed by the ground station randomly selects multiple satellite terminals from the multiple connectable satellite terminals included in the multiple satellite terminals as the satellite terminals participating in collaborative aggregation. The multiple satellite terminals participating in collaborative aggregation send the fine-tuning parameters of their respective initial on-board models to the ground station.

4. The satellite-ground collaborative annotation fine-tuning method for scarce data labels according to claim 3, wherein The ground station aggregates the fine-tuning parameters of the multiple initial on-board models received to obtain the fine-tuning parameters of the aggregated on-board model in this round, including: When the number of satellite terminals participating in collaborative aggregation is less than a preset quantity threshold, the ground station selects, from the fine-tuning parameters of multiple aggregated spaceborne models obtained from the previous N rounds of collaborative annotation fine-tuning processes, the fine-tuning parameters of the aggregated spaceborne model that are most similar to the fine-tuning parameters of the multiple initial spaceborne models sent by the multiple satellite terminals participating in collaborative aggregation to the ground station, where N≥1; The ground station aggregates based on the fine-tuning parameters of the multiple initial spaceborne models sent by the multiple satellite terminals participating in collaborative aggregation received and the most similar fine-tuning parameters of the aggregated spaceborne model to obtain the fine-tuning parameters of the aggregated spaceborne model for this round.

5. The method for fine-tuning satellite-ground collaborative annotation for scarce data labels according to claim 2, wherein The multiple satellite terminals obtaining the first converged spaceborne model includes: After the spaceborne model undergoes multiple rounds of collaborative annotation fine-tuning between the ground station and the multiple satellite terminals, the multiple satellite terminals obtain the parameters of the converged encoder network sent by the ground station; The multiple satellite terminals splice the received parameters of the converged encoder network with the parameters of the frozen decoder network to obtain the first converged spaceborne model.

6. The space-ground collaborative annotation fine-tuning method for scarce data labels according to claim 1, wherein Before the collaborative annotation fine-tuning process, it further includes: Setting initialization parameters and a model structure to obtain the pre-trained spaceborne model; Setting first relevant information for participating in the collaborative process, where the first relevant information includes: the number of the multiple satellite terminals, the frequency of each of the multiple satellite terminals participating in collaborative aggregation, the label scarcity degree of the local datasets of each of the multiple satellite terminals and the label scarcity degree of the local datasets, the deviation operators of each of the multiple satellite terminals, and the measurement fine-tuning rounds; Among them, the label scarcity degree is used to represent the proportion of labeled data in the satellite terminal, and the label deviation degree is used to represent the distribution of labeled data among the multiple satellite terminals; the deviation operator represents an operator for simulating labeled data with different label deviation degrees on the multiple satellite terminals; the measurement fine-tuning rounds represent the number of times of collaborative aggregation.

7. The satellite-ground collaborative annotation fine-tuning method for scarce data labels according to claim 2, wherein After obtaining the spaceborne models of the multiple satellite terminals after collaborative annotation fine-tuning, it further includes: Obtaining the true labels of a first test dataset, where the first test dataset contains first unlabeled data and first labeled data; Using the spaceborne model after collaborative annotation fine-tuning to label the first unlabeled data to obtain the labeled first unlabeled data; According to the true labels of the first unlabeled data in the labeled first unlabeled data and the true labels of the first test dataset, calculating the labeling accuracy of the spaceborne model after collaborative annotation fine-tuning, where the labeling accuracy of the spaceborne model after collaborative annotation fine-tuning is: the proportion of the correctly labeled first unlabeled data in the labeled first unlabeled data to all the first unlabeled data; Mixing the labeled first unlabeled data and the first labeled data to obtain mixed labeled data; Using the mixed labeled data, the spaceborne model after fine-tuning by collaborative annotation is further fine-tuned to obtain a first spaceborne model. The first spaceborne model is obtained by further fine-tuning with the mixed labeled data and then calculating the system accuracy rate of the first spaceborne model through a second test data set. The system accuracy rate of the first spaceborne model is the accuracy rate that can be achieved after the first spaceborne model is further fine-tuned with the mixed labeled data and then annotates the second test data set. Based on the annotation accuracy rate of the spaceborne model after fine-tuning by collaborative annotation and the system accuracy rate of the first spaceborne model, the performance improvement of the spaceborne model after fine-tuning by collaborative annotation is evaluated.

8. The method for fine-tuning satellite-ground collaborative annotation for scarce data labels according to claim 7, characterized in that Before evaluating the performance improvement of the spaceborne model after fine-tuning by collaborative annotation, it further includes: Setting initialization parameters and a model structure to obtain a spaceborne model to be fine-tuned. Setting second relevant information participating in the collaborative process. The second relevant information includes: the number of multiple satellite terminals, the frequency of each satellite terminal participating in the collaborative process, the label scarcity degree and label deviation degree of the local training data set of each satellite terminal, the deviation operator of each satellite terminal, and the measurement fine-tuning rounds. Each of the multiple satellite terminals participating in the collaborative process uses the third unlabeled data in the local training data set to perform unsupervised contrast fine-tuning on the spaceborne model to be fine-tuned, and obtains the fine-tuning parameters of the current fine-tuned spaceborne model. The central server of the ground station randomly selects multiple satellite terminals from the multiple satellite terminals participating in the collaborative process as satellite terminals participating in collaborative aggregation, and aggregates the fine-tuning parameters of the multiple current fine-tuned spaceborne models obtained by the selected multiple satellite terminals participating in collaborative aggregation through unsupervised contrast fine-tuning to obtain the fine-tuning parameters of the current aggregated spaceborne model. The ground sends the fine-tuning parameters of the current aggregated spaceborne model to multiple satellite terminals for the next round of collaborative process. After multiple rounds of collaborative processes, each of the multiple satellite terminals participating in the collaborative process obtains a second converged spaceborne model. Each of the multiple satellite terminals participating in the collaborative process uses the labeled data in the local training data set to perform local fine-tuning on the second converged spaceborne model to obtain a fine-tuned spaceborne model. Each of the multiple satellite terminals participating in the collaborative process uses the fine-tuned spaceborne model to perform class prediction on the third unlabeled data in the local training data set, and based on the result of the class prediction, annotates the third unlabeled data to obtain labeled third unlabeled data. According to the true label of the third unlabeled data in the local training data set and the labeled third unlabeled data, the annotation accuracy rate of the fine-tuned spaceborne model is determined. Each of the multiple satellite terminals participating in the collaborative process uses the labeled third unlabeled data in the local training data set and the third labeled data in the local training data set to perform supervised fine-tuning on the fine-tuned spaceborne model to obtain a final fine-tuned spaceborne model. Each of the multiple satellite terminals participating in the collaborative process uses the third test data set to test the accuracy of the finally fine-tuned on-board model, and obtains the system accuracy of the finally fine-tuned on-board model; Reset the second relevant information, and repeat the above steps to obtain the corresponding relationship between the annotation accuracy and the system accuracy of the finally fine-tuned on-board model under different label scarcity degrees and different label deviation degrees; The evaluation of the performance improvement of the on-board model after collaborative annotation fine-tuning includes: determining the degree of performance improvement of the on-board model after collaborative annotation fine-tuning based on the corresponding relationship, the annotation accuracy of the on-board model after collaborative annotation fine-tuning, and the system accuracy of the on-board model after collaborative annotation fine-tuning.

9. A space-ground collaborative annotation fine-tuning system for scarce satellite data labels, characterized in that, The system includes multiple satellite terminals and a ground station: The multiple satellite terminals are used to respectively receive the pre-trained on-board model sent by the ground station. The pre-trained on-board model is obtained after parameter initialization of the to-be-fine-tuned model pre-trained by the ground station. The to-be-fine-tuned model has the function of predicting the category of image data collected by any satellite terminal; In each round of collaborative annotation fine-tuning process, the multiple satellite terminals are used to respectively use the unlabeled data in the local data set to perform unsupervised contrast fine-tuning on the pre-trained on-board model to obtain the fine-tuning parameters of the initial on-board model in this round; The multiple satellite terminals are used to send the fine-tuning parameters of their respective initial on-board models to the ground station; The ground station is used to aggregate the fine-tuning parameters of the multiple initial on-board models received to obtain the fine-tuning parameters of the aggregated on-board model in this round, and send the fine-tuning parameters of the aggregated on-board model to the multiple satellite terminals; After the pre-trained on-board model undergoes multiple rounds of collaborative annotation fine-tuning between the ground station and the multiple satellite terminals, the multiple satellite terminals are used to receive the first converged on-board model, and the first converged on-board model is the model parameters with generalized knowledge representation; The multiple satellite terminals are used to respectively use the labeled data in the local data set to perform supervised fine-tuning on the first converged on-board model to obtain the on-board models after collaborative annotation fine-tuning of the multiple satellite terminals. The on-board model after collaborative annotation fine-tuning of each satellite terminal is used to predict the category of the image data collected by the satellite terminal to obtain labeled image data.

10. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory. Among them, the processor executes the computer program to implement the space-ground collaborative annotation fine-tuning method for data label scarcity as described in any one of claims 1-8.

Images