A method, apparatus, electronic device, and medium for evaluating the performance of a remote sensing model.

Abstract

This application discloses a method, apparatus, electronic device, and medium for evaluating the performance of remote sensing models. The method includes: acquiring model meta-information of the model to be evaluated, including historical tasks, datasets, and historical index scores; the historical index scores are used to characterize the accuracy of the model to be evaluated in performing corresponding historical tasks based on the dataset; based on the model meta-information, the historical index scores of the dataset on the corresponding historical tasks are normalized to obtain a data-level normalized score; the data-level normalized scores corresponding to the same historical tasks are normalized again to obtain a task-level normalized score; based on the task-level normalized score, the comprehensive score of the model to be evaluated is determined; the comprehensive score is used to characterize the comprehensive performance of the model to be evaluated; the higher the comprehensive score, the better the comprehensive performance. Therefore, multi-level normalization can suppress the bias caused by extremely high scores on a small number of tasks, improve the comparability and fairness between models, and achieve efficient and accurate model evaluation.

Description

Technical Field

[0001] This application relates to the field of remote sensing technology, and in particular to a method, apparatus, electronic device, and medium for evaluating the performance of a remote sensing model. Background Technology

[0002] With the rapid deployment of remote sensing satellites and the widespread application of Earth observation in environmental monitoring, urban planning, disaster response, and other fields, massive amounts of multi-source, multi-temporal remote sensing imagery data are emerging at an unprecedented rate. Against this backdrop, efficiently and intelligently processing and understanding remote sensing images has become a key challenge.

[0003] In recent years, deep learning technology, especially large pre-trained models (such as visual Transformer and multimodal large models), has made significant progress in the fields of computer vision and remote sensing image analysis, demonstrating strong performance advantages in remote sensing image tasks such as image classification, object detection, semantic segmentation, and change detection.

[0004] To measure the performance of remote sensing models, various model evaluation mechanisms and benchmark platforms have been gradually developed. However, most existing solutions focus on performance testing for single tasks or limited task types, lacking a systematic evaluation of the multi-task collaborative capabilities of large remote sensing models in complex application scenarios. Furthermore, the fragmented evaluation metrics, heterogeneous datasets, and inconsistent evaluation standards make it difficult to conduct cross-model performance comparisons.

[0005] Therefore, how to comprehensively evaluate the multi-task collaborative capabilities of remote sensing models and achieve horizontal comparisons between different models is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of this, one aspect of this application provides a method for evaluating the performance of a remote sensing model, the method comprising:

[0007] Obtain the model meta-information of the model to be evaluated; the model meta-information includes historical tasks, datasets, and historical metric scores; the historical metric scores are used to characterize the accuracy of the model to be evaluated in performing corresponding historical tasks based on the dataset;

[0008] Based on the model meta-information, the historical indicator scores of the dataset on the corresponding historical task are normalized to obtain the data-level normalized score.

[0009] The data-level normalized scores corresponding to the same historical tasks are normalized again to obtain the task-level normalized scores.

[0010] The comprehensive score of the model to be evaluated is determined based on the task-level normalized score; the comprehensive score is used to characterize the overall performance of the model to be evaluated; the higher the comprehensive score, the better the overall performance.

[0011] Optionally, the step of normalizing the historical indicator scores of the dataset on the corresponding historical task based on the model meta-information to obtain a data-level normalized score includes:

[0012] Obtain the pre-constructed mapping relationship; the mapping relationship includes the correspondence between downstream tasks, evaluation indicators, and indicator weights;

[0013] Based on the model meta-information and the mapping relationship, determine the target indicator weight corresponding to the historical indicator score;

[0014] Based on the target indicator weights and the historical indicator scores, the weighted score of the dataset on the corresponding historical task is determined;

[0015] The weighted scores corresponding to the same dataset and the same historical task are normalized to obtain the data-level normalized score.

[0016] Optionally, when the evaluation index corresponding to the historical index score is the average precision or the mean of the average precision, the weight of the target index is determined according to a preset correspondence, and the sum of the weights of each target index is 1; the preset correspondence is the correspondence between the range of intersection-union ratio values ​​and the index weights.

[0017] Optionally, the step of further normalizing the data-level normalized scores corresponding to the same historical tasks to obtain task-level normalized scores includes:

[0018] Determine the target average value of the data-level normalized score corresponding to the model to be evaluated under the same historical task;

[0019] The average target value corresponding to the same historical task is normalized to obtain the task-level normalized score.

[0020] Optionally, the mapping relationship also includes the correspondence between the downstream task and the capability dimension; the capability dimension is used to describe the model's performance on the corresponding downstream task.

[0021] Before determining the comprehensive score of the model to be evaluated based on the task-level normalized score, the method further includes:

[0022] Based on the mapping relationship, the target capability dimension corresponding to the historical task is determined;

[0023] Obtain the first number of historical task types corresponding to the target capability dimension, and the second number of all downstream task types corresponding to each capability dimension in the mapping relationship;

[0024] Determine the ratio of the first quantity to the second quantity;

[0025] Based on the task-level normalized score and the ratio, the capability dimension score of the model to be evaluated on each capability dimension is determined; the capability dimension score is used to describe the performance of the model to be evaluated on the corresponding capability dimension, and the higher the capability dimension score, the better the performance of the model to be evaluated on the corresponding capability dimension.

[0026] Optionally, determining the comprehensive score of the model to be evaluated based on the task-level normalized score includes:

[0027] Obtain the third number of the number of dimension types included in the target capability dimension, the fourth number of the number of dimension types included in the capability dimension, the fifth number of the number of task types included in the historical task, and the sixth number of the number of task types included in the downstream task;

[0028] Based on the third and fourth quantities, the dimensional coverage of the model to be evaluated is determined;

[0029] Based on the fifth and sixth quantities, the task coverage of the model to be evaluated is determined;

[0030] The comprehensive score is determined based on the capability dimension score, the dimension coverage, and the task coverage; both the dimension coverage and the task coverage are positively correlated with the comprehensive score.

[0031] Optionally, the method for evaluating the performance of the remote sensing model further includes:

[0032] The task-level normalized scores under different historical tasks and / or the data-level normalized scores under different datasets are sorted to obtain a task ranking list;

[0033] Based on the mapping relationship and the capability dimension score, the capability dimension scores under each capability dimension are sorted to obtain a capability ranking list;

[0034] The overall scores are sorted to obtain the final ranking.

[0035] The task rankings, ability rankings, and overall rankings are transmitted to the target terminal for display.

[0036] Another aspect of this application provides an apparatus for evaluating the performance of a remote sensing model, the apparatus comprising:

[0037] The meta-information acquisition module is used to acquire the model meta-information of the model to be evaluated; the model meta-information includes historical tasks, datasets, and historical metric scores; the historical metric scores are used to characterize the accuracy of the model to be evaluated in performing corresponding historical tasks based on the dataset;

[0038] The first normalization module is used to normalize the historical index scores of the dataset on the corresponding historical task based on the model meta information to obtain the data-level normalized score.

[0039] The second normalization module is used to normalize the data-level normalization scores corresponding to the same historical tasks again to obtain the task-level normalization scores.

[0040] The comprehensive score determination module is used to determine the comprehensive score of the model to be evaluated based on the task-level normalized score; the comprehensive score is used to characterize the comprehensive performance of the model to be evaluated; the higher the comprehensive score, the better the comprehensive performance.

[0041] Another aspect of this application provides an electronic device including a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor executes the computer program to implement the steps of the method for evaluating the performance of the remote sensing model.

[0042] Another aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for evaluating the performance of the remote sensing model.

[0043] The remote sensing model performance evaluation method, apparatus, electronic device, and medium provided in this application offer the following benefits: Based on multi-level normalization at both the dataset and task levels, it enables cross-dataset and cross-task model evaluation without manual intervention, ensuring both efficiency and accuracy. Simultaneously, multi-level normalization can suppress bias caused by extremely high scores on a small number of tasks, improving the comparability and fairness of scoring between models. Building upon the efficient and accurate evaluation of large remote sensing models, it allows for horizontal comparisons between different models based on data-level normalized scores, task-level normalized scores, and comprehensive scores, providing a model data foundation for practical applications in various scenarios. Attached Figure Description

[0044] Figure 1 A flowchart illustrating a method for evaluating the performance of a remote sensing model provided in an embodiment of this application;

[0045] Figure 2 A flowchart illustrating a method for evaluating the performance of a remote sensing model, provided as another embodiment of this application;

[0046] Figure 3 A schematic diagram illustrating the principle of a method for evaluating the performance of a remote sensing model provided in an embodiment of this application;

[0047] Figure 4 A schematic diagram of the structure of a remote sensing model performance evaluation device provided in an embodiment of this application;

[0048] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0049] The reference numerals in the attached diagram are as follows: 50 is memory, 51 is processor, 52 is display screen, 53 is input / output interface, 54 is communication interface, 55 is power supply, 56 is communication bus, 501 is computer program, 502 is operating system, and 503 is data. Detailed Implementation

[0050] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0051] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0052] Figure 1 This is a flowchart illustrating a method for evaluating the performance of a remote sensing model provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes:

[0053] S10: Obtain the model meta-information of the model to be evaluated; the model meta-information includes historical tasks, datasets, and historical metric scores; the historical metric scores are used to characterize the accuracy of the model to be evaluated in performing the corresponding historical tasks based on the dataset;

[0054] In a specific embodiment, to achieve automatic model evaluation, the model metadata of the model to be evaluated can be directly obtained. In an optional embodiment, the model file to be evaluated uploaded by the user can also be obtained; this model file can be a compressed model archive. Further, the model file is decompressed and parsed to extract model metadata from its internal structured files.

[0055] In one optional embodiment, the model metadata may include, but is not limited to, historical tasks, datasets, historical metric scores, historical evaluation metrics, model names, task identifiers, model IDs, model types, and team names. This application does not limit the type of model metadata obtained.

[0056] It should be noted that each time the model performance is evaluated, it can be performed on one model or multiple models; this application does not limit this. Furthermore, the model to be evaluated can be a large model in the remote sensing vertical domain; this application also does not limit the type of model to be evaluated.

[0057] Furthermore, it should be noted that the historical tasks in the model metadata refer to tasks previously performed by the model under evaluation, such as image classification and object detection. Historical evaluation metrics are the evaluation parameters for the corresponding historical tasks, and the historical metric scores are used to evaluate the accuracy of the model under evaluation when performing the corresponding historical tasks based on the dataset. For example, for historical image classification tasks, the corresponding historical evaluation metric could be accuracy, resulting in an accuracy score; that is, the historical metric score can characterize the accuracy of image classification.

[0058] Table 1 is a schematic table of model metadata provided in the embodiments of this application. For ease of understanding, the acquisition of model metadata will be explained below in conjunction with Table 1.

[0059] Table 1. Schematic diagram of a model metadata

[0060]

[0061] In one optional embodiment, multiple different large remote sensing models (including public and private models) are evaluated on 147 different datasets. As shown in Table 1, after parsing the user-uploaded model files, information can be obtained from the internal structured files, which may include, but is not limited to, field names, Chinese names, types, and specific content. According to Table 1, multiple model metadata information about the model to be evaluated can be obtained, such as model ID, model name, model category, number of parameters, release date, open type, and team name.

[0062] In a specific embodiment, model management is the core of model evaluation. To facilitate the management and maintenance of model information, in an optional embodiment, model metadata is built in MySQL, and when the model table is created or modified, the system will automatically synchronize and update the information to the MongoDB model table. This enables the recording and maintenance of historical tasks, datasets, historical metric scores, historical evaluation metrics, model names, task identifiers, model IDs, model types, and other metadata, thereby achieving systematic management and traceable evaluation of multi-task, multi-dimensional models.

[0063] S11: Based on the model meta-information, normalize the historical index scores of the dataset on the corresponding historical task to obtain the data-level normalized score.

[0064] S12: Normalize the data-level normalized scores corresponding to the same historical tasks again to obtain the task-level normalized scores.

[0065] Furthermore, to avoid some models having excessively high historical scores on a small number of tasks, i.e., inflated scores that would affect the fairness of horizontal comparisons between models, an optional embodiment implements a multi-layered normalization mechanism. Specifically, first, data normalization is performed at the dataset level based on model metadata, and then task-level data normalization is performed on top of the dataset-level normalization.

[0066] When performing dataset-level data normalization, the same dataset performing the same historical task on different models to be evaluated will yield a corresponding historical metric score. For example, for model A to be evaluated, performing an image classification task on dataset S1, the historical metric score corresponding to Accuracy is 95 (out of 100). Therefore, the historical metric scores of each dataset on the corresponding historical task can be normalized using the Min-Max method to obtain a dataset-level data-level normalized score.

[0067] Based on the data-level normalized score, the next level of normalization is performed at the task level. Specifically, for the same historical task, different datasets can yield a corresponding data-level normalized score. Therefore, the data-level normalized scores under the same historical task are normalized to obtain the task-level normalized score.

[0068] It should be noted that multi-level normalized scoring refers to a systematic method that decomposes a complex system or object into a hierarchical structure, normalizes the indicators at each level, and then weights and sums them to obtain a dimensionless, comparable comprehensive score. Therefore, by normalizing data, the "unfairness" caused by different dimensions and value ranges can be eliminated, enabling unified evaluation of models with different tasks and evaluation dimensions.

[0069] S13: Determine the overall score of the model to be evaluated based on the task-level normalized score; the overall score is used to characterize the overall performance of the model to be evaluated; the higher the overall score, the better the overall performance.

[0070] Furthermore, in an optional embodiment, a comprehensive score for evaluating the overall performance of the evaluation model can be obtained based on the task-level normalized score. Specifically, in an optional embodiment, the average of the task-level normalized scores over all historical tasks is calculated, and a pre-constructed mapping relationship is obtained, which includes the correspondence between downstream tasks and evaluation metrics.

[0071] Based on the mapping relationship, the task coverage rate of the model to be evaluated is determined. Task coverage rate refers to the ratio of the number of historical task types executed by the model to the total number of downstream task types in the mapping relationship. Therefore, the comprehensive score can be determined based on this ratio and the average of the task-level normalized scores. The comprehensive score is directly proportional to both the average of the task-level normalized scores and this ratio.

[0072] In this embodiment, a multi-level normalized scoring and task coverage mechanism is used to suppress the "artificially high" phenomenon where some models achieve high scores on a small number of tasks, thereby ensuring the scoring fairness in multi-task and multi-dimensional scenarios and improving the fairness of model evaluation.

[0073] In one optional embodiment, the data-level normalized score, task-level normalized score, and comprehensive score can be sent to the terminal for sorting and display, so that users can quickly obtain the performance of the model under evaluation in different dimensions, as well as the comprehensive performance in all task dimensions. It should be noted that this application does not limit the display method on the terminal, and it can be in various forms such as tables, line graphs, and bar charts.

[0074] Therefore, the remote sensing model performance evaluation method provided in this application, with its upload-and-evaluate mechanism, automatically parses the model compressed package / result file, identifies the task, and determines the comprehensive score, requiring no additional user configuration and significantly improving evaluation efficiency. Specifically, based on multi-level normalization at the dataset and task levels, it enables cross-dataset and cross-task model evaluation without manual intervention, ensuring both efficiency and accuracy. Simultaneously, multi-level normalization can suppress bias caused by extremely high scores on a small number of tasks, improving the comparability and fairness of scoring between models. Based on the efficient and accurate evaluation of large remote sensing models, horizontal comparisons between different models can be achieved according to the data-level normalized score, task-level normalized score, and comprehensive score, providing a model data foundation for practical applications in different scenarios.

[0075] Figure 2 This is a flowchart illustrating a method for evaluating the performance of a remote sensing model, provided as another embodiment of this application. In one optional embodiment, such as... Figure 2 As shown, based on the model meta-information, the historical indicator scores of the dataset on the corresponding historical tasks are normalized to obtain data-level normalized scores, including:

[0076] S110: Obtain the pre-built mapping relationship; the mapping relationship includes the correspondence between downstream tasks, evaluation metrics, and metric weights;

[0077] In a specific embodiment, to achieve fast and accurate calculation of data-level normalized scores, it is necessary to obtain specific calculation parameters related to model meta-information. Specifically, the correspondence between downstream tasks, evaluation metrics, and metric weights is pre-constructed, and the constructed mapping relationship is stored. This storage can be in tabular or text format; this application does not limit the storage method.

[0078] Table 2 is a schematic table of a mapping relationship provided in the embodiments of this application. For ease of understanding, the following will be explained in conjunction with Table 2.

[0079] Table 2. A schematic diagram of a mapping relationship.

[0080]

[0081]

[0082] As shown in Table 2, in one optional embodiment, when the model to be evaluated is a large remote sensing model in the field of remote sensing, the downstream tasks include, but are not limited to, 15 tasks such as image classification, horizontal bounding box classification, rotated bounding box classification, and polygonal region classification. In a specific embodiment, different downstream tasks can be represented using different task identifiers. Therefore, when obtaining model metadata, the row containing the corresponding downstream task can be located in Table 2 based on the identifier corresponding to the historical task, thereby obtaining the model-related information of the corresponding historical task.

[0083] Furthermore, as shown in Table 2, evaluation metrics may include, but are not limited to, accuracy and average precision (AP). Table 2 shows that different evaluation metrics correspond to different weights, or different downstream tasks correspond to different weights.

[0084] It should be noted that Table 2 represents one real-time relationship between downstream tasks, evaluation indicators, and indicator weights. This application does not impose any restrictions on the specific selection of downstream tasks, evaluation indicators, and indicator weights, or on their corresponding relationships.

[0085] Furthermore, it should be noted that in Table 2, Consensus-based Image Description Evaluation (CIDEr) is an automated evaluation metric specifically designed to assess the performance of image captioning models. It measures the generation quality by calculating the similarity between the candidate caption generated by the model and a set of human-generated reference captions; a higher CIDEr is better.

[0086] In addition, the mean Intersection over Union (mIoU) measures the degree of overlap between the predicted bounding box and the ground truth bounding box; a higher mIoU is better. The mean Average Precision (mAP) measures the overall performance of the detector across all classes; a higher mAP is better. Accuracy is one of the most basic and commonly used evaluation metrics in classification tasks, measuring the proportion of correct predictions made by the model; a higher accuracy is better.

[0087] S111: Determine the target indicator weights corresponding to historical indicator scores based on model meta-information and mapping relationships;

[0088] Based on the above embodiments, and according to the mapping relationship shown in Table 2, the target indicator weights corresponding to historical indicator scores can be determined by querying Table 2 based on the actual model meta-information of the model to be evaluated obtained through parsing. Specifically, the identifiers of the corresponding downstream tasks are first queried in Table 2 based on the identifiers of the historical tasks in the model meta-information, thereby obtaining the corresponding target weight indicators.

[0089] In an optional embodiment, when the evaluation metric corresponding to the historical metric score is the average precision (AP) or the mean average precision (mAP), the target metric weight is determined according to a preset correspondence, and the sum of the target metric weights is 1. The preset correspondence is the correspondence between the range of intersection-union ratio values ​​and the metric weights.

[0090] As shown in Table 2, in this specific embodiment, all evaluation metrics except AP and mAP have a weight of 1. The calculation of AP and mAP depends on the range of the Intersection over Union (IoU). IoU measures the degree of overlap between the predicted bounding box and the ground truth bounding box. In this specific embodiment, a predicted bounding box is considered correctly detected only when the IoU is greater than a preset threshold.

[0091] In one alternative embodiment, Therefore, the correspondence between the IoU value range and the indicator weight can be preset. For example, in Table 2, the evaluation indicator corresponding to the downstream task horizontal bounding box target detection includes "AP@0.25", which is used to indicate that the evaluation indicator is AP and the IoU value range is greater than or equal to 0.25, and the corresponding target indicator weight is 0.3. It should be noted that in this embodiment, "@" is used in combination to represent the correspondence between the evaluation indicator and the IoU value range for exemplary demonstration. This application does not limit the storage method of the preset correspondence.

[0092] In one optional embodiment, when the IoU value is greater than or equal to 0.25, it indicates that the predicted bounding box roughly covers the target. When the IoU value is greater than or equal to 0.5, it indicates that overall detection is performed. When the IoU value is greater than or equal to 0.75, it indicates that the predicted bounding box needs to fit the target very precisely.

[0093] Therefore, by adopting a unified mapping relationship between downstream tasks and indicator weights, combined with a normalization method, the scores of different models on different tasks and datasets can be directly compared.

[0094] S112: Determine the weighted score of the dataset on the corresponding historical task based on the target indicator weight and historical indicator scores;

[0095] Furthermore, a comprehensive capability evaluation is conducted on all large remote sensing models (i.e., the models to be evaluated) across different historical tasks. Each task corresponds to several datasets. Based on the target indicator weights and historical indicator scores obtained above, the weighted score of the dataset on the corresponding historical task can be calculated. That is, the weighted score of the dataset on the corresponding historical task is calculated. The model to be evaluated is based on the first The dataset is used for the first... The weighted score corresponding to each historical task. In an optional embodiment, it can be calculated according to formula (1):

[0096] (1)

[0097] in, For the weighted score, that is, the first... The dataset in the th On the model to be evaluated, the first The historical task, corresponding to the first The weighted score of the historical index scores for each evaluation indicator. The historical index scores are used to characterize the performance of the model being evaluated in the [number]th [stage]. The accuracy of historical tasks.

[0098] The target weight index is the first one. The weight indicators corresponding to each evaluation indicator, in a specific embodiment, can be seen from Table 2, are the target weight indicators. There may be one or more. In one alternative embodiment, the weighting metric is set proportionally to the dataset size. The historical indicator score, i.e., the first The historical scores corresponding to each evaluation indicator. Therefore, Each dataset can be calculated using formula (1). Weighted score .

[0099] Table 3 is a schematic table of model metadata provided in another embodiment of this application. For ease of understanding, examples will be given below in conjunction with Table 3.

[0100] Table 3. A schematic table of model metadata.

[0101]

[0102] In an optional embodiment, after parsing the model file, the model meta-information shown in Table 3 can be obtained. For a certain dataset, the target weight index corresponding to the horizontal bounding box object detection task performed on the model to be evaluated 1-Falcon is 0.3, and the historical index score is 0.89. The weight corresponding to AP@0.5 is 0.74, and the historical index score is 0.74. The weight corresponding to AP@0.75 is 0.46, and the historical index score is 0.46.

[0103] The weighted score corresponding to this dataset can be calculated as follows: That is, the weight score calculated for this dataset when performing the fifth historical task on the first model to be evaluated is 0.701.

[0104] Therefore, in this embodiment of the application, by performing a weighted calculation based on the target indicator weight and the historical indicator score, the coarse detection capability and high-precision performance can be effectively balanced, thereby improving the model evaluation accuracy.

[0105] S113: Normalize the weighted scores corresponding to the same dataset and the same historical tasks to obtain the data-level normalized score.

[0106] Based on the above, Each dataset can be calculated Weighted score Next, we need to... Weighted score The data is normalized according to the Min-Max principle to obtain the data-level normalized score. See formula (2):

[0107] (2)

[0108] in, For data-level normalized scores, that is, The dataset in the th On the model to be evaluated, the first The weighted score corresponding to each historical task. The score is obtained after normalization. That is, for... Weighted score The score is normalized.

[0109] In a specific embodiment, for Weighted score After normalization, we can obtain Data-level normalized score For example, if the same task is performed on the same model to be evaluated, and there are 5 datasets, 5 weighted scores can be calculated. After normalizing these 5 weighted scores, 5 data-level normalized scores can be obtained. If 8 models to be evaluated are performed on the same task, and there are 3 datasets, each model to be evaluated can obtain 3 data-level normalized scores, for a total of 24 data-level normalized scores.

[0110] for Weighted score The maximum value in, for Weighted score The minimum value in. In one optional embodiment, the normalized benchmark score can be set to 80. Of course, the normalized benchmark score... The settings can be configured according to actual business needs, and this application does not impose any restrictions on this.

[0111] Table 4 is a schematic table of model metadata provided in another embodiment of this application; Table 5 is a schematic table of the result of normalizing the weighted scores in Table 4; and Table 6 is a schematic table of the result of normalizing the weighted scores in Table 4. For ease of understanding, examples will be provided below in conjunction with Tables 4 to 6.

[0112] Table 4. A schematic table of model metadata.

[0113]

[0114] In one optional embodiment, as shown in Table 4, the models to be evaluated include the Falcon model, the GeoChat model, and the RemoteCLIP model, and the datasets include the OPTIMAL31 dataset, the RSOD dataset, and the SODA-A dataset. The OPTIMAL31 and RSOD datasets have historical image classification tasks performed on all three models to be evaluated, and their corresponding historical metric scores are shown in Table 4. The SODA-A dataset has historical object counting tasks performed on all three models to be evaluated, and their corresponding historical metric scores are also shown in Table 4.

[0115] Table 5. Schematic diagram of one result of normalization treatment.

[0116]

[0117] Based on Table 4, and referring to Table 5, the weighted scores for the historical tasks corresponding to the OPTIMAL31 and RSOD datasets are first calculated using formula (1). In an optional embodiment, referring to Table 2, since the target index weight for image classification is 1, the calculated weighted scores are the same as those in Table 4. Furthermore, the data-level normalized scores for each dataset on different models to be evaluated can be calculated according to formula (2). For example, for the OPTIMAL31 dataset, the data-level normalized score on the Falcon model is 80 points.

[0118] Table 6. Another schematic diagram of the results after normalization.

[0119]

[0120] Similarly, based on Table 4 and referring to Table 6, the target counting historical task corresponding to the SODA-A dataset first calculates the weighted score using formula (1), and then calculates the data-level normalized score using formula (2). As shown in Tables 4 to 6, for the same historical task, each dataset can be normalized to obtain a data-level normalized score on different evaluation models.

[0121] In one optional embodiment, the data-level normalized scores corresponding to the same historical tasks are normalized again to obtain task-level normalized scores, including:

[0122] Determine the target average value of the data-level normalized score corresponding to the model to be evaluated under the same historical task;

[0123] The average target value corresponding to the same historical task is normalized to obtain the task-level normalized score.

[0124] In a specific embodiment, based on the data-level normalized score, the target average value is calculated for a certain historical task, and the target average value is further processed by Min-Max normalization.

[0125] It should be noted that the target average can be any of the harmonic mean, arithmetic mean, and geometric mean. In an optional embodiment, because the harmonic mean is sensitive to minimum values ​​and can accurately handle rate and ratio data, it avoids the overestimation problem of the arithmetic mean. Therefore, the target average can be the harmonic mean.

[0126] In an alternative embodiment, the harmonic mean can be calculated according to formula (3):

[0127] (3)

[0128] in, The harmonic mean, i.e. The model to be evaluated is in the first... The harmonic mean over a number of historical tasks can be calculated. One harmonic mean (the same number as the historical tasks). For the first The model to be evaluated The total number of datasets corresponding to each historical task. For example, in the examples shown in Table 4 above, the datasets include the OPTIMAL31 dataset, the RSOD dataset, and the SODA-A dataset. Therefore, the total number of datasets... The value is 3. To avoid small constants that divide by zero, they can be set according to actual business needs. In one optional embodiment, .

[0129] Table 7 is a schematic table showing the results of calculating the target average value of the task-level normalized scores in Table 5. For ease of understanding, examples will be provided below in conjunction with Table 7.

[0130] Table 7. A schematic diagram of one result for calculating the target average.

[0131]

[0132] In the above embodiments, as shown in Tables 5 and 7, for historical tasks of image classification, both the OPTIMAL31 dataset and the RSOD dataset include corresponding data-level normalized scores on the same evaluation model. Therefore, the harmonic mean of the data-level normalized scores corresponding to the same evaluation model and the same historical task can be calculated according to formula (3). By calculating the data-level normalized scores shown in Table 5, the harmonic mean shown in Table 7 can be obtained.

[0133] Furthermore, based on the harmonic mean, the harmonic mean of the same historical tasks is normalized to obtain the task-level normalized score. In an optional embodiment, it can be calculated using formula (4):

[0134] (4)

[0135] in, For the task-level normalized score, that is, the first... The model to be evaluated is in the first... The score after normalizing the harmonic mean of the historical tasks ( (Normalized score of each harmonic mean). The maximum value in the harmonic mean. It is the minimum value in the harmonic mean.

[0136] Table 8 is a schematic diagram of the results after normalizing the target average value in Table 7. For ease of understanding, the following explanation will refer to Table 8.

[0137] Table 8. A schematic diagram of the results after normalizing the target average.

[0138]

[0139] As shown in Table 8, the harmonic mean obtained in Table 7 is normalized, that is, the harmonic mean between different models to be evaluated for the same historical task is normalized to obtain the task-level normalized score.

[0140] Therefore, in this embodiment, the harmonic mean is calculated first, and then normalization is performed. The reciprocal space amplifies the shortcomings, and the 0 to 1 interval flattens the scale. This allows the minimum value to have the power of life and death, and allows indicators of different dimensions to compete on the same stage, that is, to maintain comparability at the scaling stage.

[0141] Figure 3 This is a schematic diagram illustrating the principle of a remote sensing model performance evaluation method provided in an embodiment of this application. In an optional embodiment, the mapping relationship further includes the correspondence between downstream tasks and capability dimensions; the capability dimension is used to describe the model's performance on the corresponding downstream task.

[0142] As an optional embodiment, the method further includes, before determining the overall score of the model to be evaluated based on the task-level normalized score:

[0143] Based on the mapping relationship, determine the target capability dimensions corresponding to the historical tasks;

[0144] Obtain the first number of historical task types corresponding to the target capability dimension, and the second number of all downstream task types corresponding to each capability dimension in the mapping relationship;

[0145] Determine the ratio of the first quantity to the second quantity;

[0146] Based on the task-level normalized score and ratio, the capability dimension score of the model to be evaluated is determined in each capability dimension. The capability dimension score is used to describe the performance of the model to be evaluated in the corresponding capability dimension. The higher the capability dimension score, the better the performance of the model to be evaluated in the corresponding capability dimension.

[0147] like Figure 3 As shown, based on the data-level normalization and task-level normalization of the above embodiments, see [link to previous section]. Figure 3 Based on the model meta-information, the weighted score of the dataset on the corresponding historical task is first calculated, and then the weighted score of the dataset on the corresponding historical task is normalized to obtain the data-level normalized score, thus completing the data-level normalization.

[0148] Then, based on the data-level normalized scores, the harmonic mean of the data-level normalized scores for the model to be evaluated under the same historical tasks is first determined. Further, the harmonic mean for the same historical tasks is normalized to obtain the task-level normalized score, thus completing the task-level normalization.

[0149] It is understood that, in specific embodiments, different models exhibit different capabilities in performing different historical tasks. Therefore, based on the above embodiments, as an optional embodiment, such as... Figure 3 As shown, the capability dimension scores of the model to be evaluated can be calculated on each capability dimension. Table 9 is a schematic table of a mapping relationship provided by another embodiment of this application. For ease of understanding, the following explanation will be based on Table 9.

[0150] Table 9. A schematic diagram of a mapping relationship.

[0151]

[0152] For example, referring to Table 2, for image classification, the corresponding capability ID is 1. Referring to Table 9, the capability dimension corresponding to capability ID 1 is global visual perception capability. It should be noted that Tables 1 and 9 can be combined into one table or divided into two tables. In fact, in the mapping relationship, the correspondence between each pair of information can be stored in one table, or all information can be stored in one table; this application does not limit this.

[0153] In a specific embodiment, when determining the capability dimension score of the model to be evaluated, the target capability dimension corresponding to the historical task in the model meta-information can be determined according to the mapping relationship shown in Tables 2 and 9. For example, for the historical task of horizontal box region classification, the corresponding target capability dimension is fine-grained visual perception capability.

[0154] Furthermore, we need to obtain the first number of historical task types corresponding to the target capability dimension. For example, in the example in Table 4, the target capability dimension corresponding to image classification is global visual perception capability, and the target capability dimension corresponding to target counting is fine-grained visual perception capability. Therefore, the historical task types include two categories, i.e., the first number is 2. In addition, we also need to obtain the second number of all downstream task types corresponding to each capability dimension in the mapping relationship. For example, in Table 2, global visual perception capability includes a total of 5 different downstream tasks, i.e., there are 5 types of downstream tasks with capability ID 1 in Table 2.

[0155] Furthermore, the ratio of the first quantity to the second quantity is calculated to determine the capability dimension score of the model to be evaluated on each capability dimension based on the task-level normalized score and the ratio. In an optional embodiment, the capability dimension score can be determined according to formula (5):

[0156] (5)

[0157] in, For the first The model to be evaluated is in the first... Ability scores on each ability dimension. The first quantity, i.e., the number The model to be evaluated is in the first... The number of tasks involved in the evaluation across each capability dimension. The second quantity is the total number of types of all downstream tasks in the mapping relationship.

[0158] For example, in the example shown in Table 8, the target capability dimension corresponding to image classification is global visual perception capability, and this example only has this one capability dimension. Therefore, the capability dimension score of the Falcon model in global visual perception capability is 80×(0.5+0.5×1 / 5)=48. Table 10 is a schematic table of the capability dimension scores of the model to be evaluated in Table 8. As can be seen from Table 10, the scores of the model to be evaluated in different capability dimensions can be obtained by calculating the normalized scores of different task levels according to formula (5).

[0159] Table 10. Schematic diagram of the scores for one ability dimension

[0160]

[0161] It should be noted that, in specific embodiments, the capability dimension score is used to describe the performance of the model to be evaluated in the corresponding capability dimension. Specifically, the higher the capability dimension score, the better the performance of the model to be evaluated in the corresponding capability dimension, that is, the stronger the performance of the model to be evaluated in the corresponding capability dimension.

[0162] This allows us to demonstrate the model's performance across different capability dimensions (such as global visual perception, spatial and temporal reasoning, and visual language generation), and also provides more granular performance analysis, thereby improving the accuracy and reliability of model evaluation.

[0163] Based on the above embodiments, as an optional embodiment, the comprehensive score of the model to be evaluated is determined according to the task-level normalized score, including:

[0164] The third number of target capability dimensions included, the fourth number of capability dimensions included, the fifth number of historical task types included, and the sixth number of downstream task types included.

[0165] Based on the third and fourth quantities, determine the dimensional coverage of the model to be evaluated;

[0166] Based on the fifth and sixth quantities, determine the task coverage of the model to be evaluated;

[0167] The overall score is determined based on the capability dimension score, dimension coverage, and task coverage; both dimension coverage and task coverage are positively correlated with the overall score.

[0168] like Figure 3 As shown, since the capability dimension score is determined based on the task-level normalized score, in one optional embodiment, the comprehensive score of the model to be evaluated can be calculated based on the capability dimension score.

[0169] Specifically, in order to improve the fairness of the overall score across different dimensions, in one optional embodiment, such as Figure 3 As shown, dimensional coverage and task coverage are introduced. Dimension coverage refers to the ratio of the number of target capability dimensions in which the model to be evaluated participates in the evaluation to the total number of capability dimension types included in the mapping relationship. Task coverage refers to the ratio of the number of historical task types in which the model to be evaluated participates in the evaluation to the total number of downstream task types in the mapping relationship.

[0170] Therefore, when determining the overall score, first obtain the third number of dimension types included in the target capability dimension and the fourth number of dimension types included in the capability dimension. Further, based on the third and fourth numbers, determine the dimension coverage rate. Specifically, in an optional embodiment, the dimension coverage rate can be determined according to formula (6):

[0171] (6)

[0172] in, For dimensional coverage, For the first The third number of the models to be evaluated, namely, the... The third number of the target capability dimensions included in each model to be evaluated. It is the fourth quantity.

[0173] Furthermore, the fifth number of task types included in historical tasks and the sixth number of task types included in downstream tasks are obtained. Further, in an optional embodiment, the ratio of the fifth number to the sixth number is used as the task coverage rate, specifically calculated using the following formula: , For task coverage, For the first The fifth number of models to be evaluated. For the first The sixth number of models to be evaluated, where, as shown in the examples in Table 2, the sixth number... It is 15.

[0174] Therefore, the overall score can be determined based on the capability dimension score, dimension coverage, and task coverage. Specifically, in an optional embodiment, the overall score can be determined according to formula (7):

[0175] (7)

[0176] in, For the overall score, that is, the first... The overall performance score of the model to be evaluated across all capability dimensions.

[0177] According to formula (7), in the specific embodiment, the comprehensive score is... and dimensional coverage Task coverage Both are positively correlated, that is, when the dimensional coverage... The higher the value, the higher the overall score of the model being evaluated. Similarly, when the task coverage... The higher the score, the higher the overall score of the model being evaluated. Therefore, by using a weighted mechanism of task coverage and dimensional coverage, a fair comparison of models across multiple tasks and dimensions can be achieved, improving the credibility and practicality of the evaluation results.

[0178] Table 11 is a schematic table showing the comprehensive score of one of the models to be evaluated in Table 10. The following explanation will be based on Table 11.

[0179] Table 11. Schematic diagram of a comprehensive score result.

[0180]

[0181] In the example shown in Table 10 above, for the Falcon model, the dataset includes 3 tasks and the historical tasks include 2 tasks. Correspondingly, the comprehensive score calculated according to formula (7) is 4.733.

[0182] Therefore, the remote sensing model performance evaluation method provided in this application suppresses the "artificially high" phenomenon where some models achieve extremely high scores on a small number of tasks by using coverage weighting, ensuring the fairness and robustness of scoring in multi-task and multi-dimensional scenarios, and supporting traceable comparison of historical model versions. Furthermore, a hierarchical normalized scoring mechanism is constructed, including a dataset layer, task layer, capability layer, and overall leaderboard layer, unifying the scoring scale across multiple tasks and indicators to achieve fair comparison across tasks.

[0183] In an optional embodiment, the method for evaluating the performance of remote sensing models provided in this application further includes:

[0184] The task-level normalized scores under different historical tasks and / or the data-level normalized scores under different datasets are sorted to obtain the task ranking list;

[0185] Based on the mapping relationship and the ability dimension scores, the ability dimension scores under each ability dimension are sorted to obtain the ability ranking list;

[0186] The overall scores are sorted to obtain the final ranking.

[0187] The task rankings, ability rankings, and overall rankings are transmitted to the target terminal for display.

[0188] In a specific embodiment, after evaluating the new model, the evaluated data can be stored, and the historically stored data can be updated.

[0189] Specifically, the data-level normalized scores and / or task-level normalized scores can be sorted separately to obtain a task ranking list. Simultaneously, the scores of the capability dimensions can be sorted to obtain a capability ranking list, and the comprehensive scores can be sorted to obtain a final ranking list. Furthermore, the task ranking list, capability ranking list, and final ranking list are displayed on the target terminal.

[0190] In a specific embodiment, the overall leaderboard can display the final comprehensive score and scores for each dimension, the task leaderboard can display the model scores under different historical tasks or datasets, and the capability leaderboard can display the performance of each model in multiple capability dimensions such as visual perception, spatial reasoning, and visual language generation based on the mapping relationship between historical tasks and capability dimensions.

[0191] Multi-dimensional rankings and visualizations support various ranking formats, including overall rankings, capability rankings, and task rankings, providing a clear basis for model optimization and selection. Therefore, this application integrates "automatic parsing," "unified calculation," "visual rankings," and "trend tracking," significantly improving evaluation efficiency, scientific rigor, and practical value.

[0192] Furthermore, in specific embodiments, the parsed model metadata and hierarchical normalized scoring results are synchronously stored in both structured and unstructured databases, recording the evaluation timestamp, model source, and version number. Once a new model completes its scoring, the evaluation leaderboard is automatically updated and reordered without manual intervention. The multi-level normalized scoring method effectively suppresses scoring biases caused by varying task coverage and improves the comparability and fairness of evaluation between different models.

[0193] To help those skilled in the art better understand the technical solutions provided in this application, examples will be provided below for further illustration.

[0194] In one optional embodiment, the user uploads the evaluation file of the model to be evaluated, named model_file.zip. For example, in the Falcon model in Table 3 above, the model file can be named Falcon.zip. In another optional embodiment, the naming rule for the evaluation file is: [model_file_name][historical_task_name][dataset_name]_Annotations.json, for example, [Falcon][PIX_CHG][HRSCD]_Annotations.json. The specific JSON file has the following format:

[0195] {

[0196] "info": {

[0197] "task": "PIX_CHG",

[0198] "model": "Falcon",

[0199] "dataset": "HRSCD"

[0200] },

[0201] "results": {

[0202] "mDICE": 0.432902,

[0203] "mIoU": 0.340895

[0204] }

[0205] }

[0206] The scoring file consists of two parts: info (basic model information), model (model name), task (name of historical task), and dataset (name of dataset); and results (historical metric scores).

[0207] Based on the mapping relationship, historical tasks can be mapped to corresponding evaluation metrics. Referring to Table 2, the evaluation metric corresponding to PIX_CHG (change detection) is mIoU. The score for this task is identified as {taskType}#{datasetName}, i.e., PIX_CHG#HRSCD. Rounded to three decimal places, this score is 0.341. If this metric does not have a score, no score is given.

[0208] In a specific implementation, some dataset names contain a decimal point ".". Since MongoDB recognizes decimal points as secondary indicators, the decimal point "." in the task identifier is replaced with "@". After parsing all JSON files in the compressed package, the model file information is updated, specifically including the model file name (name + uploaded timestamp), filename (compressed package name), path (path to Minio upload location), and updateTime (upload time). For example, the parsing result of an evaluation file is as follows:

[0209] {

[0210] "name" : "Falcon@1743664083716169",

[0211] "fileName" : "Falcon.zip",

[0212] "modelName" : "Falcon",

[0213] "path":

[0214] " / TJEOS / WORKDIR / FILE / bench / 1743664083716169 / Falcon.zip",

[0215] "updateTime" : ISODate("2025-04-03T15:08:04.094+0000"),

[0216] "IMG_CAP#RSICD": 107.07,

[0217] "IMG_CAP_DETAILED#RSICD" : 39.819,

[0218] "IMG_CAP_DETAILED#RSICap" : 26.009,

[0219] "IMG_CAP_DETAILED#RSITMD" : 41.905,

[0220] "IMG_CLS#AID": 0.359,

[0221] "IMG_CT#DIOR": 0.77,

[0222] "PIX_CHG#HRSCD": 0.341,

[0223] "PIX_CLS#BH-Pools_Watertanks_Datasets" : 0.999,

[0224] "PIX_SEG#iSAID": 0.517,

[0225] "REG_CAP#RSVG" : 99.485,

[0226] "REG_CLS_HBB#NWPU-VHR-10" : 0.989,

[0227] "REG_DET_HBB#DOTA2@0" : 34.065,

[0228] "REG_DET_OBB#FAIR1M1@0" : 53.158,

[0229] "REG_VG#DIOR-RSVG" : 77.846,

[0230] }

[0231] After uploading the compressed file, the model's score in MongoDB needs to be dynamically updated (including at least one of the following: data-level normalized score, task-level normalized score, capability dimension score, and comprehensive score). It should check if the original metric (task identifier, such as PIX_CHG#HRSCD) exists; if it does, update it with the new value; otherwise, insert a new data entry. This ensures the model score reflects the latest calculation and allows for the addition of new dataset scores to existing results. Each time the model evaluation file is uploaded, the leaderboard is automatically updated due to the changed model score.

[0232] First, calculate the data-level normalized score and the task-level normalized score. For example, for historical image classification tasks, according to Table 2, find all model records containing IMG_CLS#, such as IMG_CLS#AID, IMG_CLS#CLRS, etc. Calculate the weighted score according to formula (1) and the data-level normalized score according to formula (2). Further, based on the data-level normalized score, calculate the harmonic mean through formula (3), and calculate the task-level normalized score based on the harmonic mean through formula (4).

[0233] After calculating the task-level score, the corresponding capability dimension score is calculated. IMG_CLS corresponds to the globalVisualPerception capability dimension, which also includes four other tasks: REG_DET_HBB, REG_DET_OBB, PIX_SEG and VQA_presence. The five capability dimensions are weighted and summarized, and the capability dimension score is calculated according to formula (5).

[0234] In a specific implementation, the results are synchronously updated to the MySQL database. Based on the scores of the four capability dimensions, dimensional coverage, and task coverage, the comprehensive score of the model to be evaluated is calculated and updated in the MySQL database. Finally, based on the number of "#" fields in the model table, the total number of valid datasets for different models is calculated, and the count field information in MySQL is updated.

[0235] Finally, the overall ranking is queried. Specifically, it returns information such as ability dimension scores, overall scores, and the number of datasets, obtained directly from a MySQL table. The ability ranking and task ranking are retrieved from a MongoDB database. This maximizes the system's flexibility; the datasets corresponding to tasks can be added indefinitely, and the tasks corresponding to abilities can be dynamically adjusted. When adding task categories, only the corresponding dimensions and scoring metrics need to be dynamically added. Since the database already contains pre-calculated data records, querying the ability ranking through the MongoDB database returns information such as ability scores, task scores, and rankings.

[0236] The query task leaderboard accepts user-input query parameters, which may include, but are not limited to, name (model name), sourceType (source type: open source / closed source), modelType (model type: image / image text), taskType (historical tasks, such as "IMG_CLS"), orderByField (sorting field, defaulting to overallScore - average task score), orderByType (sorting type: asc - ascending / desc - descending), pageNo (page number), and pageSize (number of pages per page). In a specific query implementation, the first 100 data entries are scanned based on taskType, and the regular expression "^{taskType}($|#)" is used to match feature fields containing "IMG_CLS", such as IMG_CLS#AID, thus ensuring that the query results include scores from all sub-datasets for that task.

[0237] Assemble a MongoDB query statement, for example, `lists=collection.find(query, projection).skip(skip).limit(limit).sort(sortField, order)`. `query` is the filtering statement, `projection` is a combination of basic and feature fields, `skip = (pageNo-1)*pageSize` finds records that meet the conditions, sets `overallScore` to the score corresponding to the task category (e.g., `IMG_CLS`), and sets the rank to `skip+1`.

[0238] Therefore, the remote sensing model performance evaluation method provided in this application does not require modifying the database schema when adding new historical tasks or datasets. It only requires adding the mapping relationship between downstream tasks and capability dimensions to expand the leaderboard. No manual modification is needed, avoiding inefficiency and error-prone issues. Furthermore, this application enables one-click import by parsing naming rules and mapping relationships, improving model evaluation efficiency.

[0239] Furthermore, by using MongoDB dynamic storage and MySQL aggregated tables, uploading model scoring files automatically triggers updates, avoiding repetitive manual calculations and reducing leaderboard refresh times. Dual database storage eliminates the need for layer-by-layer calculations, improving query efficiency. The fully automated calculation process avoids manual intervention, enabling end-to-end traceability from the overall leaderboard to the dataset scores.

[0240] In the above embodiments, the method for evaluating the performance of remote sensing models has been described in detail. This application also provides an embodiment of a device for evaluating the performance of remote sensing models.

[0241] Figure 4 This is a schematic diagram of the structure of a remote sensing model performance evaluation device provided in an embodiment of this application, as shown below. Figure 4 As shown, the device includes:

[0242] The meta-information acquisition module 40 is used to acquire the model meta-information of the model to be evaluated. The model meta-information includes historical tasks, datasets, and historical metric scores. The historical metric scores are used to characterize the accuracy of the model to be evaluated in performing the corresponding historical tasks based on the dataset.

[0243] The first normalization module 41 is used to normalize the historical index scores of the dataset on the corresponding historical task based on the model meta information to obtain the data-level normalized score.

[0244] The second normalization module 42 is used to normalize the data-level normalization scores corresponding to the same historical tasks again to obtain the task-level normalization scores.

[0245] The comprehensive score determination module 43 is used to determine the comprehensive score of the model to be evaluated based on the task-level normalized score; the comprehensive score is used to characterize the comprehensive performance of the model to be evaluated; the higher the comprehensive score, the better the comprehensive performance.

[0246] Furthermore, the remote sensing model performance evaluation device provided in this application embodiment also includes:

[0247] The mapping relationship acquisition module is used to acquire pre-built mapping relationships; the mapping relationships include the correspondence between downstream tasks, evaluation metrics, and metric weights.

[0248] The weight determination module is used to determine the target indicator weights corresponding to historical indicator scores based on model metadata and mapping relationships.

[0249] The weighted score determination module is used to determine the weighted score of the dataset on the corresponding historical task based on the target indicator weight and the historical indicator score.

[0250] The first normalization submodule is used to normalize the weighted scores corresponding to the same dataset and the same historical tasks to obtain data-level normalized scores.

[0251] The target average value determination module is used to determine the target average value of the data-level normalized score of the model to be evaluated under the same historical tasks.

[0252] The second normalization submodule is used to normalize the target average value corresponding to the same historical task to obtain the task-level normalized score.

[0253] The target capability dimension determination module is used to determine the target capability dimension corresponding to the historical task based on the mapping relationship; the mapping relationship also includes the correspondence between downstream tasks and capability dimensions; the capability dimension is used to describe the model's performance on the corresponding downstream task;

[0254] The quantity determination module is used to obtain the first quantity of historical task types corresponding to the target capability dimension, and the second quantity of all downstream task types corresponding to each capability dimension in the mapping relationship;

[0255] The ratio determination module is used to determine the ratio between the first quantity and the second quantity.

[0256] The capability dimension score determination module is used to determine the capability dimension score of the model to be evaluated in each capability dimension based on the task-level normalized score and ratio. The capability dimension score describes the performance of the model to be evaluated in the corresponding capability dimension. The higher the capability dimension score, the better the performance of the model to be evaluated in the corresponding capability dimension.

[0257] The quantity determination module is also used to obtain the third quantity of the number of dimension types included in the target capability dimension, the fourth quantity of the number of dimension types included in the capability dimension, the fifth quantity of the number of task types included in the historical task, and the sixth quantity of the number of task types included in the downstream task.

[0258] The dimension coverage determination module is used to determine the dimension coverage of the model to be evaluated based on the third and fourth quantities.

[0259] The task coverage determination module is used to determine the task coverage of the model to be evaluated based on the fifth and sixth quantities.

[0260] The overall score determination submodule is used to determine the overall score based on the capability dimension score, dimension coverage, and task coverage; both dimension coverage and task coverage are positively correlated with the overall score.

[0261] The sorting module is used to sort the task-level normalized scores under different historical tasks and / or the data-level normalized scores under different datasets to obtain the task ranking; based on the mapping relationship and the capability dimension scores, it sorts the capability dimension scores under each capability dimension to obtain the capability ranking; and sorts the comprehensive scores to obtain the overall ranking.

[0262] The transmission module is used to transmit the task rankings, ability rankings, and overall rankings to the target terminal for display.

[0263] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 5 As shown, the electronic device includes: a memory 50 for storing computer programs;

[0264] The processor 51 is configured to execute a computer program to implement the steps of the remote sensing model performance evaluation method as described in the above embodiments.

[0265] The electronic devices provided in this embodiment may include, but are not limited to, laptops or desktop computers.

[0266] The processor 51 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 51 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 51 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 51 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 51 may also include an Artificial Intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0267] The memory 50 may include one or more computer-readable storage media, which may be non-transitory. The memory 50 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 50 is used to store at least the following computer program 501, which, after being loaded and executed by the processor 51, is capable of implementing the relevant steps of the remote sensing model performance evaluation method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 50 may also include an operating system 502 and data 503, and the storage method may be temporary or permanent storage. The operating system 502 may include Windows, Unix, Linux, etc. The data 503 may include, but is not limited to, relevant data involved in the remote sensing model performance evaluation method.

[0268] In some embodiments, the electronic device may further include a display screen 52, an input / output interface 53, a communication interface 54, a power supply 55, and a communication bus 56.

[0269] Those skilled in the art will understand that Figure 5 The structures shown do not constitute a limitation on electronic devices and may include more or fewer components than those shown.

[0270] The electronic device provided in this application includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the remote sensing model performance evaluation method in the above embodiments.

[0271] It should be noted that although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

Claims

1. A method for evaluating the performance of a remote sensing model, characterized in that, The method comprises: obtaining model meta-information of a to-be-evaluated model; the model meta-information comprises a historical task, a data set and a historical indicator score; the historical indicator score is used to represent the accuracy of the to-be-evaluated model in performing a corresponding historical task based on the data set; the to-be-evaluated model is a remote sensing large model in the remote sensing vertical field; and the data set is a remote sensing image data set; performing normalization processing on the historical indicator score of the data set on the corresponding historical task according to the model meta-information to obtain a data-level normalized score; performing normalization processing again on the data-level normalized scores corresponding to the same historical task to obtain a task-level normalized score; determining a comprehensive score of the to-be-evaluated model according to the task-level normalized score; the comprehensive score is used to represent the comprehensive performance of the to-be-evaluated model; the higher the comprehensive score is, the better the comprehensive performance is; the normalization processing on the historical indicator score of the data set on the corresponding historical task according to the model meta-information to obtain a data-level normalized score comprises: obtaining a pre-constructed mapping relationship; the mapping relationship comprises a corresponding relationship among a downstream task, an evaluation indicator and an indicator weight; determining a target indicator weight corresponding to the historical indicator score according to the model meta-information and the mapping relationship; determining a weighted score of the data set on the corresponding historical task according to the target indicator weight and the historical indicator score; performing normalization processing on the weighted scores corresponding to the same data set and the same historical task to obtain the data-level normalized score.

2. The method of claim 1, wherein, When the evaluation indicator corresponding to the historical indicator score is average accuracy or average accuracy mean, the target indicator weight is determined according to a preset corresponding relationship, and the sum of the target indicator weights is 1; the preset corresponding relationship is a corresponding relationship between an intersection over union value range and an indicator weight.

3. The method of claim 1, wherein, the normalization processing again on the data-level normalized scores corresponding to the same historical task to obtain a task-level normalized score comprises: determining a target average value of the data-level normalized scores corresponding to the to-be-evaluated model under the same historical task; performing normalization processing on the target average values corresponding to the same historical task to obtain the task-level normalized score.

4. The method of claim 1, wherein, The mapping relationship further comprises a corresponding relationship between the downstream task and a capability dimension; the capability dimension is used to describe the performance of a model on a corresponding downstream task; before the determination of the comprehensive score of the to-be-evaluated model according to the task-level normalized score, the method further comprises: determining a target capability dimension corresponding to the historical task according to the mapping relationship; obtaining a first number of historical task types corresponding to the target capability dimension, and a second number of all downstream task types corresponding to each capability dimension in the mapping relationship; determining the ratio of the first number to the second number; According to the task-level normalized score and the ratio, a capability dimension score of the to-be-evaluated model on each capability dimension is determined; the capability dimension score is used to describe the performance of the to-be-evaluated model on the corresponding capability dimension, and the higher the capability dimension score is, the better the performance of the to-be-evaluated model on the corresponding capability dimension is.

5. The method of claim 4, wherein, The determining the comprehensive score of the to-be-evaluated model according to the task-level normalized score comprises: obtaining a third number of dimension categories included in the target capability dimension, a fourth number of dimension categories included in the capability dimension, a fifth number of task types included in the historical task, and a sixth number of task types included in the downstream task; determining a dimension coverage of the to-be-evaluated model according to the third number and the fourth number; determining a task coverage of the to-be-evaluated model according to the fifth number and the sixth number; determining the comprehensive score according to the capability dimension score, the dimension coverage, and the task coverage; the dimension coverage and the task coverage are positively correlated with the comprehensive score.

6. The method of claim 5, wherein, The method further comprises: sorting the task-level normalized scores under different historical tasks and / or the data-level normalized scores under different data sets to obtain a task list; sorting the capability dimension scores under each capability dimension according to the mapping relationship and the capability dimension scores to obtain a capability list; sorting the comprehensive scores to obtain a total list; transmitting the task list, the capability list, and the total list to a target terminal for display.

7. A device for evaluating the performance of a remote sensing model, characterized in that, The device comprises: a meta-information acquisition module configured to acquire model meta-information of a to-be-evaluated model; the model meta-information comprises historical tasks, data sets, and historical indicator scores; the historical indicator scores are used to represent the accuracy of the to-be-evaluated model in performing corresponding historical tasks based on the data sets; the to-be-evaluated model is a remote sensing large model in the remote sensing vertical field; and the data sets are remote sensing image data sets; a first normalization module configured to perform normalization processing on the historical indicator scores of the data sets on corresponding historical tasks according to the model meta-information to obtain data-level normalized scores; a second normalization module configured to perform normalization processing again on the data-level normalized scores corresponding to the same historical tasks to obtain task-level normalized scores; a comprehensive score determination module configured to determine a comprehensive score of the to-be-evaluated model according to the task-level normalized scores; the comprehensive score is used to represent the comprehensive performance of the to-be-evaluated model; the higher the comprehensive score is, the better the comprehensive performance is; a mapping relationship acquisition module configured to acquire a pre-constructed mapping relationship; the mapping relationship comprises a corresponding relationship among downstream tasks, evaluation indicators, and indicator weights; a weight determination module configured to determine a target indicator weight corresponding to the historical indicator scores according to the model meta-information and the mapping relationship; a weighted score determination module configured to determine a weighted score of the data sets on corresponding historical tasks according to the target indicator weight and the historical indicator scores. A first normalization submodule is configured to normalize the weighted scores corresponding to the same data set and the same historical task to obtain data-level normalized scores.

8. An electronic device comprising a memory and a processor, said memory having stored thereon a computer program operable to run on said processor, characterized in that, The processor executes the computer program to implement the steps of the remote sensing model performance evaluation method of any one of claims 1 to 6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the remote sensing model performance evaluation method of any one of claims 1 to 6.

Images