A model paradigm feature matching-based surrogate model adaptive data quality evaluation method

Abstract

The application discloses a kind of based on model paradigm feature matching's proxy model adaptive data quality evaluation method.The present application establishes the standardized technical rules of paradigm compatibility determination between proxy model and target model, fills the technical gap that proxy model selection lacks objective basis in third-party authentication scene;At the same time, proxy model selection is decided by team internally by default, and is converted into the paradigm compatibility determination rule system formed by four dimensions of output structure type, training target prototype, topological structure type and loss function type, so that any proxy model selection process for establishing the credibility of proxy model evaluation results when using proxy model to replace target model for data quality evaluation will not cause the technical rationality of evaluation conclusion to be unable to be verified externally due to the lack of paradigm compatibility determination rule system, so that the compliance and public credit of evaluation conclusion have technical basis support.

Description

Technical Field

[0001] This invention belongs to the field of data quality assessment technology, specifically relating to an adaptive data quality assessment method for surrogate models based on model paradigm feature matching. Background Technology

[0002] The performance of large-scale artificial intelligence models is highly dependent on the quality of their pre-training datasets. In order to evaluate the actual utility of datasets for model training under limited resource conditions, academia and industry have proposed a variety of data evaluation methods based on lightweight surrogate models. Representative solutions include dataset quality ranking based on lightweight model training gains and fast value evaluation based on surrogate tasks. The core of these methods revolves around the correspondence between the performance of the surrogate model and the performance of the target model.

[0003] The basic idea behind these methods is to use a small model with a much smaller number of parameters than the target model as a proxy, train the dataset to be evaluated on the proxy model, and infer the training value of the dataset for the target model by the performance of the proxy model. Since the training team is clearly aware of the specific type and training paradigm of its target model in an internal setting, and the selection and use of the proxy model are completely controlled by the team internally, there is no need to prove the rationality of the proxy model selection technology to external third parties. Therefore, traditional methods pre-set the proxy model to a fixed type in technical design, and the technical correspondence between the proxy model and the target model can be established by manual judgment. They do not provide a technical mechanism for differentiated matching based on the technical characteristics of the target model, which leads to the evaluation relying on the personal experience of experts.

[0004] In its earlier application CN2026107920125, the applicant disclosed a data quality assessment method based on a surrogate model mapping relationship. This method uses a technical solution to convert the assessment results by establishing a mapping relationship between the performance gain of the surrogate model and the performance gain of the target model. However, this solution focuses on the "mapping conversion" stage and does not involve the surrogate model selection stage before the establishment of the mapping relationship. In actual certification business, the target models of different customers have significant differences in training paradigms. If the surrogate model is not properly selected, even if there is an accurate mapping relationship, the assessment conclusion of the surrogate model still lacks technical basis for the representativeness of the target model.

[0005] Therefore, in the proxy model selection stage before the establishment of this mapping relationship, a systematic method is needed that can adaptively determine the appropriate proxy model type based on the technical characteristics of the target model, so as to fill the gap in the existing technology chain in the proxy model selection stage.

[0006] Specifically, in the authentication scenario, clients entrust authentication agencies to independently evaluate the quality of their datasets. Different clients may use target models with significantly different training paradigms. For example, autoregressive language models aim at sequence generation tasks, contrastive learning models aim at optimizing the similarity of representation spaces, and classification models aim at minimizing the error rate of class discrimination. Due to limitations in computing power and time, authentication agencies use surrogate models to replace the client's target models to perform the evaluation. In this case, the authentication agency must answer a question that does not exist in the internal scenario: whether there is a technically effective substitution relationship between the surrogate model and the client's target model at the training paradigm level, so that the evaluation results of the surrogate model can reasonably reflect the training effectiveness of the dataset on the client's target model.

[0007] Therefore, the existing technology lacks a systematic method to automatically determine the appropriate proxy model type based on the technical characteristics of the client's target model in a third-party certification scenario. If the certification body uses the same proxy model for all clients indiscriminately, the proxy model may have an essential difference in training paradigm from the target model of some clients, resulting in the evaluation results lacking a reasonable basis in technical logic, and the credibility and compliance of the certification conclusions will be questioned.

[0008] Therefore, it is necessary to invent an adaptive data quality assessment method for surrogate models based on model paradigm feature matching to solve the above problems. Summary of the Invention

[0009] The purpose of this invention is to provide an adaptive data quality assessment method for surrogate models based on model paradigm feature matching. By establishing standardized technical rules for determining the paradigm compatibility between the surrogate model and the target model, it fills the technical gap of lacking objective basis for surrogate model selection in third-party certification scenarios. Simultaneously, it transforms the surrogate model selection process from a default decision made internally by the team into a paradigm compatibility determination rule system composed of four dimensions: output structure type, training target prototype, topology structure type, and loss function type. This ensures that when using a surrogate model to replace the target model for data quality assessment, the surrogate model selection process for establishing the credibility of the surrogate model assessment results will not suffer from a lack of a paradigm compatibility determination rule system, preventing the technical rationality of the assessment conclusions from being independently verified externally. This provides a technical foundation for the compliance and credibility of the assessment conclusions, thus addressing the aforementioned shortcomings in the technology.

[0010] To achieve the above objectives, the present invention provides the following technical solution: an adaptive data quality assessment method for surrogate models based on model paradigm feature matching, comprising the following steps:

[0011] Step 1: Construct a label set based on multi-dimensional paradigm features Perform paradigm compatibility The determination is based on the definition of an adaptive selection function. and output the proxy model. and evaluation task set A rule version is constructed;

[0012] Step 2: Collect target model Technical description information, and determine the proxy model. With evaluation task set ;

[0013] Step 3: Driving the agent model Perform training and obtain the dataset to be evaluated. Relative performance gain index ;

[0014] Step 4: Based on the relative performance gain index The system generates evaluation conclusions that include the selection criteria based on preset thresholds.

[0015] The aforementioned adaptive data quality assessment method for surrogate models based on model paradigm feature matching, in step 1, constructs a label set based on multi-dimensional paradigm features. Perform paradigm compatibility The determination is based on the definition of an adaptive selection function. and output the proxy model. and evaluation task set To construct a rule version, the following steps are involved:

[0016] 1.1 Select the tag set ;

[0017] 1.2 Perform paradigm compatibility testing The determination;

[0018] 1.3. Define the adaptive selection function and output the proxy model. and evaluation task set ;

[0019] 1.4. Based on tag sets Paradigm compatibility Adaptive selection function Proxy model pool and evaluation task collection library Together, they constitute the rule version.

[0020] The aforementioned adaptive data quality assessment method for surrogate models based on model paradigm feature matching, in step 1.1, selects a label set. The specific steps are as follows:

[0021] 1.1.1, Arbitrary model Mapped to its paradigm eigenvector The specific formula is as follows:

[0022] ;

[0023] in, For any model paradigm feature vector, A mapping function for the paradigm feature extraction function of any model;

[0024] The task paradigm feature function is used to describe the mathematical structure of the model's output samples and the mathematical prototype of the training target.

[0025] These are architectural paradigm feature functions used to describe the topological structure of information transmission within the model;

[0026] is the feature function of the loss paradigm, used to describe the mathematical functional form of the model training loss function;

[0027] , , The value range of the three core paradigm features includes: the output structure label set. Training target prototype label set Architecture topology tag set Loss functional tag set All are tag sets that can be expanded to a limited extent. They initially contain known tags and can be dynamically incorporated into various new types that will emerge in the future through automatic generation or manual configuration.

[0028] Among them, task paradigm features Architectural Paradigm Characteristics and loss paradigm characteristics The extraction process is as follows:

[0029] 1.1.2 Extracting Task Paradigm Features The specific calculation formula is as follows:

[0030] ;

[0031] in, For the model The output sample structure type is determined by: taking the model A single output sample for any valid input According to the rules, extract the complete set of output structure type tags. The specific rules for assigning tags are as follows:

[0032] H1, if Each item ( All are taken from a finite set of symbols. That is, the output is a variable-length discrete symbol sequence. ;

[0033] in, For a single output sample The sequence length, ; This is an assignment operator; for The label in the middle represents the member label of the output of the variable-length discrete sequence;

[0034] H2, if That is, the output is a fixed-dimensional real vector. ;

[0035] in, For fixed-dimensional real vectors, These are exclusive tags for vector-type outputs.

[0036] H3, if That is, the output is discrete category labels. ;

[0037] in, For a discrete finite set of integers, This represents the total number of categories. Output labels for single discrete classification;

[0038] H4, Otherwise, according to The mathematical structure generates a unique formal identifier corresponding to a new structure label. ,make , and place ;

[0039] Among them, mathematical structure includes metric space, dimensional properties, and data type;

[0040] in, For the model The mathematical prototype of the training objective is determined by: based on the model Loss function optimized during training The functional prototype, from the training target prototype label set The corresponding training target prototype labels are assigned in the following way:

[0041] K1, if Equivalent to autoregressive likelihood maximization, i.e., for data taken from the distribution Given a sample sequence, the optimization objective is to minimize the negative log-conditional probability. exist The expectation above, then ;

[0042] in, for Position index in the sequence; Indicates position All previous elements; For the model Parameters;

[0043] For training target prototype label set Pre-defined standard labels for autoregressive likelihood maximization classes;

[0044] K2, if This is equivalent to maximizing the contrast similarity, which depends on the positive and negative sample pairs in... ,and The optimization objective is as follows: ;

[0045] in, It is a distance metric function in vector space, and its output is a nonnegative scalar; This is a preset interval hyperparameter used to control the minimum distance boundary between positive and negative sample pairs, and ;

[0046] For training target prototype label set Pre-defined standard labels that maximize the comparison similarity;

[0047] K3, if This is equivalent to minimizing the classification error rate, i.e., for values ​​taken from the joint distribution. Sample label pairs The optimization objective is to minimize the negative log-likelihood of the categories in the classification task. In distribution The expectation above, then ;

[0048] in For the input sample feature vector, For the corresponding real category label;

[0049] For training target prototype label set The pre-defined standard label minimizes the classification error rate;

[0050] K4, if This is equivalent to cross-modal alignment loss, which simultaneously processes the representation spaces of two or more different modalities and minimizes the cross-modal distance metric. ;

[0051] For training target prototype label set Pre-defined cross-modal alignment class standard tags;

[0052] K5. Otherwise, according to The basic operational structure, distance kernel type, and constraint term structure generate unique formal identifiers corresponding to new training target prototype labels. ,make , and place ;

[0053] 1.1.3 Extracting architectural paradigm features The specific calculation formula is as follows:

[0054] ;

[0055] in, It is a set of nodes, where each node corresponds to a processing unit. It is a set of directed edges, where each directed edge represents the information flow from the predecessor node to the successor node;

[0056] and For the model The topology type, whose value is the architecture topology tag set. The elements in the model are determined based on the following criteria. Calculation graph Specifically:

[0057] Y1, if If the graph is acyclic feedforward, meaning there is no directed path from any node back to itself or forming a cycle, then... ;

[0058] For architecture topology tag set Pre-defined standard labels for acyclic feedforward topologies;

[0059] Y2. If If there exists a bidirectional directed path between any two distinct nodes, i.e., fully bidirectionally connected, then... ;

[0060] For architecture topology tag set Pre-defined standard tags for fully bidirectional connected topology classes;

[0061] Y3, if Can be divided into encoder subgraphs and decoder subgraph And there exists a fixed-dimensional intermediate representation. And satisfy , ,but ;

[0062] in Encoder subgraph Input samples, Decoder subgraph The final output result; For architecture topology tag set The pre-built encoder-decoder topology class standard label;

[0063] Y4, if If it contains at least two heterogeneous input subnetworks and there are information fusion nodes that span across the subnetworks, then ;

[0064] For architecture topology tag set The pre-set standard label corresponds to the topology of the multimodal fusion computing graph containing heterogeneous input subnetworks and cross-subnetwork information fusion nodes, and matches the judgment rule of step Y4;

[0065] Y5, otherwise according to The adjacency matrix type, loop characteristics, and connection constraints generate unique formal identifiers corresponding to new labels. ,join in juxtaposition ;

[0066] in, For architecture topology tag set Dynamically added custom topology labels are only used to match special model computation graph structures that do not have preset standard labels;

[0067] 1.1.4 Extracting Loss Paradigm Features The specific calculation formula is as follows:

[0068] ;

[0069] in, For the model The training loss functional;

[0070] Its determination method is based on the training loss function. The mathematical functional form determines this, specifically:

[0071] S1, if Essentially, it involves calculating the distance or divergence between two probability distributions. ;

[0072] It is a loss functional tag set The pre-defined standard labels in the data correspond to loss function types that are based on the calculation of distance / divergence between two probability distributions, and match the judgment rules of step S1.

[0073] S2, if Based on the comparison construction of representation vectors, and having the form Or equivalent to it, then ;

[0074] in, It is a loss functional tag set Pre-defined standard tags for comparison construction; This is the representation vector of the anchor point sample; This is the representation vector of the positive samples; The representation vector of the negative samples; Here, is a distance metric function on a vector space, and its output is a nonnegative scalar; The preset interval hyperparameter is still used to control the minimum distance boundary between positive and negative sample pairs, and ;

[0075] S3, if The deviation between the predicted value and the target value is calculated element by element, using the general formula: ,but ;

[0076] in, For the first The predicted value of each element; For the first The target value of each element; This is a deviation measurement function; It is a loss functional tag set The pre-defined standard label for element-wise deviation loss is the loss function that calculates the deviation between the predicted value and the true target scalar value for each position.

[0077] S4. If If it includes the global normalization term for the entire sequence, then ;

[0078] For loss functional tag set The pre-defined standard label for global normalization loss class refers to the loss function that includes the normalization term of the whole sequence partition function;

[0079] Includes the global normalization term across the entire sequence, including the partition function in conditional random fields. ;

[0080] in Input sequence for the loss function calculation stage The normalized sum of all possible output sequence states of the model;

[0081] S5. Otherwise, according to Basic operations, distance term types, and regularization term types generate unique formal identifiers for new loss functional labels. ,join in juxtaposition ;

[0082] 1.1.5. Based on the output structure tag set Training target prototype label set Architecture topology tag set Loss functional tag set Get the tag set .

[0083] The aforementioned adaptive data quality assessment method for surrogate models based on model paradigm feature matching, in step 1.2, performs a paradigm compatibility determination, the specific steps of which are as follows:

[0084] 1.2.1 Define normal form compatibility as The formula for determining this is: ;

[0085] in, For the paradigm feature space, An output of 1 indicates that the two eigenvectors are "compatible", and an output of 0 indicates that the two eigenvectors are "incompatible".

[0086] 1.2.2. For the feature vector of the target model paradigm and surrogate model paradigm feature vector If all of the following core conditions are met, then the product is considered compatible. ;

[0087] Tag equality comparison rule: If any tag originates from a catch-all newly added tag, i.e. , , , Elements within a tag are judged based on their unique formal identifier. Two tags are considered equal if their formal identifiers are identical or they have a one-to-one mapping relationship; otherwise, they are considered unequal. The specific conditions are as follows:

[0088] (1) The output structure types are equal, that is ;

[0089] (2) The training target prototypes are equal, that is ;

[0090] (3) The loss function types are the same, that is ;

[0091] like Then additional requirements ;

[0092] like Therefore, equal architectures are not mandatory.

[0093] in, For the target model, For proxy models in the proxy model pool.

[0094] The aforementioned adaptive data quality assessment method for surrogate models based on model paradigm feature matching defines an adaptive selection function in step 1.3. and output the proxy model. and evaluation task set The specific steps are as follows:

[0095] 1.3.1 Establishing a proxy model pool With evaluation task set library Both are version-based, specifically:

[0096] When there are new additions, adjustments, or other changes to the proxy model pool or evaluation task set library, the newly added proxy model must have its task paradigm features, architecture paradigm features, and loss paradigm features extracted and recorded in advance. The newly added evaluation task set must have its applicable task paradigm tuples identified and recorded before it can be included in subsequent versions of the corresponding proxy model pool or evaluation task set library.

[0097] Among them, for the first One proxy model entry The record identifier is The eigenvectors of the paradigm are Priority weight is ;

[0098] in, ,and ;

[0099] The priority weight has a preset correspondence with the priority, and the smaller the value, the higher the priority.

[0100] For the Each evaluation task set item The record identifier is The applicable task paradigm binary is ( , ), priority weight is ;

[0101] in, ,and ;

[0102] Labels indicating the output structure type applicable to this evaluation task set. Prototype labels for training objectives applicable to this evaluation task set;

[0103] The priority weight has a preset correspondence with the priority, and the smaller the value, the higher the priority.

[0104] 1.3.2 Input target model feature vector Output proxy model With assessment task set The specific steps are as follows:

[0105] L1. Define the adaptive selection function Specifically:

[0106] ;

[0107] L2. Select the proxy model using the following formula:

[0108] ;

[0109] in, The surrogate model set consists of all surrogate model entries in the surrogate model pool that satisfy paradigm compatibility. , For the first The paradigm feature vector of each proxy model entry; if This triggers a request to expand the proxy model pool;

[0110] L3. Select the candidate evaluation task set, using the following formula:

[0111] ;

[0112] in, This is a candidate evaluation task set, whose elements are all evaluation task entries in the evaluation task set library that match the target model with the applicable task paradigm features. ; The applicable output structure type recorded in the evaluation task set library for each evaluation task set entry. The applicable output training target prototypes recorded in the evaluation task set library for the evaluation task set entries;

[0113] like This triggers a request to expand the evaluation task set library and terminates the process.

[0114] L4. Outputting the proxy model based on priority selection. and evaluation task set The specific formula is as follows:

[0115] ;

[0116] ;

[0117] in For the selected proxy model, For the selected set of evaluation tasks; For the proxy model Priority weights, For the evaluation task set The priority weight is determined by the fact that if multiple items have the same priority, one of them is selected according to the preset rules.

[0118] The aforementioned adaptive data quality assessment method for surrogate models based on model paradigm feature matching, in step 2, collects data from the target model. Technical description information, and determine the proxy model. With evaluation task set The specific steps are as follows:

[0119] 2.1 Obtain the target model to be evaluated Technical description information;

[0120] The technical description information includes the output sample structure of the target model, the prototype of the training target functional, the internal computation graph topology, and the training loss function.

[0121] 2.2. Based on the extraction and analysis of technical description information, the target model is obtained. Normative eigenvectors The specific formula is as follows:

[0122] ;

[0123] For the target model Task paradigm characteristics, by and Composition, and For the target model The output sample structure type; For the target model The mathematical prototype of the training objective; For the target model The architectural paradigm features, derived from the target model Topology type express; For the target model The loss paradigm characteristics are derived from the target model. Training loss functional express;

[0124] 2.3. The feature vector of the target model paradigm As input, the adaptive selection function is called. In the proxy model pool and evaluation task set library Determine the appropriate proxy model and evaluation task set The specific calculation formula is as follows:

[0125] ;

[0126] Among them, the selected proxy model With the target model The normal form compatibility criteria must be met between them. ,Right now: ; ;

[0127] If and only if Additional requirements .

[0128] The aforementioned adaptive data quality assessment method for surrogate models based on model paradigm feature matching, in step 3, drives the surrogate model. Perform training and obtain the dataset to be evaluated. Relative performance gain index The specific steps are as follows:

[0129] 3.1. Based on the selected proxy model Load its initial weights and configure the standard training hyperparameter set, specifically: obtain the surrogate model in the preset computing environment. Baseline performance score ;

[0130] Among them, the baseline performance score The acquisition methods include: using general benchmark datasets Training agent model Subsequently, in the evaluation task set The above evaluation results; or the proxy model Without undergoing any task-related fine-tuning, directly evaluate the task set. The above evaluation results are as follows;

[0131] 3.2 Using the dataset to be evaluated Train the surrogate model with a configuration identical to the baseline training. After training, on the same evaluation task set The evaluation results were obtained through testing. ;

[0132] 3.3 Based on baseline performance scores and performance evaluation score The dataset to be evaluated is obtained. Relative performance gain index The specific formula is as follows:

[0133] .

[0134] The aforementioned adaptive data quality assessment method for surrogate models based on model paradigm feature matching, in step 4, is based on the relative performance gain index. The evaluation conclusion, which includes the selection criteria, is generated based on preset thresholds. The specific steps are as follows:

[0135] 4.1 Preset quality level threshold mapping table;

[0136] The quality grade threshold mapping table defines a set of ordered thresholds. and the corresponding quality grade label ;

[0137] in, For the first Level threshold, ,satisfy , The total number of quality grades; the quality grade threshold mapping table is determined based on historical evaluation experience data or industry statistical benchmarks.

[0138] 4.2 The relative performance gain index Match the data with a quality grade threshold mapping table and assign quality grade labels according to a preset interval-grade correspondence. ;

[0139] 4.3 Based on the above determinations, a data quality assessment conclusion is generated. Specifically, the quality level judgment included in this data quality assessment conclusion is based on a proxy model. Relative performance gain index Based on this, the agency model With the target model The paradigm compatibility between the two methods is confirmed in terms of output structure type, training target prototype, loss function type, and topology type under specific conditions.

[0140] A system for an adaptive data quality assessment method for surrogate models based on model paradigm feature matching includes: a rule construction module for constructing rule versions containing a label set, paradigm compatibility judgment rules, adaptive selection functions, a surrogate model pool, and an evaluation task set library;

[0141] The technical parameter acquisition and adaptation module is used to obtain the technical description information of the target model, extract the target model paradigm feature vector, and call the adaptive selection function to determine the adapted proxy model and evaluation task set.

[0142] The training and evaluation module is used to drive the training of the agent model, obtain baseline performance scores and evaluation performance scores, and calculate the relative performance gain index.

[0143] The evaluation conclusion generation module is used to match the relative performance gain index with a preset quality level threshold mapping table to generate a data quality evaluation conclusion that includes paradigm compatibility confirmation information.

[0144] A computer-readable storage medium for storing a program of a proxy model adaptive data quality assessment method based on model paradigm feature matching, wherein the computer-readable storage medium, when executed by a processor, implements all the method steps in steps 1-4.

[0145] Compared with the prior art, the beneficial effects of the present invention are:

[0146] 1. This invention fills the technical gap of lacking objective basis for proxy model selection in third-party certification scenarios by establishing standardized technical rules for determining paradigm compatibility between proxy models and target models. At the same time, it transforms the proxy model selection process from a default decision within the team into a paradigm compatibility determination rule system consisting of four dimensions: output structure type, training target prototype, topology structure type, and loss function type. This ensures that when using a proxy model to replace the target model for data quality assessment, the proxy model selection process for establishing the credibility of the proxy model assessment results will not result in the lack of a paradigm compatibility determination rule system, which would prevent the technical rationality of the assessment conclusion from being independently verified externally. This provides a technical foundation to support the compliance and credibility of the assessment conclusion.

[0147] 2. By defining a finitely expandable set of labels, when a model feature type not present in the existing set is encountered, the system automatically generates a new label based on the mathematical structure of the feature and incorporates it into the label set, thus keeping the classification and judgment process closed. This ensures that the constructed rule version has continuous adaptability to any new model paradigm and there is no technical blind spot where paradigm compatibility judgment cannot be completed because the model type is not in the preset list.

[0148] 3. This invention achieves standardization, reproducibility, and independent auditability of the proxy model selection process through an adaptive selection function for the proxy model and a version-managed proxy model pool and evaluation task set library. For the same target model feature vector input, the selection result is uniquely determined. At the same time, the evaluation conclusion is required to include a paradigm compatibility proof of the proxy model selection, so that the certification conclusion issued based on this invention has verifiable technical evidence effect in various audit verifications. Attached Figure Description

[0149] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0150] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0151] This invention provides the following: Figure 1 The method for adaptive data quality assessment of surrogate models based on model paradigm feature matching, as shown, includes the following steps:

[0152] Step 1: Construct a multi-dimensional paradigm feature label set, which includes four paradigm dimensions: model output structure, training objective, architecture topology, and loss functional. Define paradigm compatibility judgment rules to verify each paradigm dimension and determine the compatibility between the target model and the proxy model in terms of paradigm feature dimensions. Define an adaptive selection function to select a suitable proxy model and its corresponding evaluation task set from the candidate proxy model set based on the compatibility judgment results. Construct a rule version that includes at least the label set, judgment rules, selection function, proxy model pool, and evaluation task set library.

[0153] Step 2: Collect the technical description information of the target model, extract the paradigm feature vector of the target model, and determine the appropriate proxy model and evaluation task set based on the adaptive selection function;

[0154] Step 3: Drive the adapted proxy model to perform baseline training without using the dataset to be evaluated and fine-tuning training with the dataset to be evaluated, respectively, and obtain the relative performance gain index of the dataset to be evaluated relative to the baseline performance.

[0155] Step 4: Generate a data quality assessment conclusion based on the relative performance gain index and the preset threshold.

[0156] Specifically, step 1: Construct a label set based on multi-dimensional paradigm features. Perform paradigm compatibility The determination is based on the definition of an adaptive selection function. and output the proxy model. and evaluation task set To construct a rule version, the following steps are involved:

[0157] 1.1 Select the tag set The specific steps are as follows:

[0158] 1.1.1, Arbitrary model Mapped to its paradigm eigenvector The specific formula is as follows:

[0159] ;

[0160] in, For any model paradigm feature vector, A mapping function for the paradigm feature extraction function of any model;

[0161] The task paradigm feature function is used to describe the mathematical structure of the model's output samples and the mathematical prototype of the training target.

[0162] These are architectural paradigm feature functions used to describe the topological structure of information transmission within the model;

[0163] is the feature function of the loss paradigm, used to describe the mathematical functional form of the model training loss function;

[0164] , , The value range of the three core paradigm features includes: the output structure label set. Training target prototype label set Architecture topology tag set Loss functional tag set All are tag sets that can be expanded to a limited extent. They initially contain known tags and can be dynamically incorporated into various new types that will emerge in the future through automatic generation or manual configuration.

[0165] Among them, task paradigm features Architectural Paradigm Characteristics and loss paradigm characteristics The extraction process is as follows:

[0166] 1.1.2 Extracting Task Paradigm Features The specific calculation formula is as follows:

[0167] ;

[0168] in, For the model The output sample structure type is determined by: taking the model A single output sample for any valid input According to the rules, extract the complete set of output structure type tags. The specific rules for assigning tags are as follows:

[0169] H1, if Each item ( All are taken from a finite set of symbols. That is, the output is a variable-length discrete symbol sequence. ;

[0170] in, For a single output sample The sequence length, ; This is an assignment operator; for The label in the middle represents the member label of the output of the variable-length discrete sequence;

[0171] H2, if That is, the output is a fixed-dimensional real vector. ;

[0172] in, For fixed-dimensional real vectors, These are exclusive tags for vector-type outputs.

[0173] H3, if That is, the output is discrete category labels. ;

[0174] in, For a discrete finite set of integers, This represents the total number of categories. Output labels for single discrete classification;

[0175] H4, Otherwise, according to The mathematical structure generates a unique formal identifier corresponding to a new structure label. ,make , and place ;

[0176] Among them, mathematical structure includes metric space, dimensional properties, and data type;

[0177] in, For the model The mathematical prototype of the training objective is determined by: based on the model Loss function optimized during training The functional prototype, from the training target prototype label set The corresponding training target prototype labels are assigned in the following way:

[0178] K1, if Equivalent to autoregressive likelihood maximization, i.e., for data taken from the distribution Given a sample sequence, the optimization objective is to minimize the negative log-conditional probability. exist The expectation above, then ;

[0179] in, for Position index in the sequence; Indicates position All previous elements; For the model Parameters;

[0180] For training target prototype label set Pre-defined standard labels for autoregressive likelihood maximization classes;

[0181] K2, if This is equivalent to maximizing the contrast similarity, which depends on the positive and negative sample pairs in... ,and The optimization objective is as follows: ;

[0182] in, It is a distance metric function in vector space, and its output is a nonnegative scalar; This is a preset interval hyperparameter used to control the minimum distance boundary between positive and negative sample pairs, and ;

[0183] For training target prototype label set Pre-defined standard labels that maximize the comparison similarity;

[0184] K3, if This is equivalent to minimizing the classification error rate, i.e., for values ​​taken from the joint distribution. Sample label pairs The optimization objective is to minimize the negative log-likelihood of the categories in the classification task. In distribution The expectation above, then ;

[0185] in For the input sample feature vector, For the corresponding real category label;

[0186] For training target prototype label set The pre-defined standard label minimizes the classification error rate;

[0187] K4, if This is equivalent to cross-modal alignment loss, which simultaneously processes the representation spaces of two or more different modalities and minimizes the cross-modal distance metric. ;

[0188] For training target prototype label set Pre-defined cross-modal alignment class standard tags;

[0189] K5. Otherwise, according to The basic operational structure, distance kernel type, and constraint term structure generate unique formal identifiers corresponding to new training target prototype labels. ,make , and place ;

[0190] 1.1.3 Extracting architectural paradigm features The specific calculation formula is as follows:

[0191] ;

[0192] in, It is a set of nodes, where each node corresponds to a processing unit. It is a set of directed edges, where each directed edge represents the information flow from the predecessor node to the successor node;

[0193] and For the model The topology type, whose value is the architecture topology tag set. The elements in the model are determined based on the following criteria. Calculation graph Specifically;

[0194] Y1, if If the graph is acyclic feedforward, meaning there is no directed path from any node back to itself or forming a cycle, then... ;

[0195] For architecture topology tag set Pre-defined standard labels for acyclic feedforward topologies;

[0196] Y2. If If there exists a bidirectional directed path between any two distinct nodes, i.e., fully bidirectionally connected, then... ;

[0197] For architecture topology tag set Pre-defined standard tags for fully bidirectional connected topology classes;

[0198] Y3, if Can be divided into encoder subgraphs and decoder subgraph And there exists a fixed-dimensional intermediate representation. And satisfy , ,but ;

[0199] in Encoder subgraph Input samples, Decoder subgraph The final output result; For architecture topology tag set The pre-built encoder-decoder topology class standard label;

[0200] Y4, if If it contains at least two heterogeneous input subnetworks and there are information fusion nodes that span across the subnetworks, then ;

[0201] For architecture topology tag set The pre-set standard label corresponds to the topology of the multimodal fusion computing graph containing heterogeneous input subnetworks and cross-subnetwork information fusion nodes, and matches the judgment rule of step Y4;

[0202] Y5, otherwise according to The adjacency matrix type, loop characteristics, and connection constraints generate unique formal identifiers corresponding to new labels. ,join in juxtaposition ;

[0203] in, For architecture topology tag set Dynamically added custom topology labels are only used to match special model computation graph structures that do not have preset standard labels;

[0204] 1.1.4 Extracting Loss Paradigm Features The specific calculation formula is as follows:

[0205] ;

[0206] in, For the model The training loss functional;

[0207] Its determination method is based on the training loss function. The mathematical functional form determines this, specifically:

[0208] S1, if Essentially, it involves calculating the distance or divergence between two probability distributions. ;

[0209] It is a loss functional tag set The pre-defined standard labels in the data correspond to loss function types that are based on the calculation of distance / divergence between two probability distributions, and match the judgment rules of step S1.

[0210] S2, if Based on the comparison construction of representation vectors, and having the form Or equivalent to it, then ;

[0211] in, It is a loss functional tag set Pre-defined standard tags for comparison construction; This is the representation vector of the anchor point sample; This is the representation vector of the positive samples; The representation vector of the negative samples; Here, is a distance metric function on a vector space, and its output is a nonnegative scalar; The preset interval hyperparameter is still used to control the minimum distance boundary between positive and negative sample pairs, and ;

[0212] S3, if The deviation between the predicted value and the target value is calculated element by element, using the general formula: ,but ;

[0213] in, For the first The predicted value of each element; For the first The target value of each element; This is a deviation measurement function; It is a loss functional tag set The pre-defined standard label for element-wise deviation loss is the loss function that calculates the deviation between the predicted value and the true target scalar value for each position.

[0214] S4. If If it includes the global normalization term for the entire sequence, then ;

[0215] For loss functional tag set The pre-defined standard label for global normalization loss class refers to the loss function that includes the normalization term of the whole sequence partition function;

[0216] Includes the global normalization term across the entire sequence, including the partition function in conditional random fields. ;

[0217] in Input sequence for the loss function calculation stage The normalized sum of all possible output sequence states of the model;

[0218] S5. Otherwise, according to Basic operations, distance term types, and regularization term types generate unique formal identifiers for new loss functional labels. ,join in juxtaposition ;

[0219] 1.1.5. Based on the output structure tag set Training target prototype label set Architecture topology tag set Loss functional tag set Get the tag set ;

[0220] The tag set supports both automatic generation by the program and manual customization and expansion. The matching rules for custom tags all fall within the feature extraction framework of this paradigm.

[0221] 1.2. Perform a normal form compatibility check. The specific steps are as follows:

[0222] 1.2.1 Define normal form compatibility as The formula for determining this is: ;

[0223] in, For the paradigm feature space, An output of 1 indicates that the two eigenvectors are "compatible", and an output of 0 indicates that the two eigenvectors are "incompatible".

[0224] 1.2.2. For the feature vector of the target model paradigm and surrogate model paradigm feature vector If all of the following core conditions are met, then the product is considered compatible. ;

[0225] Tag equality comparison rule: If any tag originates from a catch-all newly added tag, i.e. , , , Elements within a tag are judged based on their unique formal identifier. Two tags are considered equal if their formal identifiers are identical or they have a one-to-one mapping relationship; otherwise, they are considered unequal. The specific conditions are as follows:

[0226] (1) The output structure types are equal, that is ;

[0227] (2) The training target prototypes are equal, that is ;

[0228] (3) The loss function types are the same, that is ;

[0229] like Then additional requirements ;

[0230] like Therefore, equal architectures are not mandatory.

[0231] in, For the target model, For proxy models in the proxy model pool;

[0232] The additional matching conditions of the architecture can be adjusted equivalently according to the task scenario, and all equivalent constraint logics fall within the compatibility determination scope of this application.

[0233] 1.3. Define the adaptive selection function and output the proxy model. and evaluation task set The specific steps are as follows:

[0234] 1.3.1 Establishing a proxy model pool With evaluation task set library Both are version-based, specifically:

[0235] When there are new additions, adjustments, or other changes to the proxy model pool or evaluation task set library, the newly added proxy model must have its task paradigm features, architecture paradigm features, and loss paradigm features extracted and recorded in advance. The newly added evaluation task set must have its applicable task paradigm tuples identified and recorded before it can be included in subsequent versions of the corresponding proxy model pool or evaluation task set library.

[0236] Among them, for the first One proxy model entry The record identifier is The eigenvectors of the paradigm are Priority weight is ;

[0237] in, ,and ;

[0238] The priority weight has a preset correspondence with the priority, and the smaller the value, the higher the priority.

[0239] For the Each evaluation task set item The record identifier is The applicable task paradigm binary is ( , ), priority weight is ;

[0240] in, ,and ;

[0241] Labels indicating the output structure type applicable to this evaluation task set; Prototype labels for training objectives applicable to this evaluation task set;

[0242] The priority weight has a preset correspondence with the priority, and the smaller the value, the higher the priority.

[0243] 1.3.2 Input target model feature vector Output proxy model With evaluation task set The specific steps are as follows:

[0244] L1. Define the adaptive selection function Specifically:

[0245] ;

[0246] L2. Select the proxy model using the following formula:

[0247] ;

[0248] in, The surrogate model set consists of all surrogate model entries in the surrogate model pool that satisfy paradigm compatibility. , For the first The paradigm feature vector of each proxy model entry; if This triggers a request to expand the proxy model pool;

[0249] L3. Select the candidate evaluation task set, using the following formula:

[0250] ;

[0251] in, This is a candidate evaluation task set, whose elements are all evaluation task entries in the evaluation task set library that match the target model with the applicable task paradigm features. ; The applicable output structure type recorded in the evaluation task set library for each evaluation task set entry. The applicable output training target prototypes recorded in the evaluation task set library for the evaluation task set entries;

[0252] like This triggers a request to expand the evaluation task set library and terminates the process.

[0253] L4. Outputting the proxy model based on priority selection. and evaluation task set The specific formula is as follows:

[0254] ;

[0255] ;

[0256] in For the selected proxy model, For the selected set of evaluation tasks; For the proxy model Priority weights, For the evaluation task set The priority weight is determined by the rule that if multiple items have the same priority, one of them will be selected according to the preset rules.

[0257] Furthermore, the priority selection rule can be replaced by a conventional equivalent sorting method in the field, and any rule that does not change the core logic of paradigm compatibility screening is protected by this application.

[0258] 1.4. Based on tag sets Paradigm compatibility Adaptive selection function Proxy model pool and evaluation task collection library Together, they constitute the rule version;

[0259] Any addition, deletion, or modification of any of the above elements will trigger a version update and record a change log. The latest version currently in effect will be used during the evaluation to ensure that the process is traceable and auditable.

[0260] In step 1, the constructed rule version enables traceability of the entire process assessment logic and controllable rule changes, while ensuring standardized and unified deployment of assessment rules across multiple business scenarios to meet compliance audit and result reproduction requirements.

[0261] Step 2: Collect target model Technical description information, and determine the proxy model. With evaluation task set The specific steps are as follows:

[0262] 2.1 Obtain the target model to be evaluated Technical description information;

[0263] The technical description information includes the output sample structure of the target model, the prototype of the training target functional, the topology of the internal computation graph, and the training loss function; and the technical description information must meet the internal consistency, that is, the technical relationship between the output sample structure, the prototype of the training target functional, the topology of the internal computation graph, and the training loss functional conforms to the common design paradigm of the model. If information contradictions are detected, reconfirmation is required.

[0264] 2.2. Based on the extraction and analysis of technical description information, the target model is obtained. Normative eigenvectors The specific formula is as follows:

[0265] ;

[0266] For the target model Task paradigm characteristics, by and Composition, and For the target model The output sample structure type; For the target model The mathematical prototype of the training objective; For the target model The architectural paradigm features, derived from the target model Topology type express; For the target model The loss paradigm characteristics are derived from the target model. Training loss functional express;

[0267] 2.3. The feature vector of the target model paradigm As input, the adaptive selection function is called. In the proxy model pool and evaluation task set library Determine the appropriate proxy model and evaluation task set The specific calculation formula is as follows:

[0268] ;

[0269] Among them, the selected proxy model With the target model The normal form compatibility criteria must be met between them. ,Right now: ; ;

[0270] If and only if Additional requirements .

[0271] In step 2, a unified cross-model quantitative comparison benchmark was established by unifying the paradigm feature vectors and adding topological structure matching constraints. This eliminates the differences in feature representation dimensions between different models and ensures that the selection criteria for proxy models are consistent. At the same time, forced topological alignment of the architecture is applied to image-related tasks to avoid evaluation distortion caused by differences in the underlying structure and improve the accuracy of proxy matching. Furthermore, the paradigm similarity is made calculable and reproducible by relying on structured feature vectors, providing an objective quantitative basis for proxy model selection.

[0272] Step 3: Driving the Proxy Model Perform training and obtain the dataset to be evaluated. Relative performance gain index The specific steps are as follows:

[0273] 3.1. Based on the selected proxy model Load its initial weights and configure the standard training hyperparameter set, specifically: obtain the surrogate model in the preset computing environment. Baseline performance score ;

[0274] Among them, the baseline performance score The acquisition methods include: using general benchmark datasets Training agent model Subsequently, in the evaluation task set The above evaluation results; or the proxy model Without undergoing any task-related fine-tuning, directly evaluate the task set. The above evaluation results are as follows;

[0275] Among them, the baseline performance score The acquisition of the proxy model Based on the principle of initial capabilities before accessing data in the domain to be evaluated, the baseline and evaluation phases use completely consistent standard training hyperparameter sets and secure computing environments.

[0276] The standard training hyperparameter set configured here is limited to the baseline and evaluation training uniform configuration only.

[0277] 3.2 Using the dataset to be evaluated Train the surrogate model with a configuration identical to the baseline training. After training, on the same evaluation task set The evaluation results were obtained through testing. ;

[0278] 3.3 Based on baseline performance scores and performance evaluation score The dataset to be evaluated is obtained. Relative performance gain index The specific formula is as follows:

[0279] ;

[0280] Among them, the relative performance gain index The meaning is: in the evaluation task set Within the defined task metric space, any higher value indicates better model performance. The relative rate of change of a monotonic scalar metric before and after training on the dataset to be evaluated is defined as follows: This definition is closed-ended; that is, regardless of the specific performance metric chosen for the evaluation task set, such as accuracy, F1 score, perplexity reduction rate, BLEU score, MSE reduction rate, or other performance metrics, as long as the chosen performance metric satisfies the aforementioned monotonicity and scalar properties, its relative rate of change will fall within the defined task metric space. Within the calculation range, for indicators where lower original values ​​indicate better performance, their monotonic transformed form is used as the performance score and substituted into the formula to ensure... The positive and negative directions are consistent with the meaning of performance gain.

[0281] In step 3, a common relative performance gain index is adopted. It is compatible with various evaluation metrics such as accuracy, F1, and BLEU, and can be adapted to multiple evaluation scenarios. At the same time, it has built-in metric conversion logic to unify the direction of gain determination. Furthermore, it removes baseline interference by relying on the rate of change before and after training, and objectively quantifies the optimization effect of the dataset.

[0282] Step 4: Based on the relative performance gain index The evaluation conclusion, which includes the selection criteria, is generated based on preset thresholds. The specific steps are as follows:

[0283] 4.1 Preset quality level threshold mapping table;

[0284] The quality grade threshold mapping table defines a set of ordered thresholds. and the corresponding quality grade label ;

[0285] in, For the first Level threshold, ,satisfy , The total number of quality grades; the quality grade threshold mapping table is determined based on historical evaluation experience data or industry statistical benchmarks.

[0286] 4.2 The relative performance gain index Match the data with a quality grade threshold mapping table and assign quality grade labels according to a preset interval-grade correspondence. ;

[0287] 4.3 Based on the above judgment, a data quality assessment conclusion is generated, specifically:

[0288] The quality level judgment included in the data quality assessment conclusion is based on a proxy model. Relative performance gain index Based on this, the agency model With the target model The paradigm compatibility between them is confirmed in terms of output structure type, training target prototype, loss function type, and topology type under specific conditions;

[0289] In step 4, the paradigm compatibility determination is used as the technical premise for its generation. This technical premise constitutes a necessary condition for the data quality assessment conclusion to be valid in technical logic. That is, if the premise is not verified, the data quality assessment conclusion derived from the relative performance gain index of the proxy model will not have projectibility to the target model.

[0290] The data quality assessment conclusions can exist in the form of structured data objects or assessment reports, and their integrity and immutability can be ensured through digital signatures or hash verification, thereby achieving technical auditability; at the same time, the quantitative values ​​are converted into standardized quality labels, making the results intuitive and easy to interpret; it can support dual output of structured data and written reports, adapt to automated and manual review, and hash and digital signature lock the entire chain of data, meeting the requirements for compliance audit traceability, and retaining the complete selection judgment basis for easy reproduction.

[0291] A system for an adaptive data quality assessment method for surrogate models based on model paradigm feature matching includes:

[0292] The rule building module is used to build rule versions that include a tag set, paradigm compatibility judgment rules, adaptive selection functions, a proxy model pool, and an evaluation task set library;

[0293] The technical parameter acquisition and adaptation module is used to obtain the technical description information of the target model, extract the target model paradigm feature vector, and call the adaptive selection function to determine the adapted proxy model and evaluation task set.

[0294] The training and evaluation module is used to drive the training of the agent model, obtain baseline performance scores and evaluation performance scores, and calculate the relative performance gain index.

[0295] The evaluation conclusion generation module is used to match the relative performance gain index with a preset quality level threshold mapping table to generate a data quality evaluation conclusion that includes paradigm compatibility confirmation information.

[0296] A computer-readable storage medium for storing a program of a proxy model adaptive data quality assessment method based on model paradigm feature matching, wherein the computer-readable storage medium, when executed by a processor, implements all the method steps in steps 1-4.

[0297] Verification experiment:

[0298] To verify the technical effects of the present invention, the following comparative experiments illustrate the advantages of the present invention over the prior art in terms of the rationality of agent model selection, evaluation accuracy, and adaptive expansion capability of the rule system.

[0299] I. Experimental Setup:

[0300] 1. Target Model:

[0301] An autoregressive large language model with 13B0 parameters was selected. The task paradigm is AR (autoregressive language modeling), the architecture paradigm is FF (feedforward decoder), and the loss paradigm is DIV cross-entropy divergence class. The target large model is fully trained and evaluated on each dataset to be evaluated, and its measured performance gain value is used as the baseline true value.

[0302] 2. Proxy Model:

[0303] The proposed model is a lightweight feedforward decoder model with 18M parameters, in which each paradigm component is consistent with the target large model, satisfying the paradigm compatibility condition.

[0304] Incorrect choice 1: The encoder-decoder model with 18M parameters violates the compatibility condition.

[0305] Second incorrect choice: The 18M-parameter feedforward classification model violates the compatibility condition.

[0306] 3. Data set to be evaluated:

[0307] Fifteen text datasets with known quality levels (3 high-quality, 3 medium-quality, and 9 low-quality), ranging in size from 300M to 50B tokens, of which the first 12 are calibration sets and the last 3 are test sets;

[0308] Low-quality datasets include samples with a high proportion of noise (more than 30% random replacements) and machine-generated text to widen the quality differences between datasets and make it easier to distinguish the performance gains of surrogate models.

[0309] 4. Evaluation Task Set:

[0310] We have released a set of language comprehension and reasoning evaluation tasks, using the perplexity reduction rate as a performance metric.

[0311] Comparison method:

[0312] Error Selection Evaluation: Use Error Selection 1 and 2 to evaluate the test set respectively, and directly output the performance gain of the proxy model itself;

[0313] The method of this invention is as follows: Steps 1 to 4 are executed, followed by adaptive selection and evaluation.

[0314] II. Experimental Results:

[0315] 1. Comparison of assessment accuracy and resource consumption, the specific results are shown in Table 1 below;

[0316] Table 1:

[0317]

[0318] As shown in Table 1, although full training of the target large model can obtain the baseline true value, a single evaluation takes about 420 hours (about 17.5 days), which is not feasible in certification evaluation business. The correlation coefficient between the evaluation results of the method of this invention and the true value is 0.86, the average absolute error is only 0.12, and a single evaluation only takes about 3.2 hours (about half a day), and the computing power consumption is about one-thirty-sixth of that of full training.

[0319] Verification of the adaptive expansion capability of the rule system:

[0320] To verify the adaptive capability of the rule system to unforeseen model types, a model type not included in the existing output structure label set is constructed: its output sample is a graph structure (a combination of node set and edge set). The technical description of this model is input into the feature extraction function in step 1.1.

[0321] The extraction function determines that the three known structure determination conditions are not met, and then extracts the mathematical structure attribute of the output sample—the metric space is the Cartesian product of the node set and the edge set—and generates a unique formal identifier as a new label to be added to the label set. The process continues to execute based on the expanded label set and completes the compatibility determination. When the same type of model is input again, the system directly uses the existing new label to complete the determination.

[0322] The above results show that when encountering unforeseen model paradigm types, the rule system of this invention can automatically generate new labels and incorporate them into the label set through a formal description of the mathematical structure of the output samples, thus keeping the classification and judgment process closed. This adaptive expansion mechanism is also applicable to the training target prototype label set, the architecture topology label set, and the loss functional label set. When an unknown type appears in any dimension, new labels are generated in the same way and the judgment process is not interrupted.

[0323] In summary, this invention fills the technical gap of lacking objective basis for proxy model selection in third-party certification scenarios by establishing standardized technical rules for determining paradigm compatibility between proxy models and target models. Furthermore, since proxy model selection in existing internal screening scenarios is decided internally by default without external explanation, this invention transforms this implicit premise into a paradigm compatibility determination rule system composed of four dimensions: output structure type, training target prototype, topology structure type, and loss function type. This ensures that any proxy model selection process used to establish the credibility of proxy model evaluation results when using a proxy model to replace the target model for data quality assessment will not suffer from a lack of a paradigm compatibility determination rule system, preventing the technical rationality of the evaluation conclusions from being independently verified externally. This provides a technical foundation for the compliance and credibility of the evaluation conclusions.

[0324] Furthermore, by defining a finitely expandable set of labels, the system covers known types and provides an adaptive expansion mechanism across three paradigm dimensions: task paradigm feature function, architecture paradigm feature function, and loss paradigm feature function. When encountering a model feature type that is not present in the existing label set, the system automatically generates a new label based on the mathematical structure of the feature and incorporates it into the label set, keeping the classification and judgment process closed. This ensures that the constructed rule version is continuously adaptable to any new model paradigm, avoids rule failure due to model technology evolution, and eliminates the technical blind spot of being unable to complete paradigm compatibility judgment due to the model type not being in the preset list.

[0325] Furthermore, by using an adaptive selection function for the proxy model and a version-managed proxy model pool and evaluation task set library, the proxy model selection process is standardized, reproducible, and independently auditable. For the same target model feature vector input, the selection result is uniquely determined. At the same time, the evaluation conclusion is required to include a paradigm compatibility proof of the proxy model selection, enabling any third party to independently reproduce and verify the correctness of the selection based on the publicly available rule version, detached from the original certification platform. This makes the certification conclusions issued based on this invention verifiable technical evidence in various audit verifications, and provides a technical credibility advantage for data quality assessment conclusions issued using non-standardized selection methods.

[0326] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used above are only some embodiments described in this invention. Obviously, those skilled in the art can obtain other drawings based on these drawings.

[0327] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. An adaptive data quality assessment method for surrogate models based on model paradigm feature matching, characterized in that: Includes the following steps: Step 1: Construct a label set based on multi-dimensional paradigm features Perform paradigm compatibility The determination is based on the definition of an adaptive selection function. and output the proxy model. and evaluation task set A rule version is constructed; Step 2: Collect target model Technical description information, and determine the proxy model. With evaluation task set ; Step 3: Driving the agent model Perform training and obtain the dataset to be evaluated. Relative performance gain index ; Step 4: Based on the relative performance gain index The system generates evaluation conclusions that include the selection criteria based on preset thresholds.

2. The adaptive data quality assessment method for surrogate models based on model paradigm feature matching according to claim 1, characterized in that: In step 1, a label set is constructed based on multi-dimensional paradigm features. Perform paradigm compatibility The determination is based on the definition of an adaptive selection function. and output the proxy model. and evaluation task set To construct a rule version, the following steps are involved: 1.1 Select the tag set ; 1.2 Perform paradigm compatibility testing The determination; 1.

3. Define the adaptive selection function and output the proxy model. and evaluation task set ; 1.

4. Based on tag sets Paradigm compatibility Adaptive selection function Proxy model pool and evaluation task collection library Together, they constitute the rule version.

3. The adaptive data quality assessment method for surrogate models based on model paradigm feature matching according to claim 2, characterized in that: In step 1.1, the tag set is selected. The specific steps are as follows: 1.1.1, Arbitrary model Mapped to its paradigm eigenvector The specific formula is as follows: ； in, For any model paradigm feature vector, A mapping function for the paradigm feature extraction function of any model; The task paradigm feature function is used to describe the mathematical structure of the model's output samples and the mathematical prototype of the training target. These are architectural paradigm feature functions used to describe the topological structure of information transmission within the model; is the feature function of the loss paradigm, used to describe the mathematical functional form of the model training loss function; , , The value range of the three core paradigm features includes: the output structure label set. Training target prototype label set Architecture topology tag set Loss functional tag set All are tag sets that can be expanded to a limited extent. They initially contain known tags and can be dynamically incorporated into various new types that will emerge in the future through automatic generation or manual configuration. Among them, task paradigm features Architectural Paradigm Characteristics and loss paradigm characteristics The extraction process is as follows: 1.1.2 Extracting Task Paradigm Features The specific calculation formula is as follows: ； in, For the model The output sample structure type is determined by: taking the model A single output sample for any valid input According to the rules, extract the complete set of output structure type tags. The specific rules for assigning tags are as follows: H1, if Each item ( All are taken from a finite set of symbols. That is, the output is a variable-length discrete symbol sequence. ; in, For a single output sample The sequence length, ; This is an assignment operator; for The label in the middle represents the member label of the output of the variable-length discrete sequence; H2, if That is, the output is a fixed-dimensional real vector. ; in, For fixed-dimensional real vectors, These are exclusive tags for vector-type outputs. H3, if That is, the output is discrete category labels. ; in, For a discrete finite set of integers, This represents the total number of categories. Output labels for single discrete classification; H4, Otherwise, according to The mathematical structure generates a unique formal identifier corresponding to a new structure label. ,make , and place ; Among them, mathematical structure includes metric space, dimensional properties, and data type; in, For the model The mathematical prototype of the training objective is determined by: based on the model Loss function optimized during training The functional prototype, from the training target prototype label set The corresponding training target prototype labels are assigned in the following way: K1, if Equivalent to autoregressive likelihood maximization, i.e., for data taken from the distribution Given a sample sequence, the optimization objective is to minimize the negative log-conditional probability. exist The expectation above, then ; in, for Position index in the sequence; Indicates position All previous elements; For the model Parameters; For training target prototype label set Pre-defined standard labels for autoregressive likelihood maximization classes; K2, if This is equivalent to maximizing the contrast similarity, which depends on the positive and negative sample pairs in... ,and The optimization objective is as follows: ; in, It is a distance metric function in vector space, and its output is a nonnegative scalar; This is a preset interval hyperparameter used to control the minimum distance boundary between positive and negative sample pairs, and ; For training target prototype label set Pre-defined standard labels that maximize the comparison similarity; K3, if This is equivalent to minimizing the classification error rate, i.e., for values ​​taken from the joint distribution. Sample label pairs The optimization objective is to minimize the negative log-likelihood of the categories in the classification task. In distribution The expectation above, then ; in For the input sample feature vector, For the corresponding real category label; For training target prototype label set The pre-defined standard label minimizes the classification error rate; K4, if This is equivalent to cross-modal alignment loss, which simultaneously processes the representation spaces of two or more different modalities and minimizes the cross-modal distance metric. ; For training target prototype label set Pre-defined cross-modal alignment class standard tags; K5. Otherwise, according to The basic operational structure, distance kernel type, and constraint term structure generate unique formal identifiers corresponding to new training target prototype labels. ,make , and place ; 1.1.3 Extracting architectural paradigm features The specific calculation formula is as follows: ； in, It is a set of nodes, where each node corresponds to a processing unit. It is a set of directed edges, where each directed edge represents the information flow from the predecessor node to the successor node; and For the model The topology type, whose value is the architecture topology tag set. The elements in the model are determined based on the following criteria. Calculation graph Specifically: Y1, if If the graph is acyclic feedforward, meaning there is no directed path from any node back to itself or forming a cycle, then... ; For architecture topology tag set Pre-defined standard labels for acyclic feedforward topologies; Y2. If If there exists a bidirectional directed path between any two distinct nodes, i.e., fully bidirectionally connected, then... ; For architecture topology tag set Pre-defined standard tags for fully bidirectional connected topology classes; Y3, if Can be divided into encoder subgraphs and decoder subgraph And there exists a fixed-dimensional intermediate representation. And satisfy , ,but ; in Encoder subgraph Input samples, Decoder subgraph The final output result; For architecture topology tag set The pre-built encoder-decoder topology class standard label; Y4, if If it contains at least two heterogeneous input subnetworks and there are information fusion nodes that span across the subnetworks, then ; For architecture topology tag set The pre-set standard label corresponds to the topology of the multimodal fusion computing graph containing heterogeneous input subnetworks and cross-subnetwork information fusion nodes, and matches the judgment rule of step Y4; Y5, otherwise according to The adjacency matrix type, loop characteristics, and connection constraints generate unique formal identifiers corresponding to new labels. ,join in juxtaposition ; in, For architecture topology tag set Dynamically added custom topology labels are only used to match special model computation graph structures that do not have preset standard labels; 1.1.4 Extracting Loss Paradigm Features The specific calculation formula is as follows: ； in, For the model The training loss functional; Its determination method is based on the training loss function. The mathematical functional form determines this, specifically: S1, if Essentially, it involves calculating the distance or divergence between two probability distributions. ; It is a loss functional tag set The pre-defined standard labels in the data correspond to loss function types that are based on the calculation of distance / divergence between two probability distributions, and match the judgment rules of step S1. S2, if Based on the comparison construction of representation vectors, and having the form Or equivalent to it, then ; in, It is a loss functional tag set Pre-defined standard tags for comparison construction; This is the representation vector of the anchor point sample; This is the representation vector of the positive samples; The representation vector of the negative samples; Here, is a distance metric function on a vector space, and its output is a nonnegative scalar; The preset interval hyperparameter is still used to control the minimum distance boundary between positive and negative sample pairs, and ; S3, if The deviation between the predicted value and the target value is calculated element by element, using the general formula: ,but ; in, For the first The predicted value of each element; For the first The target value of each element; This is a deviation measurement function; It is a loss functional tag set The pre-defined standard label for element-wise deviation loss is the loss function that calculates the deviation between the predicted value and the true target scalar value for each position. S4. If If it includes the global normalization term for the entire sequence, then ; For loss functional tag set The pre-defined standard label for global normalization loss class refers to the loss function that includes the normalization term of the whole sequence partition function; Includes the global normalization term across the entire sequence, including the partition function in conditional random fields. ; in Input sequence for the loss function calculation stage The normalized sum of all possible output sequence states of the model; S5. Otherwise, according to Basic operations, distance term types, and regularization term types generate unique formal identifiers for new loss functional labels. ,join in juxtaposition ; 1.1.

5. Based on the output structure tag set Training target prototype label set Architecture topology tag set Loss functional tag set Get the tag set .

4. The surrogate model adaptive data quality assessment method based on model paradigm feature matching according to claim 3, characterized in that: in step 1.2, the paradigm compatibility is determined, and the specific steps are as follows: 1.2.1 Define normal form compatibility as The formula for determining this is: ; in, For the paradigm feature space, An output of 1 indicates that the two eigenvectors are "compatible", and an output of 0 indicates that the two eigenvectors are "incompatible". 1.2.

2. For the feature vector of the target model paradigm and surrogate model paradigm feature vector If all of the following core conditions are met, then the product is considered compatible. ; Tag equality comparison rule: If any tag originates from a catch-all newly added tag, i.e. , , , Elements within a tag are judged based on their unique formal identifier. Two tags are considered equal if their formal identifiers are identical or they have a one-to-one mapping relationship; otherwise, they are considered unequal. The specific conditions are as follows: (1) The output structure types are equal, that is ; (2) The training target prototypes are equal, that is ; (3) The loss function types are the same, that is ; like Then additional requirements ; like Therefore, equal architectures are not mandatory. in, For the target model, For proxy models in the proxy model pool.

5. The adaptive data quality assessment method for surrogate models based on model paradigm feature matching according to claim 4, characterized in that: in step 1.3, an adaptive selection function is defined. and output the proxy model. and evaluation task set The specific steps are as follows: 1.3.1 Establishing a proxy model pool With evaluation task set library Both are version-based, specifically: When there are new additions, adjustments, or other changes to the proxy model pool or evaluation task set library, the newly added proxy model must have its task paradigm features, architecture paradigm features, and loss paradigm features extracted and recorded in advance. The newly added evaluation task set must have its applicable task paradigm tuples identified and recorded before it can be included in subsequent versions of the corresponding proxy model pool or evaluation task set library. Among them, for the first One proxy model entry The record identifier is The eigenvectors of the paradigm are Priority weight is ; in, ,and ; The priority weight has a preset correspondence with the priority, and the smaller the value, the higher the priority. For the Each evaluation task set item The record identifier is The applicable task paradigm binary is ( , ), priority weight is ; in, ,and ; Labels indicating the output structure type applicable to this evaluation task set. Prototype labels for training objectives applicable to this evaluation task set; The priority weight has a preset correspondence with the priority, and the smaller the value, the higher the priority. 1.3.2 Input target model feature vector Output proxy model With evaluation task set The specific steps are as follows: L1. Define the adaptive selection function Specifically: ； L2. Select the proxy model using the following formula: ； in, The surrogate model set consists of all surrogate model entries in the surrogate model pool that satisfy paradigm compatibility. , For the first The paradigm feature vector of each proxy model entry; if This triggers a request to expand the proxy model pool; L3. Select the candidate evaluation task set, using the following formula: ； in, This is a candidate evaluation task set, whose elements are all evaluation task entries in the evaluation task set library that match the target model with the applicable task paradigm features. ; The applicable output structure type recorded in the evaluation task set library for each evaluation task set entry. The applicable output training target prototypes recorded in the evaluation task set library for the evaluation task set entries; like This triggers a request to expand the evaluation task set library and terminates the process. L4. Outputting the proxy model based on priority selection. and evaluation task set The specific formula is as follows: ； ； in For the selected proxy model, For the selected set of evaluation tasks; For the proxy model Priority weights, For the evaluation task set The priority weight is determined by the fact that if multiple items have the same priority, one of them is selected according to the preset rules.

6. The adaptive data quality assessment method for surrogate models based on model paradigm feature matching according to claim 1, characterized in that: In step 2, the target model is acquired. Technical description information, and determine the proxy model. With evaluation task set The specific steps are as follows: 2.1 Obtain the target model to be evaluated Technical description information; The technical description information includes the output sample structure of the target model, the prototype of the training target functional, the internal computation graph topology, and the training loss function. 2.

2. Based on the extraction and analysis of technical description information, the target model is obtained. Normative eigenvectors The specific formula is as follows: ； For the target model Task paradigm characteristics, by and Composition, and For the target model The output sample structure type; For the target model The mathematical prototype of the training objective; For the target model The architectural paradigm features, derived from the target model Topology type express; For the target model The loss paradigm characteristics are derived from the target model. Training loss functional express; 2.

3. The feature vector of the target model paradigm As input, the adaptive selection function is called. In the proxy model pool and evaluation task set library Determine the appropriate proxy model and evaluation task set The specific calculation formula is as follows: ； Among them, the selected proxy model With the target model The normal form compatibility criteria must be met between them. ,Right now: ; ; If and only if Additional requirements .

7. The adaptive data quality assessment method for surrogate models based on model paradigm feature matching according to claim 1, characterized in that: In step 3, the driving agent model is... Perform training and obtain the dataset to be evaluated. Relative performance gain index The specific steps are as follows: 3.

1. Based on the selected proxy model Load its initial weights and configure the standard training hyperparameter set, specifically: obtain the surrogate model in the preset computing environment. Baseline performance score ; Among them, the baseline performance score The acquisition methods include: using general benchmark datasets Training agent model Subsequently, in the evaluation task set The above evaluation results; or the proxy model Without undergoing any task-related fine-tuning, directly evaluate the task set. The above evaluation results are as follows; 3.2 Using the dataset to be evaluated Train the surrogate model with a configuration identical to the baseline training. After training, on the same evaluation task set The evaluation results were obtained through testing. ; 3.3 Based on baseline performance scores and performance evaluation score The dataset to be evaluated is obtained. Relative performance gain index The specific formula is as follows: 。 8. The adaptive data quality assessment method for surrogate models based on model paradigm feature matching according to claim 1, characterized in that: In step 4, based on the relative performance gain index The evaluation conclusion, which includes the selection criteria, is generated based on preset thresholds. The specific steps are as follows: 4.1 Preset quality level threshold mapping table; The quality grade threshold mapping table defines a set of ordered thresholds. and the corresponding quality grade label ; in, For the first Level threshold, ,satisfy , The total number of quality grades; the quality grade threshold mapping table is determined based on historical evaluation experience data or industry statistical benchmarks. 4.2 The relative performance gain index Match the data with a quality grade threshold mapping table and assign quality grade labels according to a preset interval-grade correspondence. ; 4.3 Based on the above determinations, a data quality assessment conclusion is generated. Specifically, the quality level judgment included in this data quality assessment conclusion is based on a proxy model. Relative performance gain index Based on this, the agency model With the target model The paradigm compatibility between the two methods is confirmed in terms of output structure type, training target prototype, loss function type, and topology structure type.

9. A system based on the adaptive data quality assessment method for surrogate models based on model paradigm feature matching as described in any one of claims 1-8, characterized in that, include: The rule building module is used to build rule versions that include a tag set, paradigm compatibility judgment rules, adaptive selection functions, a proxy model pool, and an evaluation task set library; The technical parameter acquisition and adaptation module is used to obtain the technical description information of the target model, extract the target model paradigm feature vector, and call the adaptive selection function to determine the adapted proxy model and evaluation task set. The training and evaluation module is used to drive the training of the agent model, obtain baseline performance scores and evaluation performance scores, and calculate the relative performance gain index. The evaluation conclusion generation module is used to match the relative performance gain index with a preset quality level threshold mapping table to generate a data quality evaluation conclusion that includes paradigm compatibility confirmation information.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store the program of the proxy model adaptive data quality assessment method based on model paradigm feature matching according to any one of claims 1-8, wherein all method steps are implemented when the computer-readable storage medium is executed by a processor.

Images