A small sample-based statistical index automatic indexing method

Abstract

The application discloses a kind of based on small sample's statistical index automatic indexing method, belong to artificial intelligence and machine learning technical field.The method includes: user is based on the feature of input data Select indexing model suitable for input data;Indexing model includes high power model and low power model, high power model is large language model, low power model includes the fusion model of model based on transfer learning and model based on meta learning, model based on meta learning;Indexing input data based on selected indexing model, obtain prediction result;Prediction result includes the statistical index corresponding to input data, the classification label corresponding to each statistical index;Verify and optimize prediction result, output optimized prediction result.The application ensures the accuracy and reliability of indexing result.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and machine learning technology, and in particular relates to an automatic indexing method and apparatus for statistical indicators based on small samples. Background Technology

[0002] With the deepening development of the Fourth Industrial Revolution, cutting-edge computer technologies such as artificial intelligence are rapidly becoming the core force driving the transformation of various industries, continuously reshaping production and management models. In many fields, such as healthcare, finance, education, and public administration, the systematic construction of evaluation systems is becoming increasingly important, driving a large amount of research progress centered on indicator indexing.

[0003] Indicators, as an important concept in statistics, are qualitative or quantitative reflections of the level of a specific object or its aspects. They are widely used in assessment, monitoring, and early warning scenarios, providing crucial references and basis for decision-making. With the continuous expansion and deepening of various businesses, the number of related indicators is growing exponentially. How to accurately index and classify different indicators under a given labeling system has become a key issue. Establishing efficient indicator indexing technology is not only the theoretical foundation for building an intelligent evaluation system, but also provides important support for comprehensively analyzing the current development status and planning future paths.

[0004] However, current mainstream indexing and classification methods typically rely on a large number of labeled samples for model training. But in some domains, pre-labeled index data is often difficult to obtain fully and completely due to proprietary technology or high sensitivity, resulting in a limited number of effective indexed samples. This situation severely restricts the training effect and generalization ability of the model. On the other hand, traditional manual indexing methods are not only costly and complex to operate, but also extremely time-consuming.

[0005] Furthermore, in practical applications, the construction of indicator systems is not static but evolves continuously with business needs and the development of the times. Researchers often need to index indicators from different dimensions and perspectives, and their category systems are frequently adjusted. There may be significant differences between old and new label systems. If a model is trained solely on the existing label set and directly applied to the new system, the model's indexing performance often fails to meet the requirements.

[0006] Therefore, how to leverage artificial intelligence and information technology to construct an indexing model that can adapt to small sample environments and support dynamic expansion of the labeling system has become a key technical problem that urgently needs to be solved. Although there are some existing methods based on machine learning and deep learning for automatic indexing, they generally suffer from limitations in generalization ability and adaptability when dealing with challenges such as sample scarcity and label expansion. Summary of the Invention

[0007] This invention proposes an automatic indexing method for statistical indicators based on small samples, which aims to improve the indexing effect of indicator indexing models and corresponding algorithms when the sample size of statistical indicators is small. In particular, it addresses the core challenges of scarce indicator samples and dynamic expansion of the label system, and proposes an indicator indexing method that can effectively utilize limited sample data and support dynamic label expansion.

[0008] The first aspect of this invention proposes an automatic indexing method for statistical indicators based on small samples, the method comprising:

[0009] Step S1: Determine the features of the input data; the user selects an indexing model suitable for the input data based on the features of the input data; wherein, the input data is small sample data, and the indexing model includes high-computing-power models and low-computing-power models. The high-computing-power model is a large language model, and the low-computing-power models include a fusion model based on transfer learning and a model based on meta-learning, and a model based on meta-learning.

[0010] Step S2: Index the input data based on the selected indexing model to obtain the prediction results; the prediction results include the statistical indicators corresponding to the input data and the classification labels corresponding to each statistical indicator;

[0011] Step S3: Verify and optimize the prediction results, and output the optimized prediction results.

[0012] Preferably, when the selected indexing model is a high-computing-power model, the step of obtaining the prediction result based on the indexing input data of the selected indexing model includes:

[0013] Step S201: Construct structured prompt words, input the structured prompt words into the trained large language model, and use the output of the large language model as the preliminary prediction result;

[0014] Step S202: Analyze the preliminary prediction results and construct a label dictionary; the label dictionary includes multiple mapping pairs, each mapping pair includes a statistical indicator and the corresponding classification label of the statistical indicator;

[0015] Step S203: Divide all mapping pairs into multiple sets, with each set representing a batch of data; the thread pool executor verifies each batch of data. If the verification result is successful, save the batch of data to the output set; if the verification result is unsuccessful, input the corresponding input data of the batch of data into the trained large model and regenerate the prediction result until the verification result of the mapping pair corresponding to the input data of the batch of data is successful.

[0016] The verification content includes: the amount of data in this batch, the degree of matching between this batch of data and the input data, and the consistency between the classification labels corresponding to this batch of data and the labels in the preset label pool;

[0017] Step S204: Use the output set as the prediction result of the input data.

[0018] Preferably, the fusion model of the transfer learning-based model and the meta-learning-based model includes parallel transfer learning-based models and meta-learning-based models; the transfer learning-based model and the meta-learning-based model index the input data through an equal-weighted voting mechanism.

[0019] Preferably, the model structure of the transfer learning-based model is the same as that of LightGBM, i.e., LightGBM is used as the classifier; the transfer learning-based model is divided into a non-oracle version of the transfer learning-based model and an oracle version of the transfer learning-based model; the source domain selection strategy of the non-oracle version of the transfer learning-based model is: calculate the median vector of each category of data in the target domain, and select the source domain data based on the distance between the source domain data and the median vector of each category of the target domain, so as to retain the source domain data whose similarity to the target domain exceeds a first preset threshold; the source domain selection strategy of the oracle version of the transfer learning-based model is a source domain data augmentation strategy, i.e., augmenting the source domain data.

[0020] Preferably, the training method for the non-oracle version of the transfer learning-based model includes:

[0021] Step S211: Use the training sample set as the target dataset; calculate the class median vector of the target dataset, and apply it to the target dataset. For each category c∈{1,2,…,C}, calculate the median of all its samples on each feature p∈{1,2,…,P}, forming the median vector m of that category in the target dataset. c =(m c (1),m c (2),…,m c (P)), where X i Let y be the semantic vector of the i-th training sample. i Let be the class label of the i-th training sample, i be the index of the training sample, n be the total number of training samples in the target dataset, C be the total number of classes, P be the total number of feature dimensions, and m be the total number of features. c (1),m c (2),…,m c (P) represents the median of category c in the 1st, 2nd, ..., Pth dimensions of the features;

[0022]

[0023] X ipLet I be the value of the i-th training sample on the p-th feature, and median be the median function. c This is the set of indices of all training samples belonging to category c;

[0024] Step S212: Extract the class median vector of each source dataset in the multi-source dataset. The source dataset refers to the training dataset from different domains or tasks but related to the target task; for each source dataset D m ∈{D1,D2,…,D M For each category c in each source dataset, calculate its median vector across the multi-source dataset. Where D1, D2, ..., D M Given M distinct source datasets, and:

[0025] For the source dataset D m The value of the j-th training sample on the p-th dimension feature. For the source dataset D m The set of indices of all training samples belonging to class c. For the source dataset D m The class label of the j-th training sample;

[0026] Step S213: Initialize the filtered source dataset For each category c, iterate through each source dataset D. m ,calculate With m c Euclidean distance between If the Euclidean distance is less than or equal to the threshold τ, then all source data samples belonging to category c in the source dataset are added.

[0027] Step S214: Combine the target dataset with the filtered source dataset Merge the datasets to obtain the first merged dataset. Train a non-oracle version of the transfer learning-based model on the first merged dataset to obtain M. initial LightGBM is used as the classifier during training, GOSS is used to improve training efficiency, and binary or multi-class objective function is automatically selected according to the number of classes.

[0028] Step S215: Use the target dataset to test M initial Fine-tuning and correction are performed to obtain a trained non-oracle version of the transfer learning-based model M. N .

[0029] Preferably, the Oracle version's training method for the transfer learning-based model includes:

[0030] Step S221: Merge the multi-source datasets with the target dataset to form the second merged dataset D. combined ;

[0031] Step S222: Train a transfer learning-based model based on the second merged dataset to obtain M1. initial During training, LightGBM is used as the base classifier, GOSS is used to improve training efficiency, and binary or multi-class objective function is automatically selected according to the number of classes.

[0032] Step S223: Use the target dataset to pair M1 initial Fine-tuning and correction are performed to obtain the trained transfer learning-based model M. fine-tuned .

[0033] Preferably, the model structure of the meta-learning-based model is the same as that of LightGBM, that is, LightGBM is used as the classifier;

[0034] Training methods for models based on meta-learning include:

[0035] Step S231: Use the source domain data as the training dataset, use the training dataset as the input dataset for the distance similarity extraction method, use the distance similarity extraction method to extract features from the training dataset, and use the extracted features as the first set of training feature vectors;

[0036] The distance similarity extraction method includes:

[0037] Multiple sample pairs are randomly constructed based on the input dataset. Each sample pair includes two positive samples A of the same type. k P k And a with A k P k N negative samples of different classes k Two pre-trained feature vectors X are constructed based on sample pairs. k1 X k2 and with X k1 X k2 The corresponding labels y k1 y k2 , and y k1 =1, y k2 =0; where:

[0038] X k1 =((A) k -P k ) T ,sim(A k ,P k ),euc(A k ,Pk ),man(A k ,P k ))

[0039] X k2 =((A) k -N k ) T ,sim(A k N k ),euc(A k N k ),man(A k N k ))sim(·), euc(·), and man(·) represent cosine similarity, Euclidean distance, and Manhattan distance, respectively;

[0040] Step S232: Train the LightGBM model based on the first set of training feature vectors to obtain M. pre ;

[0041] Step S233: Use the target domain support set data as the support dataset, use the support dataset as the input dataset for the distance similarity extraction method, use the distance similarity extraction method to extract features from the support dataset, and use the extracted features as the second set of training feature vectors;

[0042] Step S232: Train M based on the first set of training feature vectors pre M fine-tuned .

[0043] A second aspect of the present invention provides an automatic indexing device for statistical indicators based on small samples, the device comprising:

[0044] Selection Module: Configured to determine the features of the input data; the user selects an indexing model suitable for the input data based on the features of the input data; wherein, the input data is small sample data, and the indexing models include high-computing-power models and low-computing-power models, the high-computing-power models are large language models, and the low-computing-power models include a fusion model based on transfer learning and a model based on meta-learning, and a model based on meta-learning.

[0045] Indexing module: Configured to index input data based on the selected indexing model and obtain prediction results; the prediction results include the statistical indicators corresponding to the input data and the classification labels corresponding to each statistical indicator;

[0046] Optimization module: Configured to verify and optimize prediction results, and output the optimized prediction results.

[0047] A third aspect of the present invention provides an electronic device, the electronic device comprising:

[0048] At least one processor; and

[0049] A memory communicatively connected to the at least one processor; wherein,

[0050] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0051] A fourth aspect of the present invention provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method described above.

[0052] This invention enables efficient and accurate classification and indexing of statistical indicators even with scarce sample data, through methods such as transfer learning, meta-learning, and large language models. It maintains good model generalization performance even with limited sample data, demonstrating excellent versatility, scalability, and automation capabilities.

[0053] It can solve the following technical problems:

[0054] (1) The special nature of indicator data often leads to a scarcity of indicator samples. Traditional methods require a large number of labeled samples to train the model, but in some fields, it is difficult to obtain sufficient sample data due to confidentiality and other reasons.

[0055] (2) The index and label system in most fields will change with the development of the times, requiring the indexing model to support dynamic expansion function. However, existing methods often require retraining the entire model after the label system is updated, which is inefficient.

[0056] (3) How to maintain good model generalization performance under limited sample conditions and ensure the accuracy and reliability of indexing results;

[0057] (4) How to provide targeted solutions under different computing resource environments so that they can provide high-precision indexing in high computing power environments and basic indexing functions in low computing power environments.

[0058] Compared with the prior art, the present invention has the following advantages:

[0059] (1) This invention efficiently processes small sample data. By using transfer learning and meta-learning techniques, it effectively utilizes limited sample data for model training and prediction, and can still maintain good model generalization performance even when samples are scarce.

[0060] (2) This invention supports dynamic label expansion. Through meta-learning algorithms and model fusion strategies, it can flexibly respond to dynamic changes in the label system and adapt to the new label system without retraining the entire model.

[0061] (3) This invention has strong multi-mode adaptability. It provides two solutions: a high-computing-power mode and a low-computing-power mode. The appropriate algorithm can be selected according to the actual computing resources, which has good adaptability.

[0062] (4) This invention has high-precision indexing capabilities. In high-computing-power mode, high-precision indexing can be achieved through large language models and prompt word engineering technology; in low-computing-power mode, the accuracy of indexing results can also be guaranteed through the fusion of the LightGBM model and multiple algorithms.

[0063] (5) This invention is highly robust. Through multi-level fault-tolerant parsing, verification mechanisms, and anomaly handling, it ensures that the system can still operate stably and output reasonable results under various abnormal conditions.

[0064] (6) This invention has high security. By replacing the original data transmission with parameter migration, it ensures the physical isolation of sensitive data and meets the security requirements of special fields.

[0065] (7) This invention offers flexible deployment options. It supports encapsulating the fine-tuned model into an independent application module to meet the differentiated needs of different application scenarios. Attached Figure Description

[0066] Figure 1 This is a flowchart illustrating the automatic indexing method for statistical indicators based on small samples provided by the present invention.

[0067] Figure 2 This is a schematic diagram of the automatic indexing method for statistical indicators based on small samples provided by the present invention;

[0068] Figure 3 A schematic diagram of the large language model indexing method provided by the present invention;

[0069] Figure 4 This is a schematic diagram of a model based on multi-source data provided by the present invention;

[0070] Figure 5 A schematic diagram of the transfer learning-based model provided by this invention;

[0071] Figure 6 This is a schematic diagram of the meta-learning-based model provided by the present invention. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0073] like Figures 1-2 As shown, an automatic indexing method for statistical indicators based on small samples provided by the present invention includes:

[0074] Step S1: Determine the features of the input data; the user selects an indexing model suitable for the input data based on the features of the input data; wherein, the input data is small sample data, and the indexing model includes high-computing-power models and low-computing-power models. The high-computing-power model is a large language model, and the low-computing-power models include a fusion model based on transfer learning and a model based on meta-learning, and a model based on meta-learning.

[0075] Step S2: Index the input data based on the selected indexing model to obtain the prediction results; the prediction results include the statistical indicators corresponding to the input data and the classification labels corresponding to each statistical indicator;

[0076] Step S3: Verify and optimize the prediction results, and output the optimized prediction results.

[0077] Furthermore, such as Figure 3 As shown, when the selected indexing model is a high-computing-power model, the step of obtaining prediction results based on the indexing input data of the selected indexing model includes:

[0078] Step S201: Construct structured prompt words, input the structured prompt words into the trained large language model, and use the output of the large language model as the preliminary prediction result;

[0079] Step S202: Analyze the preliminary prediction results and construct a label dictionary; the label dictionary includes multiple mapping pairs, each mapping pair includes a statistical indicator and the corresponding classification label of the statistical indicator;

[0080] Step S203: Divide all mapping pairs into multiple sets, with each set representing a batch of data; the thread pool executor verifies each batch of data. If the verification result is successful, save the batch of data to the output set; if the verification result is unsuccessful, input the corresponding input data of the batch of data into the trained large model and regenerate the prediction result until the verification result of the mapping pair corresponding to the input data of the batch of data is successful.

[0081] The verification content includes: the amount of data in this batch, the degree of matching between this batch of data and the input data, and the consistency between the classification labels corresponding to this batch of data and the labels in the preset label pool;

[0082] Step S204: Use the output set as the prediction result of the input data.

[0083] In this invention, step S201: constructing structured prompt words, inputting the structured prompt words into the trained large language model, and using the output of the large language model as the preliminary prediction result, includes:

[0084] Step S2011: Construct structured prompt words using prompt word engineering techniques such as zero-sample prompts, few-sample prompts, and context enhancement;

[0085] Step S2012: Input the pre-trained dataset and unlabeled samples into the large language model in a structured form, and obtain the trained large language model after training;

[0086] Step S2013: Input the structured prompt words into the trained large language model, that is, call the trained large language model to generate preliminary prediction results. The preliminary prediction results are in JSON format.

[0087] Step S202: Parse the preliminary prediction results and construct a label dictionary; the label dictionary includes multiple mapping pairs, each mapping pair including a statistical indicator and the corresponding classification label, including:

[0088] Step S2021: Set up an exception handling mechanism to record JSON decoding errors;

[0089] If the initial character of the preliminary prediction result is detected, and it starts with "[", then the response content of the large language model is parsed into a JSON array, the "node" and "prediction label" fields are extracted and mapped to "indicator" and "classification label"; if it starts with "{", then the response content of the large language model is parsed into a JSON object, and the JSON object is checked to see if it contains the "results" key. If it exists, its content is returned.

[0090] Step S2022: If the preliminary prediction result contains JSON tags, extract the JSON string in the code block for secondary parsing and add field existence verification;

[0091] If JSON parsing fails, a text line scanning mode is started, matching consecutive lines starting with "index:" and "tag:" line by line, and extracting the content after the colon to build structured data;

[0092] If the parsing result is empty, output a warning;

[0093] Step S2023: The extracted content is mapped into mapping pairs. Each mapping pair includes a statistical indicator and the corresponding category label. All mapping pairs form a label dictionary.

[0094] Step S203: Divide all mapping pairs into multiple sets, with each set as a batch of data; the thread pool executor verifies the data of each batch, and when the verification result is passed, save the data of that batch to the output set; when the verification result is failed, input the input data corresponding to the data of that batch into the trained large model, regenerate the prediction result, until the verification result of the mapping pair corresponding to the input data of that batch is passed;

[0095] The verification content includes: the data volume of the batch, the degree of matching between the batch of data and the input data, and the consistency between the classification labels corresponding to the batch of data and the labels in the preset label pool, including:

[0096] Step S2031: A hierarchical batch processing architecture enables efficient parallel prediction;

[0097] Step S2032: Load the label definition file and create the results storage directory;

[0098] Step S2033: Divide the data to be predicted into multiple batches according to the preset batch processing size;

[0099] Step S2034: Process each batch of data in parallel using a thread pool executor;

[0100] Step S2035: Call the core prediction interface to obtain the model output for each batch and perform triple verification: verify that the returned result is a list type, verify that the number of verification results strictly matches the input samples, and check whether the predicted labels are within the range of the predefined label pool;

[0101] Step S2036: For batches that fail verification, automatically trigger the retry process, with a preset interval between each retry;

[0102] Step S2037: Aggregate the valid results from each batch into a unified list and return it;

[0103] Step S2038: Establish an exception monitoring system to capture and record thread execution errors and data verification exceptions in real time.

[0104] Furthermore, it also includes the verification and correction of the reasonableness of the prediction results, including:

[0105] Perform a validity check on the JSON results output by the model to ensure that the predicted labels are within the range of the predefined label pool;

[0106] If unreasonable results are found, they will be automatically re-entered into the model for correction and generation to improve the accuracy of the results.

[0107] This invention provides a specific embodiment of obtaining prediction results based on the indexing input data of the selected indexing model when the selected indexing model is a high-computing-power model.

[0108] First, prompt words are constructed and a large language model is invoked for prediction. Prompt word engineering techniques such as zero-shot prompts, few-shot prompts, and context enhancement are used to construct structured prompt words. The pre-trained dataset and unlabeled samples are input into the large language model in a structured format, and the model generates the corresponding JSON format output.

[0109] The main content of the prompt is as follows:

[0110] You are a professional statistics expert. Based on the provided label pool, label descriptions, and sample data, please predict the labels for the following categorized indicators.

[0111] **Input Information:**

[0112] 1. **Tag Pool:** (Tags in this pool must be strictly used; if the tag pool is "None", it will be ignored.)

[0113] `{label_pool}`

[0114] 2. **Tag Description:** (If applicable, please strictly adhere to their definitions for categorization.)

[0115] `{label_definition}`

[0116] 3. **Support Dataset Example:** (Includes an example of the correspondence between metrics and labels)

[0117] `{support_str}`

[0118] 4. **Indicators to be classified:** (Total {len(unknown_data)} items)

[0119] `{unknown_str}`

[0120] **Classification Rules (Please strictly adhere to):**

[0121] 1. **Core Logic:**

[0122] ***If the tag pool is empty or "none":** Do not refer to the tag pool.

[0123] ***The label pool contains only one label:** The label is assigned only if the sample matches the label definition; otherwise, the predicted label is null (`""`).

[0124] ***The label pool contains multiple labels:** Predict one or more labels for each sample (all labels).

[0125] 2. **Decision Prioritization (Applied in Order):**

[0126] ***Rule 1: If a label description exists, the classification must be based on the definition in the label description first and strictly.

[0127] ***Rule 2:** Based on Rule 1, if the **support dataset example is not empty**, you need to refer to the correspondence between the metrics and labels shown in the example.

[0128] ***Rule 3:** If the label description does not exist, then the classification must be based on the example in the support dataset.

[0129] ***Rule 4: If neither the tag description nor the support dataset example exists, the tag will be automatically categorized according to the tag system defined by the tag pool.

[0130] ***Rule 5:** If the tag pool, tag description, and support dataset examples do not exist, then classify them according to the specific tag system.

[0131] 3. **Other requirements:**

[0132] *All predictions must have a clear logical basis (based on rules 1-4).

[0133] * The output format must be strictly a JSON list, with each element containing a "node" (metric name) and a "predicted label" (the predicted label, or an empty string "").

[0134] ***No other content may be output** (such as classification reasons, classification logic, additional explanations, etc.). Only the required JSON list should be output.

[0135] **Output format example:**

[0136]

[0137]

[0138] Then, multi-level fault-tolerant parsing is implemented on the model response. The response text is checked for its starting character. If it begins with "[", it is directly parsed into a JSON array, and the "node" and "predicted label" fields of each dictionary element are extracted and mapped to "indicator" and "label". If it begins with "{", it is parsed into a JSON object, and the presence of the "results" key is checked; if it exists, its content is returned. If the response text contains "JSON tags", the JSON string within the code block is automatically extracted for secondary parsing, and field existence checks are added. If JSON parsing fails, a text line scanning mode is initiated, matching consecutive lines starting with "indicator:" and "label:", extracting the content after the colon to construct structured data. An exception handling mechanism is set up for all parsing processes, recording JSON decoding errors and the original response content, and outputting warnings for empty results. Finally, a standardized dictionary list is returned, with the structure [{"indicator":value,"label":value}.

[0139] Next, a hierarchical batch processing architecture is adopted to achieve efficient parallel prediction. First, the label definition file is loaded and a results storage directory is created. Then, the data to be predicted is divided into multiple batches according to the preset batch size (default 15 samples / batch). Then, a thread pool executor (maximum concurrency 50) processes each batch of data in parallel. During the execution of each batch, the core prediction interface is called to obtain the model output, and a triple verification mechanism is implemented—verifying that the returned result is a list type, verifying that the number of results strictly matches the input samples, and checking whether the predicted labels are within the predefined label pool range. For batches that fail verification, the system automatically triggers a retry process (default maximum 3 retries), with a 1-second interval between each retry to ensure service stability. Finally, the valid results from each batch are aggregated into a unified list and returned. Simultaneously, the system establishes an anomaly monitoring system to capture and record thread execution errors and data verification anomalies in real time, ensuring that the batch prediction process is both efficient and reliable.

[0140] Finally, the prediction results are validated and corrected for reasonableness. The JSON results output by the model are validated to ensure that the predicted labels are within the predefined label pool. If unreasonable results are found during the verification, the system will automatically re-enter them into the model for correction and generation to improve the accuracy of the final results.

[0141] The fusion model of transfer learning-based models and meta-learning-based models includes parallel transfer learning-based models and meta-learning-based models; the transfer learning-based models and meta-learning-based models index input data through an equal-weighted voting mechanism.

[0142] Furthermore, such as Figure 4As shown, the model structure of the transfer learning-based model is the same as that of LightGBM, that is, LightGBM is used as the classifier, and the performance of the transfer learning-based model on the target task is optimized through a source domain selection strategy. The transfer learning-based model is divided into a non-oracle version and an oracle version. The source domain selection strategy of the non-oracle version is as follows: calculate the median vector of each category of data in the target domain, and select source domain data based on the distance between the source domain data and the median vectors of each category in the target domain, retaining source domain data whose similarity to the target domain exceeds a first preset threshold. The source domain selection strategy of the oracle version is a source domain data augmentation strategy, that is, to augment the source domain data.

[0143] In this invention, both the non-oracle version and the oracle version of the transfer learning model employ a two-stage training strategy. First, the model is pre-trained on processed source domain data, and then fine-tuned on target domain data with a lower learning rate, thereby achieving effective knowledge transfer from the source domain to the target domain. The model can also dynamically adapt to binary and multi-class classification tasks, automatically selecting a matching loss function based on the number of classes, and using GOSS (Gradient-based One-Side Sampling) as a boosting strategy to improve training efficiency.

[0144] The training methods for non-oracle versions of transfer learning-based models include:

[0145] Step S211: Use the training sample set as the target dataset; calculate the class median vector of the target dataset, and apply it to the target dataset. For each category c∈{1,2,…,C}, calculate the median of all its samples on each feature p∈{1,2,…,P}, forming the median vector m of that category in the target dataset. c =(m c (1),m c (2),…,m c (P)), where X i Let y be the semantic vector of the i-th training sample. i Let be the class label of the i-th training sample, i be the index of the training sample, n be the total number of training samples in the target dataset, C be the total number of classes, P be the total number of feature dimensions, and m be the total number of features. c (1),m c (2),…,m c (P) represents the median of category c in the 1st, 2nd, ..., Pth dimensions of the features;

[0146]

[0147] X ip Let I be the value of the i-th training sample on the p-th feature, and median be the median function. c This is the set of indices of all training samples belonging to category c;

[0148] Step S212: Extract the class median vector of each source dataset in the multi-source dataset. The source dataset refers to the training dataset from different domains or tasks but related to the target task; for each source dataset D m ∈{D1,D2,…,D M For each category c in each source dataset, calculate its median vector across the multi-source dataset. Where D1, D2, ..., D M Given M distinct source datasets, and:

[0149] For the source dataset D m The value of the j-th training sample on the p-th dimension feature. For the source dataset D m The set of indices of all training samples belonging to class c. For the source dataset D m The class label of the j-th training sample;

[0150] Step S213: Initialize the filtered source dataset For each category c, iterate through each source dataset D. m ,calculate With m c Euclidean distance between If the Euclidean distance is less than or equal to the threshold τ, then all source data samples belonging to category c in the source dataset are added.

[0151] Step S214: Combine the target dataset with the filtered source dataset Merge the datasets to obtain the first merged dataset. Train a non-oracle version of the transfer learning-based model on the first merged dataset to obtain M. initial LightGBM is used as the classifier during training, GOSS is used to improve training efficiency, and binary or multi-class objective function is automatically selected according to the number of classes.

[0152] Step S215: Use the target dataset to test M initial Fine-tuning and correction are performed to obtain a trained non-oracle version of the transfer learning-based model M. N .

[0153] Input data into M N This yields the prediction results for the input data.

[0154] Oracle versions of training methods for models based on transfer learning include:

[0155] Step S221: Merge the multi-source datasets with the target dataset to form the second merged dataset D. combined ;

[0156] Step S222: Train a transfer learning-based model based on the second merged dataset to obtain M1. initial During training, LightGBM is used as the base classifier, GOSS is used to improve training efficiency, and binary or multi-class objective function is automatically selected according to the number of classes.

[0157] Step S223: Use the target dataset to pair M1 initial Fine-tuning and correction are performed to obtain the trained transfer learning-based model M. fine-tuned .

[0158] Input data into M fine-tuned This yields the prediction results for the input data.

[0159] Furthermore, such as Figure 5 As shown, the model structure of the meta-learning-based model is the same as that of LightGBM, using LightGBM as the classifier and constructing sample pair features through distance similarity extraction to learn similarity patterns between samples. The meta-learning-based model randomly constructs triplet samples containing positive samples of the same class and negative samples of different classes, obtaining cosine similarity, Euclidean distance, and Manhattan distance between sample pairs as similarity features. LightGBM is then used to train these similarity features to learn similarity patterns between samples. The meta-learning-based model employs a two-stage training strategy: pre-training on source domain data and then fine-tuning on the target domain support set, thereby achieving rapid adaptation to new tasks.

[0160] Training methods for models based on meta-learning include:

[0161] Step S231: Use the source domain data as the training dataset, use the training dataset as the input dataset for the distance similarity extraction method, use the distance similarity extraction method to extract features from the training dataset, and use the extracted features as the first set of training feature vectors;

[0162] The distance similarity extraction method includes:

[0163] Multiple sample pairs are randomly constructed based on the input dataset. Each sample pair includes two positive samples A of the same type. k P k And a with A k P k N negative samples of different classes k Two pre-trained feature vectors X are constructed based on sample pairs. k1 X k2 and with X k1 X k2 The corresponding labels y k1 y k2 , and y k1 =1, y k2 =0; where:

[0164] X k1 =((A) k -P k ) T ,sim(A k ,P k ),euc(A k ,P k ),man(A k ,P k ))

[0165] X k2 =((A) k -N k ) T ,sim(A k N k ),euc(A k N k ),man(A k N k ))sim(·), euc(·), and man(·) represent cosine similarity, Euclidean distance, and Manhattan distance, respectively;

[0166] Step S232: Train the LightGBM model based on the first set of training feature vectors to obtain M. pre ;

[0167] Step S233: Use the target domain support set data as the support dataset, use the support dataset as the input dataset for the distance similarity extraction method, use the distance similarity extraction method to extract features from the support dataset, and use the extracted features as the second set of training feature vectors;

[0168] Step S232: Train M based on the first set of training feature vectors pre M fine-tuned .

[0169] Furthermore, using Mfine-tuned Index the input data to obtain the prediction results, including:

[0170] Step S233: Record the input data as X test Calculate the difference vector between it and all data in the supporting dataset, and input each difference vector into M. fine-tuned The result obtained is used as the prediction result.

[0171] Users determine the labeling system corresponding to the input data based on the characteristics of the input data. When the labeling system corresponding to the input data is consistent with the labeling system of the dataset used to train the transfer learning-based model and the meta-learning-based model, a fusion model is used to vote to obtain the indexing result; otherwise, the meta-learning-based model is used to obtain the indexing result.

[0172] like Figure 6 As shown, this invention also provides a fusion model based on transfer learning and meta-learning. This fusion model employs a dual-path training structure, including a transfer learning path and a meta-learning path. The transfer learning path utilizes LightGBM to achieve knowledge transfer through source domain selection and data augmentation techniques; the meta-learning path constructs sample pair features using distance similarity extraction methods, improving the model's adaptability to new tasks. The outputs of the two paths are integrated through a weighted fusion mechanism, dynamically adjusting the weights based on task similarity. The model employs a multi-stage training strategy, constructing training datasets for both paths and training them in parallel. Then, the two models are fine-tuned using target domain data, and finally, weights are allocated based on the validation set to obtain the final fusion model.

[0173] The specific implementation method is as follows:

[0174] Obtain the pre-trained dataset and the fine-tuning dataset;

[0175] When the labels corresponding to each training sample in the pre-training dataset are different from the labels corresponding to each data in the fine-tuning dataset, the meta-learning-based model is invoked.

[0176] When the labels corresponding to each training sample in the pre-training dataset are the same as the labels corresponding to each data in the fine-tuning dataset, the fusion model is invoked, that is, the transfer learning-based model and the meta-learning-based model are invoked in parallel; among them, the transfer learning-based model and the meta-learning-based model make labeling decisions on the indicators in the input data through an equal-weight voting mechanism.

[0177] The apparatus provided for carrying out the present invention will be described below. The specific implementation process and technical effects are as described above and will not be repeated below.

[0178] Optionally, embodiments of the present invention provide an automatic indexing device for statistical indicators based on small samples, the device comprising:

[0179] Selection Module: Configured to determine the features of the input data; the user selects an indexing model suitable for the input data based on the features of the input data; wherein, the input data is small sample data, and the indexing models include high-computing-power models and low-computing-power models, the high-computing-power models are large language models, and the low-computing-power models include a fusion model based on transfer learning and a model based on meta-learning, and a model based on meta-learning.

[0180] Indexing module: Configured to index input data based on the selected indexing model and obtain prediction results; the prediction results include the statistical indicators corresponding to the input data and the classification labels corresponding to each statistical indicator;

[0181] Optimization module: Configured to verify and optimize prediction results, and output the optimized prediction results.

[0182] The above-described device is used to execute the method provided in the foregoing embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0183] These modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more digital signal processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a system-on-a-chip (SOC).

[0184] The modules described above can be connected or communicate with each other via wired or wireless connections. Wired connections may include metal cables, optical fibers, hybrid cables, or any combination thereof. Wireless connections may include connections via LAN, WAN, Bluetooth, ZigBee, or NFC, or any combination thereof. Two or more modules can be combined into a single module, and any module can be divided into two or more units. Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here.

[0185] It should be noted that these modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). Furthermore, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Additionally, these modules can be integrated together to form a System-on-a-Chip (SOC).

[0186] The electronic device includes a processor, memory, communication interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, Near Field Communication (NFC), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0187] The present invention also provides a program product, such as a computer-readable storage medium, including a program that, when executed by a processor, is used to perform the above-described method embodiments.

[0188] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0189] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0190] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0191] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for automatic indexing of statistical indicators based on small samples, characterized in that, include: Step S1: Determine the features of the input data; the user selects an indexing model suitable for the input data based on the features of the input data; wherein, the input data is small sample data, and the indexing model includes high-computing-power models and low-computing-power models. The high-computing-power model is a large language model, and the low-computing-power models include a fusion model based on transfer learning and a model based on meta-learning, and a model based on meta-learning. Step S2: Index the input data based on the selected indexing model to obtain the prediction results; the prediction results include the statistical indicators corresponding to the input data and the classification labels corresponding to each statistical indicator; Step S3: Verify and optimize the prediction results, and output the optimized prediction results.

2. The method as described in claim 1, characterized in that, When the selected indexing model is a high-computing-power model, the step of obtaining prediction results based on the indexing input data of the selected indexing model includes: Step S201: Construct structured prompt words, input the structured prompt words into the trained large language model, and use the output of the large language model as the preliminary prediction result; Step S202: Analyze the preliminary prediction results and construct a label dictionary; the label dictionary includes multiple mapping pairs, each mapping pair includes a statistical indicator and the corresponding classification label of the statistical indicator; Step S203: Divide all mapping pairs into multiple sets, with each set representing a batch of data; the thread pool executor verifies each batch of data. If the verification result is successful, save the batch of data to the output set; if the verification result is unsuccessful, input the corresponding input data of the batch of data into the trained large model and regenerate the prediction result until the verification result of the mapping pair corresponding to the input data of the batch of data is successful. The verification content includes: the amount of data in this batch, the degree of matching between this batch of data and the input data, and the consistency between the classification labels corresponding to this batch of data and the labels in the preset label pool; Step S204: Use the output set as the prediction result of the input data.

3. The method as described in claim 1, characterized in that, The fusion model of transfer learning-based models and meta-learning-based models includes parallel transfer learning-based models and meta-learning-based models; the transfer learning-based models and meta-learning-based models index input data through an equal-weighted voting mechanism.

4. The method as described in claim 3, characterized in that, The model structure of the transfer learning-based model is the same as that of LightGBM, i.e., LightGBM is used as the classifier. The transfer learning-based model is divided into a non-oracle version and an oracle version. The source domain selection strategy of the non-oracle version is to calculate the median vector of each category of data in the target domain, and select the source domain data based on the distance between the source domain data and the median vector of each category of the target domain, so as to retain the source domain data whose similarity to the target domain exceeds a first preset threshold. The source domain selection strategy of the oracle version is a source domain data augmentation strategy, i.e., to augment the source domain data.

5. The method as described in claim 4, characterized in that, The training methods for non-oracle versions of transfer learning-based models include: Step S211: Use the training sample set as the target dataset; calculate the class median vector of the target dataset, and apply it to the target dataset. For each category c∈{1,2,…,C}, calculate the median of all its samples on each feature p∈{1,2,…,P}, forming the median vector m of that category in the target dataset. c =(m c (1),m c (2),…,m c (P)), where X i Let y be the semantic vector of the i-th training sample. i Let be the class label of the i-th training sample, i be the index of the training sample, n be the total number of training samples in the target dataset, C be the total number of classes, P be the total number of feature dimensions, and m be the total number of features. c (1),m c (2),…,m c (P) represents the median of category c in the 1st, 2nd, ..., Pth dimensions of the features; X ip Let I be the value of the i-th training sample on the p-th feature, and median be the median function. c This is the set of indices of all training samples belonging to category c; Step S212: Extract the class median vector of each source dataset in the multi-source dataset. The source dataset refers to the training dataset from different domains or tasks but related to the target task; for each source dataset D m ∈{D1,D2,…,D M For each category c in each source dataset, calculate its median vector across the multi-source dataset. Where D1, D2, ..., D M Given M distinct source datasets, and: For the source dataset D m The value of the j-th training sample on the p-th dimension feature. For the source dataset D m The set of indices of all training samples belonging to class c. For the source dataset D m The class label of the j-th training sample; Step S213: Initialize the filtered source dataset For each category c, iterate through each source dataset D. m ,calculate With m c Euclidean distance between If the Euclidean distance is less than or equal to the threshold τ, then all source data samples belonging to category c in the source dataset are added. Step S214: Combine the target dataset with the filtered source dataset Merge the datasets to obtain the first merged dataset. Train a non-oracle version of the transfer learning-based model on the first merged dataset to obtain M. initial LightGBM is used as the classifier during training, GOSS is used to improve training efficiency, and binary or multi-class objective function is automatically selected according to the number of classes. Step S215: Use the target dataset to test M initial Fine-tuning and correction are performed to obtain a trained non-oracle version of the transfer learning-based model M. N .

6. The method as described in claim 4, characterized in that, Oracle versions of training methods for models based on transfer learning include: Step S221: Merge the multi-source datasets with the target dataset to form the second merged dataset D. combined ; Step S222: Train a transfer learning-based model based on the second merged dataset to obtain M1. initial During training, LightGBM is used as the base classifier, GOSS is used to improve training efficiency, and binary or multi-class objective function is automatically selected according to the number of classes. Step S223: Use the target dataset to pair M1 initial Fine-tuning and correction are performed to obtain the trained transfer learning-based model M. fine-tuned .

7. The method as described in claim 6, characterized in that, The model structure of the meta-learning-based model is the same as that of LightGBM, that is, LightGBM is used as the classifier. Training methods for models based on meta-learning include: Step S231: Use the source domain data as the training dataset, use the training dataset as the input dataset for the distance similarity extraction method, use the distance similarity extraction method to extract features from the training dataset, and use the extracted features as the first set of training feature vectors; The distance similarity extraction method includes: Multiple sample pairs are randomly constructed based on the input dataset. Each sample pair includes two positive samples A of the same type. k P k And a with A k P k N negative samples of different classes k Two pre-trained feature vectors X are constructed based on sample pairs. k1 X k2 and with X k1 X k2 The corresponding labels y k1 y k2 , and y k1 =1, y k2 =0; where: X k1 =((A k -P k ) T ,sim(A k ,P k ),euc(A k ,P k ),man(A k ,P k )) X k2 =((A) k -N k ) T ,sim(A k N k ),euc(A k N k ),man(A k N k ))sim(·), euc(·), and man(·) represent cosine similarity, Euclidean distance, and Manhattan distance, respectively; Step S232: Train the LightGBM model based on the first set of training feature vectors to obtain M pre ; Step S233: Use the target domain support set data as the support dataset, use the support dataset as the input dataset for the distance similarity extraction method, use the distance similarity extraction method to extract features from the support dataset, and use the extracted features as the second set of training feature vectors; Step S232: Train M based on the first set of training feature vectors pre M fine-tuned .

8. An automatic indexing device for statistical indicators based on small samples, characterized in that, include: Selection Module: Configured to determine the features of the input data; the user selects an indexing model suitable for the input data based on the features of the input data; wherein, the input data is small sample data, and the indexing models include high-computing-power models and low-computing-power models, the high-computing-power models are large language models, and the low-computing-power models include a fusion model based on transfer learning and a model based on meta-learning, and a model based on meta-learning. Indexing module: Configured to index input data based on the selected indexing model and obtain prediction results; the prediction results include the statistical indicators corresponding to the input data and the classification labels corresponding to each statistical indicator; Optimization module: Configured to verify and optimize prediction results, and output the optimized prediction results.

9. An electronic device, characterized in that, The device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method as described in any one of claims 1-7.

Images