Data processing method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a data processing method and device, electronic equipment, a storage medium and a program product. The data processing method includes: determining a plurality of mapping sub-data sets according to an original data set for training and / or fine tuning of a large language model (LLM); based on a mapping network and the plurality of mapping sub-datasets, determining a weight corresponding to each quality index for the original dataset, the mapping network being used for predicting a model precision corresponding to one mapping sub-dataset based on the quality index of the data in the mapping sub-dataset; a first data set for training and / or trimming of the LLM is determined from the original data set based on the determined weights.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision, and in particular to data filtering in the field of computer vision. Specifically, it relates to a data processing method and apparatus, electronic devices, storage media, and program products. Background Technology

[0002] With the rapid development of Artificial Intelligence (AI) technology, Large Language Models (LLMs) have achieved remarkable success in various fields such as Natural Language Processing (NLP), text and image generation, and multimodal applications. The success of LLMs relies not only on their complex model structure but, more importantly, on the high-quality, large-scale datasets they depend on. Data plays a fundamental role in both the pre-training and fine-tuning stages.

[0003] While LLM has achieved success in general domains, its performance in specific fields (such as semiconductors, physics, and biomedicine) has been less effective. This is due to the differences between domain-specific and general domain data. To enable LLM to better perform domain-specific tasks, data management (curation), fine-tuning, and reinforcement learning are typically required. However, in practice, the sources of data used for training or fine-tuning are complex and of varying quality, resulting in noisy or low-quality data that directly impacts LLM performance in domain-specific tasks. Therefore, effective data management (especially developing suitable datasets for training or fine-tuning) is crucial for enhancing LLM performance in specific domains and improving training efficiency. Summary of the Invention

[0004] To at least address the aforementioned problems and / or drawbacks, embodiments of this disclosure provide a data processing method and apparatus, an electronic device, a storage medium, and a program product.

[0005] According to a first aspect of the embodiments of this disclosure, a data processing method is provided, comprising: determining a plurality of mapping subsets based on an original dataset for training and / or fine-tuning a large language model (LLM); determining weights for the original dataset corresponding to each quality metric based on a mapping network and the plurality of mapping subsets, wherein the mapping network is used to predict model accuracy corresponding to a mapping subset based on a quality metric of data in a mapping subset; and determining a first dataset for training and / or fine-tuning the LLM from the original dataset based on the determined weights.

[0006] Optionally, the mapping network is determined based on multiple training subsets of the original dataset, wherein the multiple training subsets include: at least one subset having data randomly selected from the original dataset, and at least one subset having data selected based on at least one randomly determined quality metric.

[0007] Optionally, the mapping network is determined based on the plurality of training subsets by: determining the metric score for each quality metric for each data point in each training subset using a reward model; training and / or fine-tuning the LLM based on each training subset; testing the model accuracy of the LLM trained and / or fine-tuned for each training subset using a test set; and determining the mapping network that maps the metric scores to the model accuracy based on the determined metric scores and the tested model accuracy of the plurality of training subsets.

[0008] Optionally, the step of determining the weights corresponding to each quality metric for the original dataset based on the mapping network and the plurality of mapping subsets includes: randomly setting a metric score for each quality metric for each of the plurality of mapping subsets; determining the model accuracy of the LLM trained and / or fine-tuned based on the set metric scores for each mapping subset using the mapping network; selecting a subset of mapping subsets from the plurality of mapping subsets based on the plurality of model accuracies corresponding to the plurality of mapping subsets respectively; and determining the weights for each quality metric for the original dataset based on the statistical values ​​of various quality metrics of the selected subset of mapping subsets.

[0009] Optionally, the step of determining a first dataset for training and / or fine-tuning the LLM from the original dataset based on determined weights includes: determining the score of each quality metric of the data in the original dataset by using a reward model; determining the quality score of the data in the original dataset by weighting the score of each quality metric of the data based on the weight corresponding to each quality metric; and selecting a predetermined number of first data points from the original dataset to constitute the first dataset based on the quality scores of the data in the original dataset.

[0010] Optionally, the data processing method further includes: selecting second data from the first dataset to form a second dataset for training and / or fine-tuning the LLM by performing data filtering based on retrieval augmentation generation (RAG).

[0011] Optionally, the step of selecting second data from the first dataset to form a second dataset for training and / or fine-tuning the LLM by performing data filtering based on retrieval augmentation generation (RAG) includes: repeatedly performing RAG-based data filtering until the number of data in the knowledge base or the number of second data in the second dataset equals a predetermined value and / or all first data in the first dataset have been subjected to RAG-based data filtering, wherein the step of performing data filtering based on retrieval augmentation generation (RAG) includes: selecting multiple third data from the first dataset to form a third dataset; and performing the following steps for each third data in the third dataset: retrieving multiple fourth data related to the third data from the knowledge base using RAG; determining whether the knowledge base includes knowledge of the third data based on the multiple fourth data and the third data using the LLM; and adding the third data to the knowledge base and identifying it as second data based on the knowledge that the knowledge base does not include the third data.

[0012] Optionally, the step of selecting multiple third data points from the first dataset to form the third dataset includes: selecting a predetermined number of third data points with the largest Euclidean distance from the first dataset using a K-centroid greedy algorithm.

[0013] Optionally, the step of determining whether the knowledge base includes knowledge of the third data based on the plurality of fourth data and the third data using the LLM includes: adding the plurality of fourth data related to the third data to prompts for input into the LLM; obtaining the confusion level of the third data with prompts and the confusion level without prompts using the LLM; determining the ratio of the confusion level of the third data with prompts to the confusion level without prompts as the knowledge retrieval score of the third data; and determining whether the knowledge base includes knowledge of the third data based on a comparison result of the knowledge retrieval score of the third data and a threshold.

[0014] According to one aspect of an embodiment of the present disclosure, a data processing apparatus is provided, comprising: a subset determination unit configured to determine a plurality of mapping subsets based on an original dataset for training and / or fine-tuning a Large Language Model (LLM); a weight determination unit configured to determine weights for the original dataset corresponding to each quality metric based on a mapping network and the plurality of mapping subsets, wherein the mapping network is used to predict model accuracy corresponding to a mapping subset based on a quality metric of data in a mapping subset; and a first dataset determination unit configured to determine a first dataset for training and / or fine-tuning the LLM from the original dataset based on the determined weights.

[0015] Optionally, the mapping network is determined based on multiple training subsets of the original dataset, wherein the multiple training subsets include: at least one subset having data randomly selected from the original dataset, and at least one subset having data selected based on at least one randomly determined quality metric.

[0016] Optionally, the mapping network is determined based on the plurality of training subsets by: determining the metric score for each quality metric for each data point in each training subset using a reward model; training and / or fine-tuning the LLM based on each training subset; testing the model accuracy of the LLM trained and / or fine-tuned for each training subset using a test set; and determining the mapping network that maps the metric scores to the model accuracy based on the determined metric scores and the tested model accuracy of the plurality of training subsets.

[0017] Optionally, the weight determination unit is configured to determine weights for each quality metric in the original dataset based on the mapping network and the plurality of mapping subsets by: randomly setting a metric score for each quality metric for each of the plurality of mapping subsets; determining the model accuracy of the LLM trained and / or fine-tuned based on the set metric score for each mapping subset using the mapping network; selecting a subset of mapping subsets from the plurality of mapping subsets based on multiple model accuracies corresponding to the plurality of mapping subsets respectively; and determining weights for each quality metric in the original dataset based on the statistical values ​​of various quality metrics in the selected subset of mapping subsets.

[0018] Optionally, the first dataset determining unit is configured to determine a first dataset for training and / or fine-tuning the LLM from the original dataset based on determined weights by: determining the index score of each quality metric of the data in the original dataset using a reward model; determining the quality score of the data in the original dataset by weighting the index scores of each quality metric of the data based on the weights corresponding to each quality metric; and selecting a predetermined number of first data points from the original dataset to constitute the first dataset based on the quality scores of the data in the original dataset.

[0019] Optionally, the data processing apparatus further includes a second dataset determination unit, wherein the second dataset determination unit is configured to select second data from the first dataset to form a second dataset for training and / or fine-tuning of the LLM by performing data filtering based on retrieval augmentation generation (RAG).

[0020] Optionally, the second dataset determination unit is configured to select second data from the first dataset to form a second dataset for training and / or fine-tuning of the LLM by performing data filtering based on retrieval augmentation generation (RAG) through the following operations: repeatedly performing RAG-based data filtering until the number of data in the knowledge base or the number of second data in the second dataset equals a predetermined value and / or all first data in the first dataset have been subjected to RAG-based data filtering, wherein the step of performing data filtering based on retrieval augmentation generation (RAG) includes: selecting a plurality of third data from the first dataset to form a third dataset; performing the following steps for each third data in the third dataset: retrieving a plurality of fourth data related to the third data from the knowledge base using RAG; determining whether the knowledge base includes knowledge of the third data based on the plurality of fourth data and the third data using the LLM; and adding the third data to the knowledge base and determining it as second data based on the knowledge that the knowledge base does not include the third data.

[0021] Optionally, the second dataset determination unit is configured to select a number of third data points from the first dataset to form the third dataset by using a K-centroid greedy algorithm to select a predetermined number of third data points with the largest Euclidean distance from the first dataset.

[0022] Optionally, the second dataset determination unit is configured to determine whether the knowledge base includes knowledge of the third data based on the plurality of fourth data and the third data by using the LLM: adding the plurality of fourth data related to the third data to prompts for input into the LLM; obtaining the confusion level of the third data with and without prompts by using the LLM; determining the ratio of the confusion level of the third data with and without prompts as the knowledge retrieval score of the third data; and determining whether the knowledge base includes knowledge of the third data based on a comparison result of the knowledge retrieval score of the third data and a threshold.

[0023] According to one aspect of the embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, cause the at least one processor to perform the data processing method as described above.

[0024] According to one aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, wherein when instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor causes the at least one processor to perform the data processing method as described above.

[0025] According to one aspect of the embodiments of the present disclosure, a computer program product is provided, including computer-executable instructions, wherein the computer-executable instructions, when executed by at least one processor, implement the data processing method as described above.

[0026] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0027] The above and other aspects, features and advantages of certain embodiments of this disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0028] Figure 1 This is a schematic diagram illustrating the application of a data processing method according to an embodiment of the present disclosure.

[0029] Figure 2 This is a flowchart illustrating a data processing method according to an embodiment of the present disclosure.

[0030] Figure 3 This is a flowchart illustrating the detailed operation of a data processing method according to an embodiment of the present disclosure.

[0031] Figure 4 This is a flowchart illustrating operations related to a mapping network according to embodiments of the present disclosure.

[0032] Figure 5 This is a flowchart illustrating the detailed operation of a data processing method according to an embodiment of the present disclosure.

[0033] Figure 6 This is a structural block diagram illustrating a data processing apparatus according to an embodiment of the present disclosure. Detailed Implementation

[0034] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0035] The terms “comprising” or “including” as used in this specification indicate the presence of the stated features, integrals, steps, operations, elements, components, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or combinations thereof. It should be understood that although the terms “first,” “second,” “third,” etc., are used to describe various information, the information should not be limited by these terms. These terms are used only to distinguish one type of information from another. For example, without departing from the scope of this disclosure, first information may be referred to as second information; and similarly, second information may be referred to as first information. As used herein, depending on the context, the term “in response to…” may be understood to mean “when,” “in,” or “if.”

[0036] Furthermore, in this disclosure, "at least one of..." refers to three parallel cases: "any one of...", "any combination of...", and "the whole of...". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. As another example, "performing at least one of step one and step two" includes the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0037] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as understood based on the disclosure of this application and as commonly understood by one of ordinary skill in the art to which this disclosure pertains, and should not be interpreted ideally or overly formally. The use of the term “may” in relation to examples or embodiments (e.g., “...may include”) indicates the existence of at least one example or embodiment that includes or implements such a feature, and all examples are not limited thereto. Unless otherwise expressly defined, the singular form of a term also includes the plural form.

[0038] As described in the background section, data management is of great importance to LLM, and there are some data management methods in related technologies.

[0039] However, most methods represent the overall quality of each data sample through quality metrics (or quality assessment metrics) such as text-image relevance, perplexity, and security, or by using a large model to replace human-generated quality scores to represent the overall quality of each data sample, and then filtering high-quality data samples for LLM training or fine-tuning by setting thresholds for each metric. These methods largely evaluate different data samples independently, thus ignoring the complex combinatorial effects between data samples. Due to information redundancy or inconsistency among the various quality labels of data samples, combinations of high-quality data samples do not fully meet the goal of maximizing LLM knowledge acquisition. Furthermore, some methods exist for dynamically filtering data, but these methods also have many problems, such as diverse sources of quality metrics, consideration of only individual data samples without considering the combinatorial effects of multiple data samples, the need for significant human resources to build domain-specific knowledge bases, and applicability only to plain text data samples.

[0040] In summary, the relevant data management technologies suffer from at least the following problems: Methods for evaluating data sample quality based on quality metrics (such as perplexity, correctness, diversity, complexity, etc.) rely on various quality metrics derived from different models or methods, each representing different aspects of the data sample's quality. This leads to information redundancy or inconsistency, and simply combining these metrics cannot yield truly high-quality datasets. LLMs trained on general-domain datasets may exhibit illusions or poor accuracy when answering specialized questions in specific domains. In specific domains, due to factors such as data confidentiality or labor costs, it is difficult to obtain high-quality labeled data samples for fine-tuning general-domain LLMs. Therefore, the efficiency and accuracy of data selection for fine-tuning LLMs in specific domains are also difficult to improve. Dynamic data selection methods often rely on a single quality metric (such as model loss, compression ratio, etc.) without considering the changes in LLM capabilities as the selected dataset changes. That is, as data selection progresses, the capabilities of the LLM will change even with known selected data. In such cases, the initial dataset required for the LLM should not be used alone.

[0041] In order to at least address the various problems and / or deficiencies mentioned above and / or in other aspects, embodiments of this disclosure propose a data processing method and apparatus, electronic device, storage medium, and program product for dynamically selecting data with high quality, diversity, and necessity.

[0042] Embodiments of this disclosure design a Mapper Network (MN). This Mapper Network can reflect the difference between the quality metric scores of reward models in a general domain and the quality requirements of data samples in a specific domain. During data filtering based on this Mapper Network, weights for various quality metrics can be obtained using this network, and these weights can be used to weight various quality metrics in the original dataset. Therefore, the difference between the general domain and the specific domain can be eliminated to a certain extent, filtering out high-quality data samples suitable for the specific domain without the need for expensive labeling of data samples in the specific domain.

[0043] Furthermore, embodiments of this disclosure propose an iterative data filtering method that combines clustering with Retrieval Augmented Generation (RAG). This iterative data filtering method dynamically filters data based on diversity and necessity. Embodiments of this disclosure propose an metric used in the iterative data filtering method—a Knowledge Retrieval (KR) score—which indicates the degree of knowledge content of the dataset already obtained through filtering for the current individual data to be filtered. The iterative data filtering method can determine whether to select the individual data based on the KR score. Therefore, based on the filtered data, changes in the LLM's capabilities can be known, and the KR score can be dynamically updated to filter data suitable for the current LLM (i.e., the LLM with currently known filtered data), without always using an initial dataset suitable for the initial LLM requirements and without needing to fine-tune the LLM.

[0044] Furthermore, embodiments of this disclosure employ a reward model to evaluate data quality metrics, and introduce image descriptions to make the reward model applicable to multimodal text-image modes. The quality metrics of this reward model can include various LLM and human expert scores, such as quality metrics encompassing aspects like correctness, usefulness, complexity, honesty, detail, security, and perplexity. This allows for unified and comprehensive evaluation across multiple datasets and multiple quality assessment directions, effectively eliminating the discrepancies arising from different sources of various quality metrics.

[0045] The application scenarios of the data processing method according to the embodiments of this disclosure will be described first. Figure 1 This is a schematic diagram illustrating the application of a data processing method according to an embodiment of the present disclosure.

[0046] In this disclosure, the data processing method may include the data management operations described above. (See reference...) Figure 1According to embodiments of this disclosure, by fine-tuning and / or training a general LLM 101 using baseline data or a baseline dataset (also referred to as raw data or raw dataset in this disclosure) 102, a more knowledgeable LLM 103 that may be more suitable for a specific technology can be obtained. When the baseline data 102 is managed using a data curator 111, a data set 112 with higher quality or more suitable for specific technology management, filtered from the baseline data 102, can be obtained. By fine-tuning and / or training the general LLM 101 using the managed data set 112, a domain-specific LLM 113 with better performance can be obtained with less data. In other words, the dataset obtained by the data processing method according to embodiments of this disclosure can be used for better fine-tuning and / or training of the general LLM.

[0047] Although this disclosure only illustrates the use of the obtained dataset for fine-tuning and / or training, the description herein is merely illustrative, and the application of the methods according to embodiments of this disclosure is not limited thereto.

[0048] The following continues to refer to... Figures 2 to 5 A data processing method according to embodiments of the present disclosure will be described in detail.

[0049] In this disclosure, a data processing method including a first-round data filtering based on a MapperNetwork (MN) is proposed to effectively filter out high-quality data. Figure 2 This is a flowchart illustrating a data processing method S200 according to an embodiment of the present disclosure. Figure 3 This is a flowchart illustrating the operation of data filtering based on a mapping network according to an embodiment of the present disclosure. Figure 4 This is a flowchart illustrating the determination of a mapping network according to an embodiment of the present disclosure.

[0050] Reference Figure 2 In operation S210, multiple mapping subsets are determined based on the original dataset used for training and / or fine-tuning of LLM. In other words, multiple mapping subsets of the original dataset are determined by randomly selecting data from the original dataset used for training and / or fine-tuning of LLM. Here, the mapping subsets may represent the subsets selected to determine weights in operation S220.

[0051] According to embodiments of this disclosure, the original dataset (such as, Figure 1 Baseline data 102 in Figure 3 The original dataset 301 or Figure 4The original dataset 401 in the dataset may include multiple baseline data samples (or referred to as original data samples) to be used for LLM training and / or fine-tuning, wherein each of the multiple baseline data samples may include (e.g. Figure 3 The question and the corresponding answer are shown in the diagram, i.e., QA pairs.

[0052] In this disclosure, the data samples included in the dataset for training and / or fine-tuning may also be simply referred to as data. Each data may include QA pairs of plain text information, but this disclosure is not limited thereto. For example, as will be described below, each data may also include visual information, such as images, questions, and answers with corresponding relationships, i.e., VQA (Vision Question Answer) pairs, and the data may also include any other types of information.

[0053] According to embodiments of this disclosure, multiple mapping subsets (such as, Figure 3 The mapped subset (302) comprises multiple subsets of data randomly selected from the original dataset, wherein each data point in each mapped subset includes a QA pair or a VQA pair, etc. For example, a predetermined number of data points can be selected from the original dataset by true randomness to form a mapped subset.

[0054] Furthermore, according to embodiments of this disclosure, each of the plurality of mapped subsets may include the same amount of data. However, this disclosure is not limited thereto, and the plurality of mapped subsets may also include different amounts of data.

[0055] In operation S220, based on the mapping network and multiple mapping subsets, weights corresponding to each quality metric are determined for the original dataset. Each quality metric here can be any of a plurality of predefined quality metrics. For example, the plurality of predefined quality metrics may include text coherence, richness, complexity, accuracy, authenticity, security, usefulness, correctness, verbosity, instruction compliance, honesty, etc., and these predefined quality metrics can be predetermined according to user needs.

[0056] In this disclosure, mapping networks (such as, Figure 3The input to the mapping network (MN 303) can be a quality metric score corresponding to the data in the input dataset (such as a mapping subset, a training subset, etc.), and the output of the mapping network can be the model accuracy of the LLM trained and / or trained using the input data, predicted by the mapping network based on the quality metric score. Therefore, the mapping network can reflect the mapping relationship from the quality metric score to the model accuracy. According to embodiments of this disclosure, the mapping network is used to predict the model accuracy corresponding to a mapping subset based on the quality metric of the data in that mapping subset, such as the model accuracy of an LLM trained and / or fine-tuned based on that mapping subset.

[0057] Before describing the specific details of weight determination in operation S220, let's first combine... Figure 3 and Figure 4 The determination of the mapping network according to embodiments of the present disclosure is described, wherein, Figure 3 The original dataset 301 in the dataset can correspond to Figure 4 The original dataset 401, Figure 3 The MN 303 used can correspond to Figure 4 MN 405 trained in the middle.

[0058] According to embodiments of this disclosure, a mapping network is determined based on multiple training subsets of the original dataset. Here, a training subset may refer to a subset of the dataset selected for training the mapping network. For example, see... Figure 4 The training operation S400 of the mapping network may include determining N training subsets for training MN 405 from the original dataset 401, wherein the N training subsets include the first training subset 401-1, the second training subset 401-2, the third training subset 401-3, ..., the Nth training subset 401-N, where N is a positive integer, for example, N=100.

[0059] As an example, each of the multiple training subsets may include the same number of data. However, this disclosure is not limited to this, and the multiple training subsets may also include different numbers of data. For example, each training subset may include P data samples, where P is a positive integer, such as P = 1000.

[0060] In this disclosure, in order to ensure the diversity of data used to train the mapping network, training data can be randomly selected from the original dataset to form a part of the training subset, and data can be selected based on randomly determined quality metrics and / or combinations of quality metrics to form another part of the training subset.

[0061] As an example, a training subset can be constructed by selecting P data points from the original dataset using true randomness.

[0062] As an example, P data points can be selected to form a training subset based on a single quality metric randomly determined (e.g., from multiple predetermined quality metrics). For instance, when the randomly determined quality metric is safety, all data in the original dataset can be sorted based on the safety metric score, and P data points with the highest safety metric score can be selected from all data to form the training subset. (Referring to...) Figure 3 The determination of indicator scores will be described, but will not be described in detail here.

[0063] As an example, P data points can be selected to form a training subset based on a combination of multiple quality metrics randomly determined (e.g., from multiple predetermined quality metrics). For instance, when the randomly determined combination of quality metrics is security and realism, all data in the original dataset can be sorted based on the combined metric score obtained by combining the security metric score with the realism metric score (e.g., through logical operations such as addition), and P data points with the highest combined metric score can be selected from all data to form the training subset. Here, this disclosure does not limit the type of quality metrics.

[0064] As an example, the number of randomly selected training subsets may account for a predetermined proportion of the N training subsets (e.g., 30%), and the number of training subsets selected based on quality metrics may account for another predetermined proportion of the N training subsets (e.g., 70%). However, such proportions are described as examples only, and this disclosure is not limited thereto. The proportions of selected training subsets may be set according to actual needs.

[0065] Therefore, according to embodiments of this disclosure, multiple training subsets may include: at least one subset having data randomly selected from the original dataset, and at least one subset having data selected based on at least one randomly determined quality metric.

[0066] As described above, the input to the mapping network can be the quality metric scores of the data. Correspondingly, during the training of the mapping network, the input can be the quality metric scores of the data in the training subset. In this disclosure, this can be based on a reward model (such as...). Figure 4 RM 402) determines the score of the quality indicator for the data. Here, Figure 4 The RM 402 used for training can be Figure 3 The RM 304 in the diagram can also be another RM used for training the mapping network, and this disclosure does not limit it to this.

[0067] According to embodiments of this disclosure, in operation S410, various quality metrics of the data in multiple training subsets 401-1 to 401-N are scored (i.e., quality assessed) using RM 402, obtaining a metric score (or reward score) 403 for each quality metric for each data point. For example, when the number of training subsets N is 100 and the number of data points P in each training subset is 1000, up to 100 × 1000 metric scores can be obtained for a certain quality metric using the reward model.

[0068] In this disclosure, a reward model (i.e., RM 402 or RM 304) is used to evaluate the quality of data samples. This reward model is trained on multiple datasets, including manually labeled and generatively pre-trained Transformer-4 (GPT-4) labeled datasets, which can cover multiple domains such as general technology, security, code, and mathematics. The output of the reward model includes scores for various quality metrics as described above.

[0069] The following reference Figure 3 This section provides a detailed description of how the reward model outputs quality metric scores based on the input data. Figure 3 The description of RM 304 in the diagram can be applied to RM 402 for training the mapping network.

[0070] For example, refer to Figure 3 When the data includes plain text data (i.e., QA pairs containing questions and answers), the text-based QA pairs can be input into the RM 304 for text processing. The RM 304 can score the data according to a series of quality indicators, that is, obtain an indicator score for each quality indicator.

[0071] For example, refer to Figure 3When the data includes multimodal data (e.g., VQA pairs containing questions, answers, and images), the images from the VQA pairs can be input into an LLM 306, which can convert the images into image descriptions in text form, detailing the specific content of the images. For example, an LLM (such as the Intern-VL-2.0-26B model) 306 can generate detailed image descriptions for each image and add the generated image descriptions as background knowledge to the system role content of the message. The LLM 306 here can be an LLM that will be trained and / or fine-tuned according to embodiments of this disclosure, but this disclosure is not limiting; the LLM 306 can also be another LLM that can convert images into image descriptions. The text-form image descriptions, questions, and answers are then input together into a reward model 304 for processing the text. For example, the image descriptions are combined with the QA pairs to form a message, and then the message is input into the RM 304. Similar to the processing of QA pairs, the RM 304 can score the data against various quality metrics.

[0072] To avoid unnecessary obscuring of details in this disclosure, the specific calculation details of the reward model will not be described here. The reward model may be, but is not limited to, the ArmoRM-Llama3-8B-v0.1 model.

[0073] Therefore, through Figure 4 Operation S410, for each training subset input to RM 402, RM 402 can output the index score 403 for each quality metric of each data in the training subset.

[0074] Here, the index scores of various quality metrics of the data in the training subset obtained by using the reward model can be used as input to the mapping network to train the mapping network into a model that determines the model accuracy of the LLM based on the quality metrics of multiple data points.

[0075] In operation S420, the general LLM (i.e., the LLM to be trained and / or fine-tuned according to embodiments of this disclosure) is trained and / or fine-tuned using N training subsets 401-1 to 401-N respectively. Figure 1The LLM (e.g., the Intern-VL-2.0-26B model) is fine-tuned to obtain N fine-tuned LLMs corresponding to N training subsets, namely, the first fine-tuned LLM 411-1, the second fine-tuned LLM 411-2, the third fine-tuned LLM 411-3, ..., the Nth fine-tuned LLM 411-N. To avoid unnecessary obscuring of details, this disclosure does not describe in detail the method of fine-tuning the LLM using data; any suitable LLM fine-tuning method can be used to obtain the fine-tuned LLM.

[0076] In operation S430, by using N fine-tuned LLMs 411-1 to 411-N respectively on the same test set, the model accuracy S1, S2, S3, ..., S1 for each of the N training subsets is obtained. N 404, where the model accuracy 404 obtained from the N tests can be used as a quality label for training MN 405. Here, to avoid unnecessary obscuring of details in this disclosure, the method of testing model accuracy through the model inference process using a fine-tuned LLM will not be described in detail. Any suitable model accuracy testing method can be used to obtain the model accuracy of the fine-tuned LLM.

[0077] According to embodiments of this disclosure, the LLM whose model accuracy is determined for each training subset is an LLM fine-tuned based on the corresponding training subset. That is, the LLM is fine-tuned using data from the training subset, and the model accuracy of the fine-tuned LLM is tested using a test set.

[0078] However, although embodiments of this disclosure illustrate fine-tuning the LLM using a training subset and obtaining the test model accuracy of the fine-tuned LLM, this disclosure is not limited thereto. The LLM can also be trained using a training subset and the test model accuracy of the trained LLM can be obtained. That is, the details described in operations S420 and S430 for the fine-tuning operation also apply to the training operation and will not be repeated here. Therefore, the LLM is learned using a training subset to update the LLM, and the updated LLM is tested for model accuracy using a test set.

[0079] According to embodiments of this disclosure, a mapping network (MN) is determined by using the model accuracy of the N tests obtained as quality labels and the index scores of various quality metrics obtained by rewarding the model for each training subset, to map the index scores to the quality labels (i.e., model accuracy).

[0080] Therefore, according to embodiments of this disclosure, the step of determining a mapping network based on multiple training subsets of the original dataset includes: determining a metric score for each quality metric of each data point in each training subset using a reward model; training and / or fine-tuning the LLM based on each training subset; testing the model accuracy of the trained and / or fine-tuned LLM corresponding to each training subset using a test set; and determining a mapping network that maps the metric scores to the model accuracy based on the determined metric scores and the tested model accuracy of the multiple training subsets. That is, during the training of the mapping network, for a given training subset, the metric scores for multiple quality metrics of each training data point in that training subset are obtained using a reward model, and the mapping network can be trained based on these metric scores obtained by the reward model and the quality labels corresponding to that training subset (i.e., the model accuracy based on the test set and the adjusted LLM test). In this case, the model accuracy determined using the trained mapping network can indirectly measure the effectiveness of the dataset for training and / or fine-tuning the LLM.

[0081] In related technologies, public domain data is typically used to train and / or fine-tune LLMs, but such trained and / or fine-tuned LLMs are not suitable for specific domains. Due to the diversity of quality metrics involved in the original dataset and the complexity of the combinations of these metrics, this disclosure proposes a data filtering method based on a mapping network to find an optimal combination of quality metrics suitable for a specific domain. Specifically, as described above, since N training subsets are randomly selected based on diversity, N model accuracies are obtained by testing N trained and / or fine-tuned LLMs (trained and / or fine-tuned using the N training subsets with diversity). These N model accuracies can be used as quality labels to train a mapping network together with reward scores obtained through a reward model. The mapping network determined in this way can learn which types of quality metrics yield better model accuracy and can predict the corresponding model accuracy based on the quality metrics, thereby enabling the filtering of higher-quality data from the original dataset. Therefore, given a mapping network that can predict the corresponding model accuracy based on the quality metrics of the dataset, data filtering can be performed using the determined mapping network.

[0082] According to embodiments of this disclosure, weights applied to the original dataset are determined based on a mapping network. In determining the weights, a subset of data (corresponding to a subset of mapped data selected from the multiple mapped datasets) is selected by using the mapping network to predict the model accuracy corresponding to randomly selected data (corresponding to multiple mapped subsets) in the original dataset. Then, the quality metrics of the subset of data are used to assign weights to the quality metrics of the original dataset.

[0083] As an example, for each of the multiple mapped subsets, the scores of various quality metrics for the data in each mapped subset can be randomly assigned. That is, for each of the Q mapped subsets, a combination of K scores for K quality metrics is randomly set, where Q and K are positive integers. Q may represent the number of the multiple mapped subsets determined in operation S210, and K may represent the number of various predetermined quality metrics, such as Q=100000 and K=10. For example, for a mapped subset Q1, 10 scores for 10 quality metrics are randomly assigned to each of its data samples (such as QA pairs). Accordingly, for each of the Q mapped subsets with 100 data samples, at most a total of Q×100×10 scores can be randomly assigned.

[0084] Then, by using a mapping network to predict the model accuracy corresponding to each mapping subset, Q model accuracies corresponding to Q mapping subsets are obtained. For example, by inputting the 10 randomly assigned metric scores mentioned above into MN 303, the model accuracy of an LLM trained and / or fine-tuned based on mapping subset Q1 is predicted.

[0085] As an example, based on the Q model accuracies corresponding to the Q mapping subsets, M mapping subsets corresponding to the M highest model accuracies are determined from the Q mapping subsets, where M is a positive integer less than or equal to Q, such as M = 1000. For example, 1000 mapping subsets corresponding to the 1000 highest model accuracies are selected from 100,000 mapping subsets.

[0086] As an example, for each quality indicator in the M mapped subsets, the ratio of the previously randomly assigned score of each indicator to the sum of the scores of all quality indicators is calculated, and this ratio is used as the weight of that quality indicator. Here, the values ​​of Q and M in this disclosure are merely examples, and the values ​​of Q and M can be arbitrarily set according to actual needs, or the relationship between Q and M can be set, such as the ratio of M to Q being 30% or 40%. For example, the ratios of the randomly assigned scores of each of the 10 quality indicators in the above 1000 mapped subsets to the sum of the scores of the 10 quality indicators (the sum of the ratios is 1) are calculated to obtain 10 ratios of the 10 quality indicators as the weights of the 10 quality indicators in the original dataset.

[0087] As an example, the pseudocode for determining the weights is shown below: Pseudocode for determining weights in Algorithm 1

[0088] As an example, the weight W can be calculated using the following equations (1) and (2):

[0089] Where M represents the number of mapped subsets, i can represent the index of the mapped subset, and k can represent the index of the quality metric. R can represent the score of the k-th quality metric for the data in the i-th mapped subset of M mapped subsets (such as the sum or average of the scores of the k-th quality metric for all data in the i-th mapped subset), i It can represent the sum of the scores of K quality indicators for the data in the i-th mapping subset of M mapping subsets (such as the sum or average of the scores of K quality indicators for all data in the i-th mapping subset).

[0090] In other words, in this disclosure, since there is a mapping relationship between the quality index scores and the model accuracy, a subset of data is arbitrarily selected and index scores are arbitrarily assigned to the subset of data. The model accuracy is determined based on the arbitrarily assigned index scores, and the importance of the index scores of the subset of data with high model accuracy is selected to determine the weight of the index scores applied to the original dataset while ensuring the model accuracy.

[0091] Therefore, according to embodiments of this disclosure, operation S220 may include: for each of the plurality of mapping subsets, randomly setting a metric score for each quality metric; determining the model accuracy of the LLM trained and / or fine-tuned based on the set metric score for each mapping subset by using a mapping network; selecting a subset of mapping subsets from the plurality of mapping subsets based on the plurality of model accuracies corresponding to the plurality of mapping subsets respectively; and determining the weight of each quality metric for the original dataset based on the statistical values ​​of various quality metrics of the selected subset of mapping subsets.

[0092] Here, a weight for each quality metric is determined based on a mapping network for at least a portion of the data in the original dataset, and these weights are treated as weights for each quality metric across all data in the original dataset. This is because the total amount of data in the original dataset is typically large, and predicting accuracy for all data might unnecessarily waste computational resources. Therefore, embodiments of this disclosure use randomly assigned quality metrics from multiple mapping subsets with diversity to predict accuracy, and select the necessary mapping subsets with high accuracy to obtain weights for filtering all data, thereby saving computational steps while ensuring accuracy.

[0093] After obtaining the weight of each quality indicator in the above operation S220, the obtained weights can be used to perform the first round of data filtering on all the data in the original dataset.

[0094] In operation S230, a first dataset for training and / or fine-tuning of LLM is determined from the original dataset based on the determined weights.

[0095] As an example, based on the weight of each quality indicator, the scores of each quality indicator for all data in the original dataset are weighted to obtain the weighted index score for each quality indicator for all data. For instance, for a specific dataset, the weighted index score for security is obtained by multiplying the determined security weight by the security index score for that dataset. Similarly, the weighted index score for each quality indicator for that dataset is obtained by multiplying the weight of each quality indicator by the index score of each quality indicator for that dataset.

[0096] As an example, the quality metric scores for which weighted operations are performed can be determined from the original dataset using a reward model. For instance, each data point in the original dataset 301 can be input into RM 304 to obtain the score for various quality metrics of that data. Since the operational details of RM 304 have already been described above, they will not be repeated here.

[0097] As an example, the quality score of a data point is obtained by summing the weighted scores of multiple quality metrics for each data point. For instance, for a given data point, the quality score is obtained by summing the weighted scores of all quality metrics, such as security, authenticity, and accuracy.

[0098] As an example, all data are sorted according to the quality score of each data point, and multiple first data points constituting the first dataset are selected from all data in the original dataset. For example, a predetermined number of data points with the highest quality scores are selected as the desired high-quality data.

[0099] Therefore, according to embodiments of this disclosure, operation S230 may include: determining the score of each quality indicator of the data in the original dataset by using a reward model; determining the quality score of the data in the original dataset by weighting the score of each quality indicator of the data based on the weight corresponding to each quality indicator; and selecting a predetermined number of first data points constituting a first dataset from the original dataset based on the quality scores of the data in the original dataset. That is, the quality score of the data is determined by weighting the quality indicators of the data based on the weights determined based on the mapping network, and the data is filtered based on the quality scores of all the data. Since each data point in the original dataset can be measured from the perspective of various quality indicators, high-quality desired data applicable to a specific field (such as the semiconductor field) can be obtained by filtering the data using weights determined based on the quality indicators.

[0100] For example, in Figure 3 In this process, MN 303 determines the weights of each quality metric for all data in the original dataset 301 based on the mapped subset dataset 302. RM 304 receives input questions and answers or image descriptions / questions and answers based on QA pairs or VQA pairs for each data point, and outputs the metric score for each quality metric for each data point. By applying the weights determined by MN 303 to the metric scores of the quality metrics determined by RM 304 and performing an addition operation, the quality score 307 for all data in the original dataset 301 is obtained. Based on the quality score 307, a first data point is selected from the original dataset 301, thereby determining the first dataset 305 containing the first data point.

[0101] In order to evaluate the technical effectiveness of the above-described data filtering method based on mapping networks according to embodiments of the present disclosure, ablation experiments were conducted.

[0102] Specifically, to verify that the method proposed in the embodiments of this disclosure can effectively screen high-quality, diverse, and necessary data samples, the following experiment was conducted: The Semi-ArxivQA Test, a test set in the semiconductor field, was used. This test set contains 1000 multiple-choice questions in the semiconductor field. The data samples in this test set may include semiconductor-related QA generated by GPT-4 from the ArxivQA public dataset, as well as QA generated by researchers using the Intern-VL model for figure-caption pairs in semiconductor-related literature. These were determined through manual verification and combination. Public datasets may include TextVQA, GQA, and ScienceQA. The base model may be the LLaVA-1.6-Vicuna 7B model. In the experiments, the effectiveness of the method was evaluated using the accuracy based on the number of correctly answered questions. This disclosure evaluates the accuracy of schemes based on the following data selection methods: all data, randomly selected data (Random_curated_142K), data selected by sorting by the total score of all quality metrics (All metrics_curated_142K), manually selected data based on the meaning of the quality metrics (Human_curated_142K), and data selected using the weights of a mapping network according to embodiments of this disclosure (Mapper_weight_curated_142K).

[0103] Table 1

[0104] The experimental results above show that the Semi_ArXivQA Test dataset contains some noise and poor-quality data in the original dataset. Filtering by all quality metrics resulted in only a small improvement in accuracy (70.7% → 71.1%), while manual filtering based on the meaning of the quality metrics significantly improved performance (70.7% → 72.2%). Filtering based on the mapping network according to embodiments of this disclosure achieved the best performance on the Semi_ArXivQA Test dataset (70.7% → 73.0%).

[0105] This indicates that the mapping network according to the embodiments of this disclosure can reflect, to some extent, the relationship between various quality indicators of the reward model and the overall quality of the data samples required by the test set, which is beneficial for screening high-quality data samples suitable for training and / or fine-tuning of the LLM.

[0106] Therefore, in the data processing of the original dataset containing some noise and / or poor quality data, a first dataset for training and / or fine-tuning the LLM can be obtained by using MN-based data filtering, thereby, for example, using the first dataset to obtain an LLM with the best performance test model accuracy compared to related techniques.

[0107] Furthermore, by adding image descriptions as background knowledge, data filtering using reward models can help to more accurately select high-quality data samples, thereby improving the training and / or fine-tuning of domain-specific LLMs.

[0108] In addition, after obtaining a data sample set with a certain quality assurance through the first round of data screening, this disclosure also proposes to conduct a second round of data screening for the data sample set by combining diversity and necessity, taking into account the relationship between the various data samples in the current sample data set.

[0109] According to embodiments of this disclosure, the data processing method S200 may further include operation S240: selecting second data from the first dataset to form a second dataset for training and / or fine-tuning of LLM by performing data filtering based on Retrieval-Augmented Generation (RAG). (Refer to...) Figure 5 The operation S240 is described in detail.

[0110] Figure 5 This is a flowchart illustrating the detailed operation of a data processing method according to an embodiment of the present disclosure.

[0111] As an example, in Figure 5 In this process, the first dataset 501 can be regarded as a pool of candidate data samples, and a RAG-based data filtering operation can be performed on the pool of candidate data samples to determine the second dataset 502.

[0112] The first dataset 501 here can, for example, correspond to... Figure 3 The first dataset 305 is mentioned. However, this disclosure is not limited thereto; the first dataset 501 may also correspond to... Figure 3 The original dataset 301.

[0113] As an example, in operation S510, a third dataset 503 is determined or formed by selecting a relatively distant portion of the data with diversity from the pool of candidate data samples through a clustering method.

[0114] For example, by using the k-Center-Greedy algorithm, the S third datasets with the largest Euclidean distance are selected from the first dataset 501 to form the third dataset 503, where S is a predefined positive integer that can be determined based on hardware capabilities such as the central processing unit (CPU), graphics processing unit (GPU), and memory.

[0115] According to embodiments of this disclosure, the data contained in the final knowledge database (or simply knowledge base or temporary dataset) 504 of the LLM 506 is considered as the second data in the second dataset 502. Therefore, for each piece of third data in the third dataset 503, this disclosure proposes determining the degree of knowledge inclusion of that third data in the current knowledge database 504 of the LLM 506 and determining the second data from the third dataset based on the determination result. In this application, the knowledge base may not contain any data or knowledge before the second data is determined.

[0116] As an example, for a third data point, RAG is used to retrieve X fourth data points 508 most relevant to that third data point from the general LLM 506 knowledge base 504, and these X fourth data points 508 are added to a prompt 505 for the LLM, where X is a predefined positive integer determined based on hardware capabilities such as the CPU, GPU, and memory. The prompt 505 with the added X fourth data points 508 is then used as background knowledge, along with the third data point (such as a QA pair or VQA pair for that third data point), and input into the general LLM 506.

[0117] Here, LLM 506 can be an LLM to be trained and / or fine-tuned according to embodiments of this disclosure, such as the Intern-VL-2.0-26B model. Furthermore, LLM 306 and LLM 506 can be the same LLM or different LLMs.

[0118] As an example, LLM 506 can evaluate whether the current knowledge base 504 contains knowledge of the third data based on prompt 505 and the third data. That is, for example, for a QA pair of third data, X fourth data 508 most relevant to the QA pair are selected from the knowledge base 504. LLM 506 determines whether the X fourth data 508 contain knowledge of the QA pair by using them as background knowledge, and determines whether to select the QA pair based on the determination result. Here, knowledge that the knowledge base contains knowledge of a certain data can mean that the knowledge base contains data that is the same as that data and / or data that is similar to that data (such as data containing knowledge of that data). That is, the case where the knowledge base contains knowledge of a certain data is not limited to the case where it includes that data. The evaluation of third data will be described in detail below, and will not be elaborated here.

[0119] As an evaluation result, if the current knowledge base 504 contains knowledge about the third data, and this third data may already be the same as or similar to some data in knowledge base 504, it may not be determined as the second data, that is, it will not be selected into knowledge base 504 and / or the second dataset 502. Therefore, this third data can be discarded in this screening, that is, this third data is not the third data 507 to be added.

[0120] Optionally, if the current knowledge base 504 contains knowledge about the third data, the third data can be added back to the candidate data sample pool, that is, the third data is not removed from the first dataset 501. Optionally, if the current knowledge base 504 contains knowledge about the third data, the third data can also be directly discarded and not added back to the candidate data sample pool, that is, the third data is removed from the first dataset 501.

[0121] As another evaluation result, if the current knowledge base 504 does not contain knowledge of the third data, there may be no data in knowledge base 504 that is the same as or similar to the third data. This data can be identified as the second data, i.e., it can be selected for inclusion in knowledge base 504 and the second dataset 502. Therefore, the third data (i.e., the third data 507 to be added) can be directly added to the current knowledge base 504, added as the second data to the second dataset 502, and the added third data removed from the first dataset 501.

[0122] According to embodiments of this disclosure, the above operation can be repeated for each third data in the third dataset until the determination of whether all third data is included in the second dataset is completed. In this application, the process of determining the third dataset and determining whether each third data in the third dataset will be included in the second dataset can be referred to as RAG-based data filtering.

[0123] As an example, once the determination of whether all third data is included in the second dataset is complete, the RAG-based data filtering can be performed again until the iteration stopping condition is met. For example, return to operation S510 and perform the RAG-based data filtering again, that is, select multiple third data from the first dataset to form the third dataset again and perform the same filtering process as above for each of the multiple third data. This will not be described again here. In this application, the process of repeatedly performing the RAG-based data filtering operation can also be referred to as RAG-based iterative data filtering.

[0124] As an example, the iteration stopping conditions for the above-mentioned RAG-based iterative data filtering may include, but are not limited to, the number of data in knowledge base 504 or the number of second data in second dataset 502 being equal to a predetermined value and / or all data in first dataset 501 being subjected to RAG-based data filtering, etc.

[0125] Here, although embodiments of this disclosure show that data is added to the second dataset 502 during each iteration, it is also possible not to add data to the second dataset 502 during the iteration, but to use the data in the knowledge base 504 as the second data at the end of the iteration.

[0126] Therefore, according to embodiments of this disclosure, the step of performing data filtering based on retrieval enhancement to generate RAG includes: repeatedly performing RAG-based data filtering until the number of data in the knowledge base or the number of second data in the second dataset equals a predetermined value and / or all first data in the first dataset has been subjected to RAG-based data filtering, wherein performing data filtering based on retrieval enhancement to generate RAG includes: selecting a plurality of third data from the first dataset to form a third dataset; performing the following steps for each third data in the third dataset: retrieving a plurality of fourth data related to the third data from the knowledge base using RAG; determining whether the knowledge base includes knowledge of the third data based on the plurality of fourth data and the third data using LLM; and adding the third data to the knowledge base and identifying it as second data based on knowledge that the knowledge base does not include the third data.

[0127] According to embodiments of this disclosure, this disclosure proposes using Knowledge Retrieval (KR) to determine the degree to which knowledge base 504 contains knowledge of the third data, thereby determining whether knowledge base 504 contains knowledge of the third data. The evaluation of the third data based on KR scores is described in detail below.

[0128] As an example, for each third data point, LLM 506 can obtain the perplexity of the third data point with the prompt (such as a prompt 505 that adds X fourth data points 508) and / or the third data point (such as a QA pair or a VQA pair) based on the prompt (such as a prompt 505 that adds X fourth data points 508) and / or the third data point (such as a QA pair or a VQA pair).

[0129] As an example, LLM 506 can determine the level of confusion with a prompt based on prompt 505 and the third data, can determine the level of confusion without a prompt based on the third data, and can take a ratio of the level of confusion with a prompt to the level of confusion without a prompt for the third data, which in this disclosure can be defined as a KR score 509.

[0130] Since perplexity represents the probability that an LLM is confident in the current answer, the greater the perplexity, the less the LLM understands the knowledge. Therefore, the KR score determined in the above way can reflect the difficulty of an LLM in generating an answer that approximates the question based on the prompts, and thus the necessity of the current data can be measured based on the KR score.

[0131] If the KR score 509 determined by LLM 506 is small (e.g., less than a predetermined threshold), indicating that LLM 506 can simply generate an answer approximating the question based on the prompt 505, then the current knowledge base 504 may already contain knowledge related to that data, and therefore, the data may not need to be added to the current knowledge base.

[0132] Conversely, if the KR score 509 determined by LLM 506 is large (e.g., greater than or equal to a predetermined threshold), the current knowledge base 504 may not contain knowledge related to that data, and therefore, the data may need to be added to the current knowledge base 504.

[0133] Similarly, the process of retrieving the fourth data and determining the KR score is repeated for each third data point in the third dataset.

[0134] As an example, third data that has a value greater than or equal to a predetermined threshold can be identified as third data 507 to be added.

[0135] As an example, a predetermined number of third data points with the highest KR score can be selected from multiple third data points in the third dataset 503 as the third data points to be added 507.

[0136] As an example, a predetermined number of third data points with the largest KR score among at least one third data point having a predetermined threshold can also be determined as third data points to be added 507.

[0137] However, the description based on KR scores here is merely an example, and this disclosure is not limited thereto.

[0138] According to embodiments of this disclosure, the step of determining whether a knowledge base includes knowledge of the third data based on multiple fourth data and the third data using LLM includes: adding multiple fourth data related to the third data to prompts for input into the LLM; obtaining the confusion level of the third data with prompts and the confusion level without prompts using LLM; determining the ratio of the confusion level of the third data with prompts to the confusion level without prompts as the knowledge retrieval score of the third data; and determining whether the knowledge base includes knowledge of the third data based on a comparison result of the knowledge retrieval score of the third data and a threshold.

[0139] Through the RAG-based iterative data filtering described above, a more diverse and necessary second dataset can be obtained, which can then be used for LLM training and / or fine-tuning.

[0140] In addition, this disclosure also conducted ablation experiments based on the aforementioned RAG-based iterative data screening.

[0141] Specifically, the dataset that has undergone the first round of screening based on the mapping network may have a certain quality, and this disclosure further iterates and screens the dataset. Researchers have evaluated the accuracy of the following data screening schemes: data obtained by continuing to screen the data based on the mapping network using weighted screening (Mapper_weight_curated_100K), data screened based on the Instruction Following Difficulty (IFD) score (142k_Diverse_IFD_100K), and data screened by iterative RAG-based screening according to embodiments of this disclosure (142k_diverse_RAG_100K).

[0142] Table 2

[0143] Experimental results show that, with the same amount of data, the RAG-based iterative data filtering according to the embodiments of this disclosure has the best performance compared to related calculations.

[0144] Furthermore, although this disclosure shows a data processing method in which MN-based data filtering (operations S210 to S230) and RAG-based iterative data filtering (operation S240) are performed sequentially, this disclosure is not limited thereto. That is, only MN-based data filtering (operations S210 to S230) or RAG-based iterative data filtering (operation S240) may be performed, or RAG-based iterative data filtering (operation S240) may be performed first and then MN-based data filtering (operations S210 to S230) may be performed.

[0145] Furthermore, two-stage ablation experiments showed that, when a high-quality data sample set was obtained through MN-based data screening, RAG-based iterative data screening achieved better performance improvement.

[0146] Through the two rounds of data filtering described above, data that simultaneously meets the requirements of high quality, diversity, and necessity can be selected from the original dataset, thereby improving the efficiency and accuracy of model fine-tuning.

[0147] Figure 6 This is a structural block diagram illustrating a data processing apparatus according to an embodiment of the present disclosure.

[0148] Reference Figure 6 The data processing device 600 includes a subset determination unit 610, a weight determination unit 620, and a first dataset determination unit 630.

[0149] The subset determination unit 610 can be configured to determine multiple mapped subsets based on the original dataset used for training and / or fine-tuning of LLM.

[0150] The weight determination unit 620 can be configured to determine the weights for each quality metric in the original dataset based on a mapping network and multiple mapping subsets, wherein the mapping network is used to predict the model accuracy corresponding to the mapping subset based on the quality metric of the data in the mapping subset.

[0151] The first dataset determination unit 630 can be configured to determine a first dataset for training and / or fine-tuning LLM from the original dataset based on determined weights.

[0152] In other words, the subset determination unit 610 can be configured to perform the operation corresponding to operation S210, the weight determination unit 620 can be configured to perform the operation corresponding to operation S220, and the first dataset determination unit 630 can be configured to perform the operation corresponding to operation S230. Therefore, the above reference Figures 2 to 5 The details of the method described are applicable to the data processing apparatus 600, therefore, they will not be described in detail here.

[0153] According to embodiments of this disclosure, the mapping network may be determined based on multiple training subsets of the original dataset, wherein the multiple training subsets may include: at least one subset having data randomly selected from the original dataset, and at least one subset having data selected based on at least one randomly determined quality metric.

[0154] According to embodiments of this disclosure, the mapping network can be determined based on the plurality of training subsets by: determining a metric score for each quality metric for each data in each training subset using a reward model; training and / or fine-tuning the LLM based on each training subset; testing the model accuracy of the LLM trained and / or fine-tuned for each training subset using a test set; and determining a mapping network that maps the metric scores to the model accuracy based on the determined metric scores and the tested model accuracy from the plurality of training subsets.

[0155] According to embodiments of this disclosure, the weight determination unit 620 may be configured to determine weights for each quality metric in the original dataset based on a mapping network and multiple mapping subsets by: randomly setting a metric score for each quality metric for each of the multiple mapping subsets; determining the model accuracy of an LLM trained and / or fine-tuned based on the set metric score for each mapping subset using the mapping network; selecting a subset of mapping subsets from the multiple mapping subsets based on multiple model accuracies corresponding to the multiple mapping subsets respectively; and determining the weights for each quality metric in the original dataset based on the statistical values ​​of various quality metrics in the selected subset of mapping subsets.

[0156] According to embodiments of this disclosure, the first dataset determination unit 630 may be configured to determine a first dataset for training and / or fine-tuning LLM from the original dataset based on determined weights by: determining the index score of each quality metric of the data in the original dataset by using a reward model; determining the quality score of the data in the original dataset by weighting the index scores of each quality metric of the data based on the weights corresponding to each quality metric; and selecting a predetermined number of first data points constituting the first dataset from the original dataset based on the quality scores of the data in the original dataset.

[0157] According to embodiments of the present disclosure, the data processing apparatus 600 further includes a second dataset determination unit.

[0158] According to embodiments of this disclosure, the second dataset determination unit may be configured to: select second data from the first dataset to form a second dataset for training and / or fine-tuning of LLM by performing data filtering based on retrieval enhancement to generate RAG.

[0159] According to embodiments of this disclosure, a second dataset determination unit may be configured to select second data from a first dataset to form a second dataset for LLM training and / or fine-tuning by performing data filtering based on retrieval enhancement to generate RAGs: repeatedly performing RAG-based data filtering until the number of data in the knowledge base or the number of second data in the second dataset equals a predetermined value and / or all first data in the first dataset has been filtered based on RAGs, wherein the step of performing data filtering based on retrieval enhancement to generate RAGs includes: selecting a plurality of third data from the first dataset to form a third dataset; performing the following steps for each third data in the third dataset: retrieving a plurality of fourth data related to the third data from the knowledge base using RAGs; determining whether the knowledge base includes knowledge of the third data based on the plurality of fourth data and the third data using LLMs; and adding the third data to the knowledge base and determining it as second data based on knowledge that the knowledge base does not include the third data.

[0160] According to embodiments of this disclosure, the second dataset determining unit may be configured to select a plurality of third data from the first dataset to form a third dataset by using a K-centroid greedy algorithm to select a predetermined number of third data with the largest Euclidean distance from the first dataset.

[0161] According to embodiments of this disclosure, the second dataset determination unit may be configured to determine whether a knowledge base includes knowledge of the third data based on a plurality of fourth data and the third data by using an LLM: adding a plurality of fourth data related to the third data to a prompt for input into the LLM; obtaining, by using an LLM, the perplexity of the third data with and without prompts; determining the ratio of the perplexity of the third data with and without prompts as the knowledge retrieval score of the third data; and determining whether the knowledge base includes knowledge of the third data based on a comparison of the knowledge retrieval score of the third data with a threshold.

[0162] Furthermore, it should be understood that the various modules or units in the data processing apparatus 600 according to embodiments of this disclosure may be implemented as hardware components and / or software components. Those skilled in the art may implement the various modules or units, for example, using field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs), according to the processes performed by the defined modules or units.

[0163] Furthermore, according to embodiments of this disclosure, this disclosure also provides an electronic device, including: at least one processor; at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, cause the at least one processor to perform the data processing method as described above.

[0164] According to embodiments of this disclosure, the electronic device may be a PC computer, tablet device, personal digital assistant, smartphone, or other device capable of executing the aforementioned set of instructions. Here, the electronic device is not necessarily a single electronic device, but may be any collection of devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. The electronic device may also be part of an integrated control system or system manager, or may be configured to interconnect with a portable electronic device locally or remotely (e.g., via wireless transmission) through an interface.

[0165] In electronic devices, processors may include central processing units (CPUs), graphics processing units (GPUs), programmable logic devices, dedicated processor systems, microcontrollers, or microprocessors. By way of example and not limitation, processors may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc.

[0166] The processor can execute instructions or code stored in memory, which can also store data. Instructions and data can also be sent and received over a network via a network interface device, which can employ any known transport protocol.

[0167] Memory can be integrated with the processor; for example, RAM or flash memory can be housed within an integrated circuit microprocessor. Alternatively, memory can comprise a separate device, such as an external disk drive, storage array, or other storage device that can be used by any database system. Memory and processor can be operatively coupled, or can communicate with each other, for example, via I / O ports, network connections, etc., enabling the processor to read files stored in the memory.

[0168] In addition, electronic devices may include video displays (such as liquid crystal displays) and user interaction interfaces (such as keyboards, mice, touch input devices, etc.). All components of the electronic device may be interconnected via buses and / or networks.

[0169] Furthermore, according to embodiments of this disclosure, a computer program product including computer-executable instructions is also provided, wherein the computer-executable instructions, when executed by at least one processor, implement the data processing method described above.

[0170] Furthermore, according to embodiments of this disclosure, a computer-readable storage medium is also provided, wherein when instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor causes the at least one processor to perform the data processing method as described above.

[0171] According to embodiments of this disclosure, examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in electronic devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0172] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0173] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the claims.

Claims

1. A data processing method, comprising: Multiple mapping subsets are determined based on the original dataset used for training and / or fine-tuning the Large Language Model (LLM). Based on the mapping network and the multiple mapping subsets, weights corresponding to each quality metric are determined for the original dataset, wherein the mapping network is used to predict the model accuracy corresponding to the mapping subset based on the quality metric of the data in the mapping subset. A first dataset for training and / or fine-tuning the LLM is determined from the original dataset based on the determined weights.

2. The data processing method according to claim 1, wherein, The mapping network is determined based on multiple training subsets of the original dataset, and The plurality of training sub-datasets include: at least one sub-dataset having data randomly selected from the original dataset, and at least one sub-dataset having data selected based on at least one quality metric determined randomly.

3. The data processing method according to claim 2, wherein, The mapping network is determined based on the multiple training subsets in the following manner: By using a reward model, the score of each quality metric for each data point in each training subset is determined. The LLM is trained and / or fine-tuned based on each training subset dataset; The accuracy of the LLM model trained and / or fine-tuned for each training subset is tested using a test set. Based on the determined index scores and tested model accuracy of the multiple training subsets, a mapping network is determined that maps the index scores to the model accuracy.

4. The data processing method according to claim 1, wherein, Based on the mapping network and the multiple mapping subsets, the step of determining the weights for each quality metric in the original dataset includes: For each of the multiple mapped subsets, a score is randomly assigned to each quality metric. By using the mapping network, the accuracy of the LLM model trained and / or fine-tuned based on the metric score set for each mapping subset is determined. Based on the multiple model accuracies corresponding to the multiple mapping subsets respectively, a portion of the mapping subsets are selected from the multiple mapping subsets; Based on the statistical values ​​of various quality metrics of the selected partial mapping subset, the weight of each quality metric for the original dataset is determined.

5. The data processing method according to claim 1, wherein, The step of determining a first dataset for training and / or fine-tuning the LLM from the original dataset based on determined weights includes: By using a reward model, the score of each quality indicator for the data in the original dataset is determined; The quality score of the data in the original dataset is determined by weighting the scores of each quality indicator based on the weights corresponding to each quality indicator. Based on the quality scores of the data in the original dataset, a predetermined number of first data points are selected from the original dataset to constitute the first dataset.

6. The data processing method according to claim 1 further includes: By performing data filtering based on retrieval enhancement to generate RAG, second data is selected from the first dataset to form a second dataset for training and / or fine-tuning of the LLM.

7. The data processing method according to claim 6, wherein, The step of selecting second data from the first dataset to form the second dataset for training and / or fine-tuning of the LLM by performing data filtering based on retrieval enhancement to generate RAG includes: Repeatedly perform RAG-based data filtering until the number of data in the knowledge base or the number of second data in the second dataset equals a predetermined value and / or all the first data in the first dataset has been filtered using RAG-based data filtering. The steps for performing RAG-based data filtering include: Select multiple third datasets from the first dataset to form the third dataset; For each third data in the third dataset, perform the following steps: retrieve multiple fourth data related to the third data from the knowledge base using RAG; determine whether the knowledge base includes knowledge of the third data based on the multiple fourth data and the third data using LLM; and add the third data to the knowledge base and identify it as second data based on the knowledge that the knowledge base does not include the third data.

8. The data processing method according to claim 7, wherein, The steps for selecting multiple third datasets from the first dataset to form the third dataset include: By using the K-centroid greedy algorithm, a predetermined number of third data points with the largest Euclidean distance are selected from the first dataset.

9. The data processing method according to claim 7, wherein, The steps of determining whether the knowledge base includes knowledge of the third data based on the plurality of fourth data and the third data using the LLM include: Add the plurality of fourth data related to the third data to the prompts used for input into the LLM; By using the LLM, the perplexity of the third data with and without prompts is obtained; The ratio of the confusion level of the third data with prompts to the confusion level without prompts is determined as the knowledge retrieval score of the third data. Based on the comparison result of the knowledge retrieval score of the third data with the threshold, it is determined whether the knowledge base includes the knowledge of the third data.

10. A data processing apparatus, comprising: The subset determination unit is configured to determine multiple mapping subsets based on the original dataset used for training and / or fine-tuning of the large language model LLM; The weight determination unit is configured to determine the weights for each quality metric of the original dataset based on the mapping network and the plurality of mapping subsets, wherein the mapping network is used to predict the model accuracy corresponding to the mapping subset based on the quality metric of the data in the mapping subset; The first dataset determination unit is configured to determine a first dataset for training and / or fine-tuning the LLM from the original dataset based on determined weights.

11. An electronic device, comprising: At least one processor; At least one memory that stores computer-executable instructions. Wherein, when the computer-executable instructions are executed by the at least one processor, they cause the at least one processor to perform the data processing method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, wherein, When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor causes the at least one processor to perform the data processing method as described in any one of claims 1 to 9.

13. A computer program product comprising computer-executable instructions, wherein, When the computer-executable instructions are executed by at least one processor, they implement the data processing method as described in any one of claims 1 to 9.

Images