Model training data set determination method and apparatus, electronic device, and storage medium

Abstract

This application applies to the field of computer technology and provides a method, apparatus, electronic device, and storage medium for determining a training dataset for a model. The method includes: performing multiple clustering operations on data in an original dataset to obtain multiple clustering identifiers for each data in the original dataset; wherein each data in the original dataset is assigned a clustering identifier each time it is clustered; and obtaining a training dataset from the original dataset based on the multiple clustering identifiers for each data in the original dataset, so as to train a large language model based on the training dataset.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus, electronic device and storage medium for determining the training dataset of a model. Background Technology

[0002] Fine-tuning general-purpose large models to train industry-specific large models has become the mainstream approach for deploying large models in industry applications. High-quality training sets are the cornerstone of efficient inference for these large models. The current method for selecting datasets in related technologies involves random selection based on a certain proportion, with a high percentage allocated to the training set. This method not only results in unstable inference performance but also leads to high costs associated with manual labeling due to the large proportion of the training set. Summary of the Invention

[0003] In view of this, embodiments of this application provide a method, apparatus, electronic device, and storage medium for determining the training dataset of a model.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] On one hand, embodiments of this application provide a method for determining the training dataset of a model, the method comprising:

[0006] The data in the original dataset are clustered multiple times to obtain multiple cluster identifiers for each data point in the original dataset; wherein, each data point in the original dataset is assigned a cluster identifier in each clustering process;

[0007] Based on multiple clustering identifiers for each data point in the original dataset, a training dataset is obtained from the original dataset to train a large language model.

[0008] In the above scheme, obtaining the training dataset from the original dataset based on multiple clustering identifiers for each data point in the original dataset includes:

[0009] Based on the multiple clustering identifiers of each data in the original dataset, data with the same multiple clustering identifiers are grouped into the same dataset to obtain multiple datasets;

[0010] The training dataset is formed by selecting data from the multiple datasets.

[0011] In the above scheme, selecting data from the plurality of datasets to form the training dataset includes:

[0012] The same number of data points are selected from each dataset to form the training dataset.

[0013] In the above scheme, the step of performing multiple clustering operations on the data in the original dataset includes:

[0014] Extract the key features of each data point in the original dataset;

[0015] Based on the key features, the data in the original dataset are clustered.

[0016] In the above scheme, the step of performing multiple clustering operations on the data in the original dataset based on the key features includes:

[0017] Determine the similarity between key features of the data in the original dataset;

[0018] The data with the same similarity are assigned the same clustering identifier.

[0019] In the above scheme, before clustering the original dataset, the method further includes:

[0020] The original dataset is deduplicated based on key fields.

[0021] On the other hand, embodiments of this application provide a model reasoning method, the method comprising:

[0022] The data to be inferred is input into a large language model, which is used for inference; the training dataset of the large language model is constructed by performing multiple clustering operations on the data in the original dataset to obtain multiple clustering identifiers for each data, and then extracting data from the original dataset based on the multiple clustering identifiers for each data in the original dataset; wherein, each data in the original dataset is assigned a clustering identifier in each clustering operation.

[0023] Obtain the reasoning result of the large language model on the data to be reasoned.

[0024] On the other hand, embodiments of this application provide a training dataset determination device for a model, comprising:

[0025] The clustering module is used to perform multiple clustering operations on the data in the original dataset to obtain multiple clustering identifiers for each data point in the original dataset; wherein, each data point in the original dataset is assigned a clustering identifier in each clustering operation.

[0026] The acquisition module is used to acquire a training dataset from the original dataset based on multiple clustering identifiers for each data point in the original dataset, so as to train a large language model based on the training dataset.

[0027] On the other hand, embodiments of this application provide a model reasoning method, the method comprising:

[0028] An input module is used to input the data to be inferred into a large language model, which is used for inference; the training dataset of the large language model is formed by performing multiple clustering operations on the data in the original dataset to obtain multiple clustering identifiers for each data, and then obtaining data from the original dataset based on the multiple clustering identifiers for each data in the original dataset; wherein, each data in the original dataset is assigned a clustering identifier in each clustering operation.

[0029] The reasoning module is used to obtain the reasoning results of the large language model on the data to be reasoned.

[0030] On the other hand, embodiments of this application also provide an electronic device, including: a processor and a memory for storing a computer program capable of running on the processor, wherein the processor, when running the computer program, executes the steps of the training dataset determination method or model inference method of the model in the above-described scheme.

[0031] On the other hand, embodiments of this application also provide a computer storage medium storing a computer program, which, when executed by a processor, implements the steps of the training dataset determination method or model inference method of the above-described scheme.

[0032] On the other hand, embodiments of this application also provide a computer program product, including a computer program, which, when executed by a processor, implements the steps of the training dataset determination method or model inference method of the above-described scheme.

[0033] This application embodiment performs multiple clustering operations on the data in the original dataset to obtain multiple clustering identifiers for each data point in the original dataset. Each data point in the original dataset is assigned a clustering identifier during each clustering. Based on these multiple clustering identifiers for each data point in the original dataset, a training dataset is obtained from the original dataset to train a large language model. This application embodiment, by performing multiple clustering operations on the data in the original dataset, obtains a training dataset with low redundancy based on the clustering identifiers from these multiple clustering operations. The data in the training dataset is more comprehensive, enabling a small training set to cover the data features of a large test set, which is beneficial for improving the training effect of the large language model. Attached Figure Description

[0034] Figure 1 A flowchart illustrating a method for determining the training dataset of a model provided in an embodiment of this application;

[0035] Figure 2 A flowchart illustrating a model reasoning method provided in an embodiment of this application;

[0036] Figure 3 This is a schematic diagram of the overall process of a dataset selection method provided in an embodiment of this application;

[0037] Figure 4 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0038] The present application will now be described in further detail with reference to the accompanying drawings and embodiments.

[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0040] Traditional dataset selection methods involve random selection based on a proportional ratio, with a high proportion of training data, for example, a training:validation:test ratio of 7:1:2. This method not only results in unstable inference performance but also leads to high manual labeling costs due to the high proportion of training data.

[0041] To address the shortcomings of the prior art, this application provides a method for determining the training dataset of a model. To illustrate the technical solution described in this application, specific embodiments are provided below.

[0042] Figure 1 This application provides a flowchart illustrating a method for determining the training dataset of a model, as illustrated in the embodiments of this application. Figure 1 The method includes:

[0043] S101, perform multiple clustering operations on the data in the original dataset to obtain multiple clustering identifiers for each data in the original dataset; wherein, each data in the original dataset is assigned a clustering identifier each time clustering is performed.

[0044] Here, the data in the original dataset is used to train and test the large language model.

[0045] Data clustering refers to dividing data into clusters based on its inherent properties. Elements in each cluster should have as many similar characteristics as possible, while the differences in characteristics between different clusters should be as large as possible.

[0046] For example, a density-based clustering method with noise (Hierarchical Density-Based Spatial Clustering of Applications with Noise, HDBSCAN) can be used. HDBSCAN is a density-based clustering method used to group data together based on the similarity between data points. In time series clustering, HDBSCAN can discover dense clusters of time series data and dynamically adjust the number of clusters.

[0047] HDBSCAN divides data into a hierarchical clustering structure by finding data regions that are denser than their surroundings. Similar to islands at sea, multiple clusters of closely packed islands can be grouped together. HDBSCAN's clustering principle involves: calculating the coreity of each data point; connecting data points with high coreity to form connected regions; assigning data points within connected regions to adjacent clusters; and repeating this process until all data points have been assigned to clusters.

[0048] For each data point in the original dataset, a cluster identifier (ID) is assigned to each data point in each clustering process. After N clustering iterations, each data point in the original dataset corresponds to N cluster identifiers.

[0049] For example, in the original dataset, data A, after three clustering operations, has multiple cluster labels of 1-5-5. 1, 5, and 5 are the cluster labels assigned to data A in each clustering operation. The cluster labels assigned in each clustering operation can be the same or different.

[0050] This application does not limit the number of clustering operations. In one embodiment, two clustering operations can be performed, which can both ensure the clustering effect and improve the clustering efficiency.

[0051] S102, Based on multiple clustering identifiers for each data point in the original dataset, obtain a training dataset from the original dataset to train a large language model based on the training dataset.

[0052] The original dataset is divided into multiple datasets based on the multiple clustering identifiers of each data point. Then, data is extracted from these multiple datasets to construct the training dataset, which can reduce the duplication of the training dataset.

[0053] To address the shortcomings of single-clustering, such as excessive noise, potential inaccuracy, and excessive data volume in a particular cluster, this application's embodiments employ multiple clustering methods. Based on multiple clustering identifiers for each data point, it is possible to understand the category of each data point, thereby enabling accurate classification of all data.

[0054] For example, in the original dataset, after three clustering operations, if data A has multiple cluster labels including 1-5-5, with cluster label 5 being the majority, then data A is assigned to the dataset labeled 5; if data B has cluster labels including 3-5-5, then data B is also assigned to the dataset labeled 5.

[0055] Alternatively, data with identical cluster labels in all three clusters can be grouped into the same dataset. In other words, multiple cluster labels form the label of a dataset. If multiple cluster labels of data match the label of a dataset, then the data is grouped into that dataset. For example, if data A has multiple cluster labels including 1-5-5, then this data will be grouped into the dataset labeled 1-5-5.

[0056] For example, N (N can be user-defined) data points can be randomly selected from each dataset to form a training dataset.

[0057] In one embodiment, the training dataset is only a small portion of the original dataset; for example, the training dataset occupies one-tenth of the original dataset.

[0058] The selected training dataset is used to train a large language model. In some embodiments, the large language model can be a Generative Pre-trained Transformer (GPT) large model. GPT is a deep learning model, especially popular in the field of Natural Language Processing (NLP). For example, chatGPT, Tongyi Qianwen, and Wenxin Yiyan are all GPT large models. Specifically, the GPT model is based on the Transformer architecture, especially its self-attention mechanism, which enables the model to capture long-distance dependencies.

[0059] Large language models can be trained on a base model, which can be a model trained on a large amount of general corpus, such as the Llama series models open sourced by Meta, the QWEN model open sourced by Tongyi Qianwen, etc.

[0060] It should be understood that it is necessary to label the data in the training dataset, and then train a large language model based on the labeled training dataset to learn the relationship between the data and the labels.

[0061] This application embodiment performs multiple clustering operations on the data in the original dataset to obtain multiple clustering identifiers for each data point in the original dataset. Each data point in the original dataset is assigned a clustering identifier during each clustering. Based on these multiple clustering identifiers for each data point in the original dataset, a training dataset is obtained from the original dataset to train a large language model. This application embodiment, by performing multiple clustering operations on the data in the original dataset, obtains a training dataset with low redundancy based on the clustering identifiers from these multiple clustering operations. The data in the training dataset is more comprehensive, enabling a small training set to cover the data features of a large test set, which is beneficial for improving the training effect of the large language model.

[0062] In one embodiment, obtaining the training dataset from the original dataset based on multiple clustering identifiers for each data point in the original dataset includes:

[0063] Based on the multiple clustering identifiers of each data in the original dataset, data with the same multiple clustering identifiers are grouped into the same dataset to obtain multiple datasets;

[0064] The training dataset is formed by selecting data from the multiple datasets.

[0065] For example, if data A and data B in the original dataset have multiple cluster labels of 1-5-5 after 3 clusterings, and data B also has multiple cluster labels of 3-5-5, then data A and data B can be classified into the same dataset.

[0066] In this way, the dataset will inevitably contain data of the same type. When selecting a training dataset, it is not necessary to select too much data from a single dataset, thus obtaining a training dataset with low repetition.

[0067] In one embodiment, selecting data from the plurality of datasets to form the training dataset includes:

[0068] The same number of data points are selected from each dataset to form the training dataset.

[0069] Obtaining the same amount of data from each dataset ensures the uniformity of data types in the constructed training dataset, thus guaranteeing the model's training effectiveness.

[0070] In one embodiment, performing multiple clustering operations on the data in the original dataset includes:

[0071] Extract the key features of each data point in the original dataset;

[0072] Based on the key features, the data in the original dataset are clustered.

[0073] For example, key features of data can be extracted using the Term Frequency-Inverse Document Frequency (TF-IDF) algorithm, where key features refer to term frequency and inverse document frequency.

[0074] TF-IDF is a commonly used weighted statistical technique in information retrieval and text mining to assess the importance of a word within a text set or corpus. A word's importance increases directly with its frequency in the text and decreases inversely with its frequency in the corpus. Simply put, if a word appears frequently in a text, it is considered important within that text. Conversely, the less frequently a word appears in other texts in the corpus, the more representative it is of that text.

[0075] For example, data with the same TF-IDF index can be grouped into the same dataset.

[0076] In one embodiment, the step of performing multiple clustering operations on the data in the original dataset based on the key features includes:

[0077] Determine the similarity between key features of the data in the original dataset;

[0078] The data with the same similarity are assigned the same clustering identifier.

[0079] Here, the similarity between key features can be calculated. If the similarity between two key features is greater than a threshold, the two data points can be assigned the same clustering label.

[0080] In one embodiment, before clustering the original dataset, the method further includes:

[0081] The original dataset is deduplicated based on key fields.

[0082] Here, deduplication is a data cleaning step for the original dataset. Deduplication considers two datasets with identical key fields as duplicate datasets, and removes invalid datasets. Datasets whose encoding format cannot be identified or whose key fields are all empty are considered invalid datasets.

[0083] Which fields to deduplicate is customizable. For example, for static data in a database, key fields could be the data source name, database name, table name, column name, etc.

[0084] In one embodiment, the method further includes:

[0085] Add key fields to data in the original dataset that are missing key fields.

[0086] For datasets with missing key fields, the key fields can be filled in by inputting external reference data; the filled dataset has richer features and can achieve better clustering results.

[0087] In one embodiment, the method further includes:

[0088] The data in the original dataset is segmented into words.

[0089] The program segments a complete sentence and outputs a list of corresponding words. It requires external input of some commonly used pause words to make the segmented words more human-like.

[0090] In one embodiment, the method further includes:

[0091] The data in the training dataset are labeled.

[0092] Related technologies employ manual labeling, rule-based automatic labeling, and AI-based feature analysis-based recommendation labeling. Manual labeling is inefficient and costly; rule-based labeling has poor generalization ability and a high false positive rate; AI-based recommendation labeling has significantly improved efficiency and generalization ability, but the false positive rate remains high.

[0093] This application's embodiments utilize a large model for data labeling. This requires training a large labeling model. The training data is initially labeled using an automated labeling tool developed using traditional methods (rule-based or AI feature-based). The main purpose is to reduce the amount of manually labeled data and improve the efficiency of training set labeling. Then, the results of the automated labeling tool are manually calibrated; data not covered by the tool requires manual labeling. Using a large model for labeling allows for generalization to all types of data, significantly improving the efficiency of data classification and grading while maintaining high accuracy, and substantially reducing the cost of manual labeling for data classification and grading.

[0094] Figure 2 A flowchart illustrating another model inference method is provided for embodiments of this application. (Refer to...) Figure 2 The method includes:

[0095] S201, the data to be inferred is input into the large language model, which is used for inference; the training dataset of the large language model is formed by obtaining multiple clustering identifiers for each data by performing multiple clustering on the data in the original dataset, and then obtaining data from the original dataset based on the multiple clustering identifiers for each data in the original dataset; wherein, each data in the original dataset is assigned a clustering identifier each time clustering is performed.

[0096] S202, Obtain the reasoning result of the large language model on the data to be reasoned.

[0097] This process involves performing multiple clustering operations on the data in the original dataset to obtain multiple cluster identifiers for each data point. Each data point in the original dataset is assigned a cluster identifier during each clustering process. Based on these multiple cluster identifiers for each data point in the original dataset, a training dataset is then obtained from the original dataset.

[0098] For example, based on multiple clustering identifiers for each data point in the original dataset, data points with the same multiple clustering identifiers are grouped into the same dataset, resulting in multiple datasets;

[0099] The training dataset is formed by selecting data from the multiple datasets.

[0100] The same number of data points can be selected from each dataset to form the training dataset.

[0101] The large language model is used to infer the data to be inferred, and the inference result is obtained.

[0102] The reasoning here can also be understood as classifying and grading the data to be reasoned. Using a large language model can improve the efficiency of data classification and grading.

[0103] The embodiments of this application can be applied to products such as data security brain and data security situation awareness, and are used to classify and grade data.

[0104] Figure 3 This is a schematic diagram of the overall process of a dataset selection method provided in an embodiment of this application, including:

[0105] The entire dataset is cleaned, keyword completion is performed, word segmentation is performed, features are extracted, first clustering is performed, second clustering is performed, and training set selection is performed.

[0106] The data cleaning module deduplicates the dataset and removes invalid data; the key field completion module uses externally input data to fill in missing key fields in the dataset; the word segmentation module segments a complete sentence and outputs a word list; the feature extraction module performs vectorized analysis on the word list and outputs a feature matrix; the clustering module performs hierarchical clustering on the feature matrix, and uses the cluster IDs of multiple clusters to group those with the same cluster ID into the same dataset, outputting multiple independent clusters of different sizes; the training set selection module selects data from various datasets to form the training set.

[0107] This application embodiment uses multiple clustering and selects training set data based on the cluster IDs of the multiple clusterings. This allows a small training set to cover the data features of a large test set, resulting in low redundancy in the training dataset. It also solves the problems of excessive noise in a single clustering or too much data in a certain cluster.

[0108] This application embodiment also provides a training dataset determination device for a model, which corresponds to the training dataset determination method for the model described above. The steps in the training dataset determination method embodiment for the model described above are also fully applicable to this device embodiment.

[0109] The apparatus for determining the training dataset for this model includes:

[0110] The clustering module is used to perform multiple clustering operations on the data in the original dataset to obtain multiple clustering identifiers for each data point in the original dataset; wherein, each data point in the original dataset is assigned a clustering identifier in each clustering operation.

[0111] The acquisition module is used to acquire a training dataset from the original dataset based on multiple clustering identifiers for each data point in the original dataset, so as to train a large language model based on the training dataset.

[0112] In one embodiment, the acquisition module is specifically used for:

[0113] Based on the multiple clustering identifiers of each data in the original dataset, data with the same multiple clustering identifiers are grouped into the same dataset to obtain multiple datasets;

[0114] The training dataset is formed by selecting data from the multiple datasets.

[0115] In one embodiment, the acquisition module is specifically used for:

[0116] The same number of data points are selected from each dataset to form the training dataset.

[0117] In one embodiment, the clustering module is specifically used for:

[0118] Extract the key features of each data point in the original dataset;

[0119] Based on the key features, the data in the original dataset are clustered.

[0120] In one embodiment, the clustering module is specifically used for:

[0121] Determine the similarity between key features of the data in the original dataset;

[0122] The data with the same similarity are assigned the same clustering identifier.

[0123] In one embodiment, the device further includes:

[0124] The deduplication module is used to deduplicate the original dataset based on key fields.

[0125] This application also provides a model reasoning device, which corresponds to the model reasoning method described above. The steps in the model reasoning method embodiment are also fully applicable to this device embodiment.

[0126] The model inference device includes:

[0127] An input module is used to input the data to be inferred into a large language model, which is used for inference; the training dataset of the large language model is formed by performing multiple clustering operations on the data in the original dataset to obtain multiple clustering identifiers for each data, and then obtaining data from the original dataset based on the multiple clustering identifiers for each data in the original dataset; wherein, each data in the original dataset is assigned a clustering identifier in each clustering operation.

[0128] The reasoning module is used to obtain the reasoning results of the large language model on the data to be reasoned.

[0129] In practical applications, the clustering and acquisition modules described above can be implemented by a processor in an electronic device. Of course, the processor needs to run a computer program stored in memory to perform its functions.

[0130] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above program modules. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the apparatus can be divided into different program modules to complete all or part of the processing described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and will not be described again here.

[0131] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiments of this application, the embodiments of this application also provide an electronic device. Figure 4 This is a schematic diagram of the hardware composition structure of an electronic device provided in an embodiment of this application, such as... Figure 4 As shown, the electronic device includes:

[0132] The communication interface 401 enables information exchange with other devices, such as network devices.

[0133] The processor 402 is connected to the communication interface 401 to enable information interaction with other devices and to execute the methods provided by one or more of the above-mentioned technical solutions when running a computer program. The computer program is stored in the memory 403.

[0134] Of course, in practical applications, the various components in an electronic device are coupled together through a bus system 404. It can be understood that the bus system 404 is used to realize the connection and communication between these components. In addition to the data bus, the bus system also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in... Figure 4 The general designated all buses as Bus System 404.

[0135] The memory 403 in this embodiment is used to store various types of data to support the operation of the computer device. Examples of such data include any computer program used to operate on the electronic device.

[0136] It is understood that memory 403 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM). The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memory.

[0137] The methods disclosed in the embodiments of this application can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory. The processor reads the program from the memory and, in conjunction with its hardware, completes the steps of the aforementioned method.

[0138] Optionally, when the processor 402 executes the program, it implements the corresponding processes implemented by the computer device in the various methods of the embodiments of this application. For the sake of brevity, these will not be described in detail here.

[0139] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a first memory storing a computer program, which can be executed by a processor of a computer device to complete the steps described in the aforementioned method. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0140] In the several embodiments provided in this application, it should be understood that the disclosed apparatus, computer devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0141] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0142] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0143] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0144] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0145] In an exemplary embodiment, this application also provides a computer program product, including a computer program that can be executed by a processor 402 of an electronic device to perform the steps described in the method for determining the training dataset of the model in this application embodiment.

[0146] It should be noted that terms such as "first" and "second" are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0147] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0148] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for determining a training data set of a model, characterized in that, The method includes: The data in the original dataset are clustered multiple times to obtain multiple cluster identifiers for each data point in the original dataset; wherein, each data point in the original dataset is assigned a cluster identifier in each clustering process; Based on multiple clustering identifiers for each data point in the original dataset, a training dataset is obtained from the original dataset to train a large language model.

2. The method of claim 1, wherein, The step of obtaining the training dataset from the original dataset based on multiple clustering identifiers for each data point in the original dataset includes: Based on the multiple clustering identifiers of each data in the original dataset, data with the same multiple clustering identifiers are grouped into the same dataset to obtain multiple datasets; The training dataset is formed by selecting data from the multiple datasets.

3. The method of claim 1, wherein, The process of performing multiple clustering operations on the data in the original dataset includes: Extract the key features of each data point in the original dataset; Based on the key features, the data in the original dataset are clustered.

4. The method of claim 1, wherein, Before clustering the original dataset, the method further includes: The original dataset is deduplicated based on key fields.

5. A model inference method, comprising: The method includes: The data to be inferred is input into a large language model, which is used for inference; the training dataset of the large language model is constructed by performing multiple clustering operations on the data in the original dataset to obtain multiple clustering identifiers for each data, and then extracting data from the original dataset based on the multiple clustering identifiers for each data in the original dataset; wherein, each data in the original dataset is assigned a clustering identifier in each clustering operation. Obtain the reasoning result of the large language model on the data to be reasoned.

6. A model training data set determination apparatus characterized by comprising: include: The clustering module is used to perform multiple clustering operations on the data in the original dataset to obtain multiple clustering identifiers for each data point in the original dataset; wherein, each data point in the original dataset is assigned a clustering identifier in each clustering operation. The acquisition module is used to acquire a training dataset from the original dataset based on multiple clustering identifiers for each data point in the original dataset, so as to train a large language model based on the training dataset.

7. A model inference apparatus characterized by comprising: include: An input module is used to input the data to be reasoned into a large language model, which is used for reasoning. The training dataset of the large language model is constructed by performing multiple clustering operations on the data in the original dataset to obtain multiple clustering identifiers for each data point, and then extracting data from the original dataset based on the multiple clustering identifiers for each data point in the original dataset; wherein, each data point in the original dataset is assigned a clustering identifier in each clustering operation. The reasoning module is used to obtain the reasoning results of the large language model on the data to be reasoned.

8. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4 or the steps of the method according to claim 5.

9. An electronic device, comprising: include: A processor and memory for storing computer programs that can run on the processor, wherein, The processor is configured to execute the steps of the method of any one of claims 1 to 4 or the steps of the method of claim 5 when running the computer program.

10. A computer storage medium having stored thereon a computer program, characterized in that The computer program, which when executed by the processor, implements the steps of the method of any one of claims 1 to 4 or the steps of the method of claim 5.

Images