Large model optimization method and device, equipment and storage medium

Abstract

The invention relates to the technical field of large language models, and discloses a large model optimization method and device, equipment and a storage medium, and the method comprises the steps: obtaining a sampling corpus set; performing data compression on corpus samples in the sampling corpus set based on the data similarity to obtain a data set; and training a large model based on the data set to obtain an optimized large language model. According to the method, data compression is performed according to the data similarity to remove redundant data in the big language model training data, so that the data quality is effectively improved, then model training is performed through the compressed data, and the model training efficiency and the model performance are effectively improved.

Description

Technical Field

[0001] This application relates to the technical field of large language models, and particularly to large model optimization methods, devices, equipment, and storage media. Background Art

[0002] Incremental pre-training (or continued pre-training) of large models has become an important means to improve the performance of basic models. In current practices, problems such as data redundancy and semantic repetition are faced. This challenge not only affects the effectiveness of learning, but also consumes a large amount of computing resources and storage space. When dealing with large-scale text data, duplicate data is likely to cause bias, which in turn leads to the "hallucination" effect - that is, the large model generates inaccurate or illogical outputs. Although existing technologies have shown significant value in their respective fields, challenges still remain in aspects such as how to efficiently remove duplicate data, reduce resource consumption, and enhance the knowledge retention ability during the large model training process.

[0003] The above content is only used to assist in understanding the technical solution of this application, and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a large model optimization method, device, equipment, and storage medium, aiming to solve the technical problem that data redundancy in large-scale text data affects model performance.

[0005] To achieve the above purpose, this application proposes a large model optimization method, and the method includes:

[0006] Obtain a sampled corpus set;

[0007] Perform data compression on the corpus samples in the sampled corpus set based on data similarity to obtain a data set;

[0008] Train a large model based on the data set to obtain an optimized large language model.

[0009] In one embodiment, the performing data compression on the corpus samples in the sampled corpus set based on data similarity to obtain a data set includes:

[0010] Perform data compression on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set;

[0011] Perform parallel data sampling and data merging on the compressed corpus set to obtain a data set.

[0012] In one embodiment, the performing data compression on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set includes at least one of the following:

[0013] Mutually exclusive compression is performed on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set;

[0014] Bilingual quality assessment compression is performed on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set;

[0015] Clustering compression is performed on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set;

[0016] Duplicate removal compression is performed on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set.

[0017] In one embodiment, the mutually exclusive compression is performed on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set, including:

[0018] Calculate the similarity between the corpus sample pairs in the sampled corpus set to obtain the similarity between the corpus sample pairs;

[0019] Construct a similarity matrix based on the similarity between the corpus sample pairs;

[0020] Based on the similarity matrix and a preset similarity threshold, perform mutually exclusive screening on the corpus samples in the sampled corpus set to obtain a compressed corpus set.

[0021] In one embodiment, the mutually exclusive screening based on the similarity matrix and a preset similarity threshold to obtain a compressed corpus set includes:

[0022] Compare the elements in the similarity matrix with the preset similarity threshold to obtain a comparison result;

[0023] When the comparison result is that the element is less than the preset similarity threshold, obtain the index of the element;

[0024] Perform mutually exclusive screening on the corpus samples in the sampled corpus set according to the index of the element to obtain a mutually exclusive set;

[0025] Calculate the compression rate based on the mutually exclusive set to obtain the current compression rate;

[0026] When the current compression rate reaches the target compression rate, use the mutually exclusive set as the compressed corpus set;

[0027] When the current compression rate does not reach the target compression rate, adjust the preset similarity threshold and re-execute the step of comparing the elements in the similarity matrix with the preset similarity threshold to obtain a comparison result.

[0028] In one embodiment, the bilingual quality assessment and compression of corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set includes:

[0029] Perform language detection and classification on the corpus samples in the sampled corpus set to obtain a first corpus sample and a second corpus sample, where the languages of the first corpus sample and the second corpus sample are different;

[0030] Perform quality assessment on the first corpus sample and the second corpus sample respectively to obtain corresponding quality assessment scores;

[0031] Sort the first corpus sample and the second corpus sample based on the quality assessment scores to obtain a sorting result;

[0032] Perform bilingual mixed sampling on the corpus samples in the sampled corpus set according to the sorting result to obtain a compressed corpus set.

[0033] In one embodiment, the clustering compression of corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set includes:

[0034] Perform label prediction on the corpus samples in the sampled corpus set based on a preset clustering model to obtain clustering labels corresponding to each corpus sample;

[0035] Cluster the corpus samples in the sampled corpus set according to the clustering labels to obtain a clustered corpus set;

[0036] Perform similarity calculation based on the clustered corpus set to obtain the similarity between the corpus samples corresponding to each category;

[0037] Screen the corpus samples within the cluster based on the similarity to obtain a compressed corpus set.

[0038] In one embodiment, the duplicate removal compression of corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set includes:

[0039] Vectorize the corpus samples in the sampled corpus set, and use any corpus sample vector in the sampled corpus set as a target query;

[0040] Calculate the similarity between the target query and other corpus sample vectors in the sampled corpus set respectively;

[0041] Remove other corpus sample vectors with a similarity greater than or equal to a preset similarity threshold as redundant corpus sample vectors to obtain a compressed corpus set.

[0042] In one embodiment, the parallel data sampling and data merging of the compressed corpus set to obtain a data set includes:

[0043] Dividing the compressed corpus set into multiple data blocks;

[0044] Performing parallel data sampling on the data blocks to obtain multiple corpus subsets;

[0045] Performing data merging on the corpus subsets to obtain a data set.

[0046] In one embodiment, the obtaining of the sampled corpus set includes:

[0047] Randomly selecting corpus data from a corpus based on a preset ratio to obtain original corpus, where the corpus includes multiple corpus data sets;

[0048] Performing word segmentation on the original corpus to obtain the word-segmented original corpus;

[0049] Performing data sampling based on the word-segmented original corpus to obtain a sampled corpus set.

[0050] In one embodiment, the training of a large model based on the data set to obtain an optimized large language model includes:

[0051] Training a large model based on the data set to obtain a trained large model;

[0052] Performing performance evaluation on the trained large model to obtain performance metrics;

[0053] When the performance metrics reach the corresponding metric thresholds, taking the trained large model as the optimized large language model;

[0054] When the performance metrics do not reach the corresponding metric thresholds, adjusting the training parameters of the large model and re-executing the step of training the large model based on the data set.

[0055] In addition, to achieve the above object, the present application also proposes a large model optimization device, which includes:

[0056] An acquisition module for acquiring a sampled corpus set;

[0057] A compression module for compressing the corpus samples in the sampled corpus set based on data similarity to obtain a data set;

[0058] A training module for training a large model based on the data set to obtain an optimized large language model.

[0059] In one embodiment, the compression module is further configured to perform data compression on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set;

[0060] Perform parallel data sampling and data merging on the compressed corpus set to obtain a data set.

[0061] In one embodiment, the compression module is further configured to perform mutually exclusive compression on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set;

[0062] Perform bilingual quality assessment compression on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set;

[0063] Perform clustering compression on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set;

[0064] Perform deduplication compression on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set.

[0065] In one embodiment, the compression module is further configured to calculate the similarity between the corpus sample pairs in the sampled corpus set to obtain the similarity between the corpus sample pairs;

[0066] Construct a similarity matrix based on the similarity between the corpus sample pairs;

[0067] Perform mutually exclusive screening on the corpus samples in the sampled corpus set based on the similarity matrix and a preset similarity threshold to obtain a compressed corpus set.

[0068] In one embodiment, the compression module is further configured to compare the elements in the similarity matrix with a preset similarity threshold to obtain a comparison result;

[0069] When the comparison result is that the element is less than the preset similarity threshold, obtain the index of the element;

[0070] Perform mutually exclusive screening on the corpus samples in the sampled corpus set according to the index of the element to obtain a mutually exclusive set;

[0071] Calculate the compression ratio based on the mutually exclusive set to obtain the current compression ratio;

[0072] When the current compression ratio reaches the target compression ratio, use the mutually exclusive set as the compressed corpus set;

[0073] In the case that the current compression ratio does not reach the target compression ratio, adjust the preset similarity threshold, and re - execute the step of comparing the elements in the similarity matrix with the preset similarity threshold to obtain a comparison result.

[0074] In one embodiment, the compression module is further configured to perform language detection and classification on the corpus samples in the sampled corpus set to obtain a first corpus sample and a second corpus sample, where the languages of the first corpus sample and the second corpus sample are different;

[0075] Perform quality assessment on the first corpus sample and the second corpus sample respectively to obtain corresponding quality assessment scores;

[0076] Sort the first corpus sample and the second corpus sample based on the quality assessment scores to obtain a sorting result;

[0077] Perform bilingual mixed sampling on the corpus samples in the sampled corpus set according to the sorting result to obtain a compressed corpus set.

[0078] In one embodiment, the compression module is further configured to perform label prediction on the corpus samples in the sampled corpus set based on a preset clustering model to obtain clustering labels corresponding to each corpus sample;

[0079] Cluster the corpus samples in the sampled corpus set according to the clustering labels to obtain a clustered corpus set;

[0080] Calculate the similarity between the corpus samples corresponding to each category based on the clustered corpus set;

[0081] Filter the corpus samples within the cluster based on the similarity to obtain a compressed corpus set.

[0082] In addition, to achieve the above - mentioned purpose, the present application also proposes a large - model optimization device, where the device includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the large - model optimization method as described above.

[0083] In addition, to achieve the above - mentioned purpose, the present application also proposes a storage medium, where the storage medium is a computer - readable storage medium, and a computer program is stored on the storage medium, and when the computer program is executed by a processor, it implements the steps of the large - model optimization method as described above.

[0084] In addition, to achieve the above - mentioned purpose, the present application also provides a computer program product, where the computer program product includes a computer program, and when the computer program is executed by a processor, it implements the steps of the large - model optimization method as described above.

[0085] The present application provides a large model optimization method. First, a sampled corpus set is obtained; the corpus samples in the sampled corpus set are data-compressed based on data similarity to obtain a data set; and the large model is trained based on the data set to obtain an optimized large language model. In this way, redundant data in the training data of the large language model is removed by data compression according to data similarity, effectively improving the data quality. Then, model training is performed using the compressed data, effectively improving the model training efficiency and model performance. Description of the Drawings

[0086] The drawings herein are incorporated into and constitute a part of this specification, showing embodiments consistent with the present application and, together with the specification, are used to explain the principles of the present application.

[0087] To more clearly illustrate the technical solutions in the embodiments of the present application or in the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative efforts.

[0088] Figure 1 It is a schematic flowchart provided for the first embodiment of the large model optimization method of the present application;

[0089] Figure 2 It is a schematic flowchart provided for the second embodiment of the large model optimization method of the present application;

[0090] Figure 3 It is a schematic diagram showing the sample division effect with different clustering parameters provided for one embodiment of the large model optimization method of the present application;

[0091] Figure 4 It is a schematic flowchart provided for one embodiment of the large model optimization method of the present application;

[0092] Figure 5 It is a schematic diagram of the module structure of the large model optimization device according to the embodiment of the present application;

[0093] Figure 6 It is a schematic diagram of the device structure of the hardware operating environment involved in the large model optimization method according to the embodiment of the present application.

[0094] The realization of the object, functional features, and advantages of the present application will be further described with reference to the embodiments and the drawings. Detailed Embodiments

[0095] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0096] To better understand the technical solution of this application, the following will be described in detail in conjunction with the accompanying drawings of the specification and specific implementation manners.

[0097] The main solution of the embodiment of this application is: obtain a sampled corpus set; perform data compression on the corpus samples in the sampled corpus set based on data similarity to obtain a data set; and train a large model based on the data set to obtain an optimized large language model.

[0098] Incremental pre-training (or continued pre-training) of large models has become an important means to improve the performance of the basic model. In current practices, problems of data redundancy and semantic repetition are faced. This challenge not only affects the effectiveness of learning, but also consumes a large amount of computing resources and storage space. When dealing with large-scale text data, duplicate data is prone to cause bias, which in turn leads to the "hallucination" effect - that is, the large model generates inaccurate or illogical outputs.

[0099] This application effectively improves the data quality by removing redundant data in the training data of the large language model according to data similarity, and then effectively improves the model training efficiency and model performance by training the model with the compressed data.

[0100] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device, a large model optimization device, etc. that can implement the above functions. Hereinafter, taking the large model optimization device as an example, this embodiment and the following embodiments will be described.

[0101] Based on this, the embodiment of this application provides a large model optimization method, which is applied to a control host, referring to Figure 1 , Figure 1 is a schematic flowchart of the first embodiment of the large model optimization method of this application.

[0102] In this embodiment, the large model optimization method includes steps S10 to S30:

[0103] Step S10, obtain a sampled corpus set.

[0104] It should be noted that the sampled corpus set refers to a corpus data set obtained by proportioning and sampling the original corpus data, which contains various corpus samples for large model training. These corpus samples can be in the form of text, dialogue, article fragments, etc., and this embodiment does not make specific restrictions on this. The construction of the sampled corpus set is to ensure that the large model can learn from diverse data, thereby improving the generalization ability and performance of the model.

[0105] In a feasible implementation, step S10 may include: randomly selecting corpus data from a corpus based on a preset ratio to obtain raw corpus, where the corpus includes multiple corpus data sets; segmenting the raw corpus to obtain the segmented raw corpus; and performing data sampling based on the segmented raw corpus to obtain a sampled corpus set.

[0106] It should be noted that a corpus is a database storing a large amount of corpus data, which can be used for the training and optimization of large models. In this implementation, the corpus can be regarded as an original data set, which contains multiple common corpus data sets. For example, the corpus can include more than 60 common corpus data sets, with a total of up to 173TB. These data are proportioned and sampled according to a preset ratio to achieve the required data scale.

[0107] It should be understood that the common corpus data sets may include more than 60 types such as government, wanjuan_zh_patent_high_quality, 360wenku, 360wenda, 360tiku, etc., with a total capacity of up to 173TB. To optimize the training process of large language models, it is necessary to sample a specific scale of data subset, such as 20 billion tokens, from these massive data to ensure the best balance between training efficiency and effect.

[0108] It is worth noting that the data proportions and data set sizes of different corpus data sets are different. For example, the data proportions of data such as government, wanjuan_zh_patent_high_quality, 360wenku, 360wenda, and 360tiku are 0.00277, 0.00206, 0.10715, 0.0274573, and 0.0469119 respectively, and the data set sizes are 20G, 17G, 5.4T, 551G, and 74G respectively. Among them, government is a text data set related to the government, including corpus related to government policies, legal documents, government reports, etc.; wanjuan_zh_patent_high_quality is a Chinese patent data set, including patent text corpus; 360wenku is a shared Chinese document data set, such as academic papers, technical reports, e-books, etc. corpus; 360wenda is a Chinese Q&A data set, including various questions and answers, etc. corpus; 360tiku is an educational resource data set, including test questions, courseware, and teaching plans, etc. corpus.

[0109] It should be understood that by sampling from the corpus according to a preset ratio, a customized dataset of the expected scale can be generated, that is, the original corpus. By tokenizing the text in the original corpus and converting the text into a series of token ID sequences, not only the speed of subsequent model training is improved, but also the overall training quality is enhanced. Data sampling is performed on the tokenized original corpus to select representative and diverse samples from the huge corpus data and construct a sampled corpus set. In this way, the existing large-scale corpus resources can be efficiently utilized to accelerate and optimize the learning process of large language models.

[0110] Step S20: Compress the corpus samples in the sampled corpus set based on data similarity to obtain a dataset.

[0111] It should be noted that data compression aims to reduce data redundancy and increase knowledge density, improve data quality, and thus optimize the model training process. In this embodiment, data compression is performed based on data similarity, that is, removing duplicate or highly similar data based on data similarity and retaining representative and unique samples.

[0112] It should be understood that to solve the problem of global semantic compression and minimize data repetition, in this embodiment, a global sample mutually exclusive data compression algorithm can be used for data compression, and bilingual-QuRating, kmenas clustering compression method based on adaptive sampling, FAISS similarity deduplication compression algorithm, etc. can also be used for data compression. This embodiment does not make specific limitations on this.

[0113] Step S30: Train a large model based on the dataset to obtain an optimized large language model.

[0114] It should be noted that the data after compression processing will be used in the pre-training process of the large model, and the trained large model will be evaluated on 14 standard test sets (benchmarks) to verify the improvement of its performance.

[0115] It can be understood that the large model in this embodiment can be the LLaMA3-8B base model, and the nexttoken prediction method is used for training. The core technologies mainly focus on parameter adjustment, performance optimization, and cost estimation.

[0116] In a feasible implementation, step S30 may include: training a large model based on the dataset to obtain a trained large model; evaluating the performance of the trained large model to obtain performance metrics; when the performance metrics reach the corresponding metric thresholds, using the trained large model as the optimized large language model; when the performance metrics do not reach the corresponding metric thresholds, adjusting the training parameters of the large model and re-executing the step of training the large model based on the dataset.

[0117] It should be noted that in this implementation, the hardware configuration for large model pre-training is a cluster composed of 8 H-800 servers (each with 8 H800 GPUs and 2T of memory), and it takes 24 hours to train 200B tokens. If 4 H-800 servers are used, the training time will double.

[0118] It can be understood that when the large model is the LLaMA3-8B base model, the dataset can be of two scales, one is the data containing 50B tokens, and the other is the data containing 200B tokens. The initial loss value is 2.2, and the global batch size is set to 1024. The training goal is achieved through technical means such as gradient accumulation, data parallelism, and model parallelism.

[0119] For the 50B token dataset, training is completed after 11,920 iterations, and the loss value is reduced to 1.73. It takes about 25 hours when using 8 computing nodes. In contrast, the 200B token dataset requires 47,683 iterations to reach a similar loss value (1.69). During the training process with the same number of steps, the loss values of the two are basically the same. The result of using only 1 / 4 of the data volume compared with the 200B token data shows that the average score on the knowledge-based dataset is reduced by 0.06 points, but it is increased by 0.63 points on all types of datasets.

[0120] To compare the effects of different data volumes and training parameters on model performance, quadrupling the learning rate or reducing the batch size of the 50B token dataset to one-fourth of the original can alleviate the problem of the difference in the total number of training steps or the update amplitude.

[0121] Furthermore, additional pre-training was performed on the randomly sampled 50B data and compared with the selected 50B data. The results show that when using the LLaMA3-8B model, the average score on the knowledge-based dataset is increased by 1.57 points, and all types of datasets are increased by 0.5 points.

[0122] The above test results not only reveal the impact on training efficiency under different scales and parameter settings, but also demonstrate that the performance of the model on specific tasks can be significantly improved by carefully selecting training data.

[0123] Optionally, the large model can also be a small Transformer model of scratch-1.3B. To evaluate the data compression scheme without interference and improve the experimental efficiency during this process, the small Transformer model of scratch-1.3B is trained and analyzed from scratch. This model is based on the architecture of Llama 3, and the parameter scale is controlled to about 1.3B by adjusting the number of layers and vector dimensions. 50B tokens of data are randomly selected from a 200B-token dataset as the initial training dataset v1. In addition, another set of different 50B data sets v2 are carefully selected based on the 200B dataset by applying a deduplication strategy. The effectiveness of the data compression scheme is evaluated by comparing the effects of pre-training the model using these two different datasets.

[0124] Specifically, the model structure and parameters are as follows: the basic architecture is a Transformer layer based on Llama 3; the number of parameters is controlled at 1.3B (i.e., scratch-1.3B). The pre-training parameters are: the learning rate is set to 2e-4, and the cosine annealing learning rate scheduling strategy is adopted. The final loss value during training on the carefully selected dataset v2 is approximately 2.4, while the final loss obtained from training on the randomly selected dataset v1 is approximately 2.6; when attempting to adjust the learning rate to 0.01, it is found that the training effect of the model does not change significantly, and the final loss remains unchanged.

[0125] The effectiveness of different compression schemes is evaluated by comparing the differences in the training effects of using the deduplicated and carefully selected dataset (v2) and the randomly selected dataset (v1). The results show that under the same learning rate setting, choosing the optimized deduplicated dataset can enable the model to reach a lower loss faster, showing better convergence performance and a more efficient learning process. Through this experimental design, not only can the effectiveness of specific data processing strategies in the training of small Transformer models be verified, but the overall experimental efficiency can also be improved to a certain extent.

[0126] It is worth noting that during the model training process, the performance metrics can be perplexity and loss value. Perplexity reflects the prediction uncertainty of the model for test data, and the lower the value, the stronger the prediction ability of the model. The loss value quantifies the gap between the model's prediction result and the true label. Similarly, a lower loss value means that the model's prediction is more accurate.

[0127] Optionally, accuracy is adopted in the downstream task evaluation. The downstream task evaluation includes: 1) Knowledge ability evaluation: use knowledge-based datasets (mmlu, cmmlu, ceval, gpqa_diamond, arc-c) to evaluate the average value; 2) General ability evaluation: use all datasets to evaluate the average value; 3) Experimental settings: full data V.S. random sampling V.S. selected data compression.

[0128] When reducing the volume of the pre-training data collection to 25% of the original while ensuring that the performance of the model remains basically the same as before compression. The experimental results show that: (1) Compressing the volume of the pre-training data to 25% (from 200B tokens to 50B tokens) saves 3 / 4 of the model training time. (2) Continue pre-training, 200B data V.S. 50B selected data. The average score of Llama3-8b on the knowledge-based datasets decreased by 0.06 points, and the average score on all datasets increased by 0.63 points. (3) Continue pre-training, 50B randomly sampled data V.S. 50B selected data. The average score of Llama3-8b on the knowledge-based datasets increased by 1.57 points, and the average score on all datasets increased by 0.5 points. (4) Pre-train from scratch: 50B randomly sampled data V.S. 50B selected data. Randomly initialize a large model of 1.5B and pre-train from scratch Scratch-1.5B. The average score of the knowledge-based datasets increased by 0.66 points, and the average score on all datasets increased by 0.11 points.

[0129] This embodiment provides a large model optimization method. In this embodiment, first, a sampled corpus collection is obtained; the corpus samples in the sampled corpus collection are data-compressed based on data similarity to obtain a dataset; and the large model is trained based on the dataset to obtain an optimized large language model. In the above manner, redundant data in the training data of the large language model is removed by data compression according to data similarity, effectively improving the data quality. Furthermore, the model training efficiency and model performance are effectively improved by training the model with the compressed data.

[0130] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar content as in the above-mentioned first embodiment can be referred to the above introduction and will not be repeated hereinafter. On this basis, please refer to Figure 2 that the step S20 further includes steps S201-S202:

[0131] Step S201, data-compress the corpus samples in the sampled corpus collection based on data similarity to obtain a compressed corpus collection.

[0132] It should be noted that, in order to solve the problem of global semantic compression and minimize data redundancy, the global sample mutually exclusive data compression algorithm can be used for data compression in this embodiment.

[0133] Optionally, data compression in this embodiment can also adopt methods such as bilingual - QuRating, kmenas clustering compression method based on adaptive sampling, FAISS similarity deduplication compression algorithm, etc., and this embodiment does not make specific limitations on this.

[0134] In a feasible embodiment, step S201 may include at least one of steps A10 to A13:

[0135] Step A10: Mutually exclusive compression is performed on the corpus samples in the sampled corpus set based on data similarity to obtain the compressed corpus set.

[0136] It should be noted that this embodiment uses the global sample mutually exclusive data compression algorithm for mutually exclusive compression. This algorithm calculates the similarity between corpus samples, identifies and removes highly similar redundant samples, thereby retaining the most representative data. This method can not only significantly reduce the size of the data set, but also effectively avoid the overfitting phenomenon in the model training process and improve the generalization ability of the model.

[0137] In a feasible embodiment, step A10 may include: calculating the similarity between corpus sample pairs in the sampled corpus set to obtain the similarity between the corpus sample pairs; constructing a similarity matrix based on the similarity between the corpus sample pairs; and performing mutually exclusive screening on the corpus samples in the sampled corpus set based on the similarity matrix and a preset similarity threshold to obtain the compressed corpus set.

[0138] It should be noted that existing data deduplication methods rely on the FAISS database. However, it faces the problem of being unable to adaptively search for the optimal threshold for different subsets and cannot achieve global sample mutual exclusivity. In this embodiment, global sample mutual exclusion compression is achieved through an adaptive compression threshold search and sample screening algorithm. In terms of computing, the utilization efficiency of the GPU is significantly improved by using a data partitioning algorithm.

[0139] It can be understood that, in order to improve data compression efficiency, before constructing the similarity matrix, the sampled corpus set can be preliminarily divided, and the corpus samples are initially divided into several subsets. Then, similarity calculation is performed within each subset, a similarity matrix within the subset is constructed, and mutually exclusive screening is performed according to the preset similarity threshold. In this way, each subset can retain the most representative corpus samples while removing redundancy, and this embodiment does not make specific limitations on this.

[0140] It is worth noting that the similarity matrix is a two-dimensional array, in which each element represents the similarity between two corpus samples. The similarity can be calculated using algorithms such as cosine similarity, Jaccard similarity or edit distance, and the specific choice depends on the characteristics of the corpus samples and the application scenario. In this embodiment, in order to more accurately measure the semantic similarity between corpus samples, a semantic similarity calculation method based on the BERT model can be used to capture the deep semantic relationship between corpus samples, so as to more accurately identify and remove redundant samples.

[0141] It should be understood that after constructing the similarity matrix, by setting a preset similarity threshold, the corpus samples can be mutually exclusive screened. That is, for corpus sample pairs whose similarity exceeds the threshold, only one of them is retained as a representative sample, thereby achieving data compression. This mutually exclusive screening method can effectively reduce redundant information in the data set, improve data quality, and thus improve the efficiency and performance of model training. Optionally, the selection of the preset similarity threshold is crucial, which directly affects the effect of data compression and the final result of model training. In order to find the optimal preset similarity threshold, this embodiment further proposes an adaptive compression threshold search algorithm. The algorithm automatically selects the best preset similarity threshold by iteratively adjusting the preset similarity threshold and evaluating the model training effect of the compressed data set under different thresholds.

[0142] In a feasible implementation manner, the mutually exclusive screening based on the similarity matrix and the preset similarity threshold to obtain a compressed corpus set includes: comparing the elements in the similarity matrix with the preset similarity threshold to obtain a comparison result; when the comparison result is that the element is less than the preset similarity threshold, obtaining the index of the element; performing mutually exclusive screening on the corpus samples in the sampled corpus set according to the index of the element to obtain a mutually exclusive set; calculating the compression rate based on the mutually exclusive set to obtain the current compression rate; when the current compression rate reaches the target compression rate, using the mutually exclusive set as the compressed corpus set; when the current compression rate does not reach the target compression rate, adjusting the preset similarity threshold, and re-executing the step of comparing the elements in the similarity matrix with the preset similarity threshold to obtain the comparison result.

[0143] It should be noted that in this embodiment, an exclusive set extraction algorithm is used to achieve efficient screening and compression of corpus samples. The goal of the exclusive set extraction algorithm is to refine the data set by eliminating rows (or columns in a symmetric matrix) corresponding to elements with pairwise similarity lower than a predefined threshold, so as to extract an exclusive set of samples. Utilizing the properties of the symmetric matrix, this algorithm simplifies the calculation and effectively processes the input sample set represented as a two-dimensional matrix. This algorithm accepts two parameters: the matrix itself containing the samples and the similarity threshold. It outputs two key values: the compression ratio "ratio" and the set of retained row indices "keep set". This algorithm first constructs a similarity matrix that captures the pairwise similarity between samples. Subsequently, it filters out elements with similarity lower than the specified threshold. This is achieved by comparing each element of the similarity matrix with the threshold and using the operation "(inner < Threshold).nonzero()" to determine the positions of the elements that do not meet the criteria. Here, "inner" represents the pre-computed similarity matrix, and "nonzero()" returns the indices of the elements that do not exceed the threshold. To determine the retained rows, this algorithm only focuses on the row indices (recognizing that column indices are redundant in a symmetric matrix). It extracts the first dimension of the index array, converts it into a set "keep set" to eliminate duplicates, and calculates the compression ratio based on the number of retained rows. This method ensures that only the most representative exclusive samples are retained, thus promoting effective data set compression while retaining key information.

[0144] The implementation of the exclusive set extraction algorithm is as follows:

[0145] Input: matrix, threshold

[0146] Output: ratio, keep_set

[0147] 1: inner = matrix * matrix^T

[0148] 2: keeps = (inner < threshold).nonzero()

[0149] 3: keeps_set = set(keeps[:, 0])

[0150] 4: return len(keeps_set) / batch, keeps_set

[0151] It is worth noting that by using an adaptive compression threshold search algorithm to determine the optimal preset similarity threshold, the effect of data compression can be significantly improved, and the performance of model training can be further optimized.

[0152] Optionally, the adaptive compression threshold search algorithm adopts a binary search method. This algorithm gradually approaches the optimal preset similarity threshold by continuously adjusting the preset similarity threshold and evaluating the model training effect of the compressed dataset after each adjustment. The core of the binary search method is that it can halve the search range in each iteration, thus quickly converging to the optimal solution. This efficient search strategy enables the adaptive compression threshold search algorithm to quickly find the best preset similarity threshold with limited computing resources, thereby optimizing the data compression effect and model training performance.

[0153] In specific implementation, an initial preset similarity threshold range can be set first, and then the binary search method is used to search within this range. In each iteration, the midpoint of the current search range is calculated as the new preset similarity threshold, and the sampled corpus set is screened and compressed mutually exclusively based on this threshold. Subsequently, the compressed dataset is used for model training, and the model training effect is evaluated. According to the evaluation results, the search range is adjusted, and the next iteration is continued until the preset search termination conditions are met, such as the search range being less than the preset threshold or the improvement range of the model training effect being less than the preset threshold, etc. Finally, the preset similarity threshold obtained in the last iteration is used as the optimal solution for subsequent data compression and model training processes.

[0154] It should be noted that the binary search method is used to adaptively determine the optimal threshold within a specified subset of the given sample matrix, where the subset is defined by the "start" and "end" indices, such that the resulting ratio of the retained elements is close to a predefined target ratio ("target ratio"). The algorithm initializes the "keeps set" variable to accumulate the row indices identified as below the current threshold by the "removesim" function in each iteration. The core of the algorithm lies in the binary search loop (lines 2 - 10), which iteratively narrows the search interval until its width is below the predefined "precision" threshold, thus ensuring high precision. In each iteration, the midpoint ("mid") of the current interval is used as the candidate threshold. By applying this threshold to the subset of the matrix ("matrix[start:end,]") through the "remove sim" function, the algorithm calculates the proportion of elements ("keep_portion") below the threshold and the corresponding index set. Based on the comparison between the "keep portion" and the "target ratio", the algorithm dynamically adjusts the boundaries ("left" and "right") of the search interval. After reaching the termination criterion, i.e., the interval width is less than "precision", the left boundary ("left") is adopted as the threshold closest to but not exceeding "target_ratio", following the inherent property of binary search. Subsequently, the algorithm re - calls the "remove_sim" function with this final threshold to obtain the final index set "final_keeps_set". To prepare the output, the algorithm sorts the indices in "final_keeps_set" (line 12) to maintain consistency and facilitate subsequent processing. Given that the indices are related to the subset, the algorithm converts them to absolute indices by offsetting each index by the "start" value, thus reflecting their positions in the entire matrix. Finally, the algorithm returns the sorted absolute index set "sorted_keep_set", which represents the optimal row selection that meets the target ratio criterion.

[0155] The implementation of the adaptive compression threshold search algorithm is as follows:

[0156] Input: matrix, start, end, left, right, precision, target_ratio

[0157] Output: sorted_keep_set

[0158] 1: keeps_set = None

[0159] 2: while left + precision < right do

[0160] 3: mid = left + (right - left) / 2

[0161] 4: keep_portion, keeps_set = remove_sim(matrix[start:end,], threshold = mid)

[0162] 5: if keep_portion < target_ratio then

[0163] 6: left = mid

[0164] 7: else

[0165] 8: right = mid

[0166] 9: end if

[0167] 10: end while

[0168] 11: _, final_keeps_set = remove_sim(matrix[start:end,], threshold = left)

[0169] 12: sorted_keep_set = sorted(final_keeps_set) + start

[0170] 13: return sorted_keep_set

[0171] It should be noted that by adopting the above-mentioned adaptive compression threshold search algorithm and mutually exclusive set extraction algorithm, efficient compression and optimization of the sampled corpus set can be achieved, thereby improving the data quality and model training efficiency. This optimization method can not only significantly reduce the size and redundant information of the data set, but also effectively avoid the overfitting phenomenon during the model training process, and improve the generalization ability and prediction accuracy of the model.

[0172] Step A11: Based on data similarity, perform bilingual quality assessment and compression on the corpus samples in the sampled corpus set to obtain the compressed corpus set.

[0173] It should be noted that in this embodiment, the bilingual-QuRating data compression method is adopted for data compression. The bilingual-QuRating data compression method combines bilingual alignment and a scoring mechanism. By performing bilingual alignment processing on the corpus samples, giving a score according to the alignment quality, and then selecting the corpus samples according to the score level. This method helps to ensure the diversity and quality of the data set and further improve the performance of the model in cross-language tasks.

[0174] It is understandable that the original Qurating model was mainly trained in English, and its scoring for Chinese is generally lower than that for English. Therefore, there is a deviation in the proportion of Chinese and English corpora. In response to the above problem, in this embodiment, based on the data partitioning mechanism for bilingual classification modeling, the problem of unbalanced sampling ratios of Chinese and English is solved. The overall workflow is divided into data source / language detection and classification, data partitioning, parallel computing, bilingual independent scoring and ranking, and bilingual mixed sampling mechanism. In addition, the original model is limited by the single-machine operation mode and cannot be used to process large-scale data. In this embodiment, by combining the data partitioning method, the model is adapted to scenarios that can process large-scale data, resulting in a significant improvement in the overall processing efficiency.

[0175] In a feasible embodiment, step A11 may include: performing language detection and classification on the corpus samples in the sampled corpus set to obtain a first corpus sample and a second corpus sample, where the languages of the first corpus sample and the second corpus sample are different; performing quality assessment on the first corpus sample and the second corpus sample respectively to obtain corresponding quality assessment scores; sorting the first corpus sample and the second corpus sample based on the quality assessment scores to obtain a sorting result; and performing bilingual mixed sampling on the corpus samples in the sampled corpus set according to the sorting result to obtain a compressed corpus set.

[0176] It should be noted that the first corpus sample and the second corpus sample respectively represent corpus data in different languages. For example, Chinese corpus and English corpus. By independently scoring and ranking the Chinese and English corpora, it can be ensured that the quality of each language's corpus is fully considered during the mixed sampling process. In the bilingual mixed sampling stage, different sampling ratios or priorities can be set according to the sorting result to correct the deviation in the Chinese and English corpus ratio of the original Qurating model. This method can not only maintain the diversity of the dataset but also ensure the balanced distribution of Chinese and English corpora, thereby improving the generalization ability and performance of the model in cross-language tasks. In addition, by combining the data partitioning method and parallel computing technology, the processing efficiency and scale adaptability of the bilingual-QuRating data compression method can be further improved, enabling it to be better applied to the compression and optimization of large-scale multilingual datasets.

[0177] It is understandable that the language detection and classification of corpus samples can be achieved through a classifier, which can be trained based on machine learning algorithms and can accurately identify the language type of corpus samples. The formula for language detection and classification is:

[0178] X en ,X zh =F lang (X)

[0179] Among them, X is all samples, en represents English, zh represents Chinese, and Z zh is the first corpus sample, and Z en is the second corpus sample, and F lang (X) is a classifier that divides samples into Chinese and English.

[0180] The quality evaluation score of the first corpus sample is:

[0181] Y zh = Rating(X 2h )

[0182] Among them, Y zh is the quality evaluation score of the first corpus sample, and Rating() represents the scoring model.

[0183] The quality evaluation score of the second corpus sample is:

[0184] Y en = Rating(X en )

[0185] Among them, Y en is the quality evaluation score of the second corpus sample, and Rating() represents the scoring model.

[0186] It should be noted that during the bilingual mixed sampling process, by setting a reasonable sampling ratio or priority, the distribution of Chinese and English corpora in the dataset can be effectively balanced, thus solving the imbalance problem of the original Qurating model in the Chinese-English ratio. The sampling strategy can be dynamically adjusted according to the quality evaluation scores of the Chinese and English corpora. For example, when the quality of the Chinese corpus is generally high, the sampling ratio of the Chinese corpus can be appropriately increased; conversely, when the quality of the English corpus is more advantageous, the sampling priority of the English corpus can be increased. This flexible sampling mechanism not only helps to maintain the diversity and quality of the dataset, but also significantly improves the generalization ability and prediction accuracy of the model in cross-language tasks.

[0187] Optionally, in order to optimize the performance of the bilingual-QuRating data compression method, machine learning and deep learning technologies can be combined to iteratively optimize the scoring model and the sampling strategy. By continuously introducing new corpus data and training samples, the accuracy and robustness of the scoring model can be continuously improved. At the same time, by fine-tuning the sampling strategy, the effect and efficiency of data compression can be further improved. This method of continuous optimization and improvement helps to ensure that the bilingual-QuRating data compression method can adapt to the changing data environment and task requirements, thus providing users with a more efficient and reliable data compression and optimization solution.

[0188] Step A12: Cluster and compress the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set.

[0189] It should be noted that in this embodiment, the kmeans clustering compression method based on adaptive sampling is used for data compression. This method performs clustering analysis on the corpus samples, groups similar corpus samples into one category, and only retains the representative samples of each category, thereby achieving data compression.

[0190] It should be understood that during the clustering process, first calculate the similarity between corpus samples, and then construct a clustering tree or use other clustering algorithms to group the corpus samples according to the similarity. By setting the number of clusters or the clustering threshold, the size of the compressed data set can be controlled. After obtaining the clustering result, select one or more of the most representative samples from each cluster as the retained samples, and the remaining samples are deleted or merged. This method can retain the key information and diversity of the data set while significantly reducing the size and redundant information of the data set.

[0191] It is worth noting that in the clustering compression stage, an adaptive sampling strategy can be combined to dynamically adjust the sampling ratio and priority according to the clustering result and data distribution, thereby further improving the effect and efficiency of data compression. The core of adaptive sampling is that it can intelligently select representative samples according to the clustering tightness and data distribution characteristics. In areas where the clustering is relatively tight, the sampling quantity can be appropriately reduced because the information redundancy provided by similar samples is relatively high; while in areas where the clustering is relatively dispersed, the sampling quantity should be increased to ensure the diversity and integrity of the data set.

[0192] In a feasible embodiment, step A12 may include: predicting the clustering labels of the corpus samples in the sampled corpus set based on a preset clustering model to obtain the clustering labels corresponding to each corpus sample; clustering the corpus samples in the sampled corpus set according to the clustering labels to obtain a clustered corpus set; calculating the similarity between the corpus samples corresponding to each category based on the clustered corpus set; and screening the corpus samples within the cluster based on the similarity to obtain a compressed corpus set.

[0193] It should be noted that by randomly sampling a certain proportion of samples, training the kmeans clustering model, then performing the operation of predicting labels on the test samples, and performing adaptive sampling. Existing clustering methods only select the clustering parameters to train the model according to experience, while in this embodiment, after predicting the clustering labels, the function of parameter adjustment of adaptive sampling is introduced, and an optimized clustering scheme is realized by combining clustering dimensionality reduction visualization, as Figure 3 shown Figure 3 is a schematic diagram of the sample division effect for different clustering parameters.

[0194] It is understandable that the preset clustering model can be a kmeans clustering model, which gradually approaches the optimal clustering effect by iteratively optimizing the clustering parameters and sampling strategies. In each iteration, the model first predicts the labels of the corpus samples according to the current clustering parameters, and then clusters the samples according to the prediction results. Next, the model calculates the similarity between the samples within each cluster and filters the samples according to the similarity, retaining the most representative samples. By continuously adjusting the clustering parameters and sampling strategies, the model can gradually optimize the clustering effect and achieve efficient data compression. In addition, combined with the clustering dimensionality reduction visualization technology, the clustering results and data distribution characteristics can be intuitively displayed, providing strong support for further optimizing the clustering scheme. This method can not only improve the efficiency and quality of data compression, but also provide users with a more intuitive and visual data compression effect evaluation.

[0195] It is worth noting that the formula for the clustering label is:

[0196] Y = kmeans(X, m)

[0197] Among them, X = [x1, x2, x3,..., xn] represents the set composed of all samples, Y = [y1, y2, y3,..., yn] represents the label corresponding to each sample, and m represents the maximum number of clustering categories.

[0198] The formula for adjusting the parameters of adaptive sampling is:

[0199]

[0200] Among them, F() represents the adaptive sampling function, and Y δ represents the control of the number of samples sampled under each category.

[0201] Step A13: Dedup and compress the corpus samples in the sampled corpus set based on data similarity to obtain the compressed corpus set.

[0202] It should be noted that in this embodiment, a compression method based on FAISS similarity deduplication is adopted for data compression. This method uses the FAISS (Facebook AI Similarity Search) library to achieve efficient similarity calculation and deduplication operations. FAISS is a library developed by Facebook, specifically for handling large-scale vector similarity search and clustering problems. During the data compression process, the feature vectors of the corpus samples are first calculated and these vectors are input into the FAISS library. Then, using the similarity search function of FAISS, similar corpus samples are found. To improve the deduplication efficiency, a similarity threshold can be set. Only when the similarity between two samples exceeds this threshold are they considered duplicates. After identifying duplicate samples, only one of them is retained as a representative sample, and the remaining samples are deleted. This method can significantly reduce the redundant information in the dataset while retaining the key information. It is worth noting that during the deduplication process, domain knowledge and context information can also be combined to further optimize the deduplication strategy. For example, in certain specific domains, even if the text content of two samples is not exactly the same, but if the information and meaning they convey are similar, they can also be regarded as duplicate samples for deduplication. This deduplication strategy combining domain knowledge can further improve the effect and accuracy of data compression. By adopting the above compression method based on FAISS similarity deduplication, efficient deduplication compression of the sampled corpus set can be achieved, thereby further improving the data quality and model training efficiency.

[0203] In a feasible embodiment, step A13 may include: vectorizing the corpus samples in the sampled corpus set, and using any corpus sample vector in the sampled corpus set as the target query; calculating the similarity between the target query and other corpus sample vectors in the sampled corpus set respectively; removing the other corpus sample vectors whose similarity is greater than or equal to the preset similarity threshold as redundant corpus sample vectors to obtain the compressed corpus set.

[0204] It should be noted that the compression based on FAISS similarity deduplication is a greedy compression scheme. By setting a fixed filtering threshold, samples similar to the target query can be removed in sequence, and the above operations are repeatedly executed in the remaining sample set until the remaining sample set no longer changes, thereby obtaining the finally compressed corpus set.

[0205] It is understandable that vectorization is the process of converting corpus samples into numerical feature vectors, which can transform text data into a form that can be processed by machine learning models. In the vectorization process, methods such as the bag-of-words model, TF-IDF, word embeddings (such as Word2Vec, BERT) can be used to extract the features of corpus samples. These feature vectors can capture the lexical, syntactic, and semantic information in the corpus samples, thus providing strong support for subsequent similarity calculation and deduplication operations. Through vectorization processing, each corpus sample in the sampled corpus collection can be converted into a high-dimensional vector, and these vectors have certain distribution characteristics in the vector space.

[0206] It is worth noting that similar corpus samples are closer in the vector space, while dissimilar corpus samples are farther apart. Utilizing this property, efficient similarity calculation and deduplication operations can be achieved. When calculating similarity, methods such as cosine similarity and Euclidean distance can be used to measure the similarity degree between two vectors. When the similarity of two vectors exceeds a preset threshold, it can be considered that the corresponding corpus samples are duplicates, and deduplication processing is performed. By continuously iterating and optimizing this process, redundant information in the dataset can be gradually reduced while retaining key information, thereby achieving efficient deduplication and compression of the sampled corpus collection. This method can not only improve the efficiency and quality of data compression but also provide more accurate and reliable data support for subsequent model training and evaluation.

[0207] Step S202: Perform parallel data sampling and data merging on the compressed corpus collection to obtain a dataset.

[0208] It should be noted that in the parallel data sampling stage, a distributed computing framework such as Apache Spark or Hadoop can be used to split the compressed corpus collection into multiple subsets and perform sampling operations in parallel on different computing nodes. This can make full use of the computing power of multi-core processors and large-scale computing clusters, significantly improving the speed and efficiency of data sampling. At the same time, during the sampling process, the sampling ratio and sampling strategy can be dynamically adjusted according to the distribution characteristics of the data and business requirements to ensure the representativeness and diversity of the sampling results. After sampling is completed, the sampling results on each subset can be merged to obtain the final dataset. In the data merging stage, attention needs to be paid to data deduplication and consistency processing to avoid data redundancy and inconsistency problems. By using the above methods of parallel data sampling and data merging, the efficiency and accuracy of data processing can be further improved.

[0209] In a feasible implementation manner, step S202 may include: dividing the compressed corpus collection into multiple data blocks; performing parallel data sampling on the data blocks to obtain multiple corpus subsets; and performing data merging on the corpus subsets to obtain a dataset.

[0210] It should be noted that during the data merging process, various strategies can be adopted to ensure data integrity and consistency. For example, hash functions can be used to detect and remove duplicate data items while maintaining the original order of the data. In addition, for situations where data may be missing or abnormal, interpolation, mean filling, or machine learning-based methods can be used for data patching and cleaning to improve the quality and usability of the dataset. By carefully designing and optimizing the processes of parallel data sampling and data merging, modern computing resources can be fully utilized to achieve efficient, accurate, and reliable data processing.

[0211] It is worth noting that in order to improve the processing efficiency of large-scale training data, a data compression acceleration computing framework is proposed in this embodiment. The framework aims to achieve efficient data compression and significantly accelerate the computing speed by optimizing data statistics, sampling, Embeddings generation, and compression processes.

[0212] (1) Framework overview. The pre-training data compression framework includes a series of core algorithms and technical processes:

[0213] Accelerate text statistics: Count the number of statistical samples, tokens, and files in a large-scale corpus.

[0214] Accelerate Embeddings generation: Adopt multi-machine and multi-GPU parallel text embedding transformation to generate high-quality semantic representations.

[0215] Accelerate compression: Improve the running speed of the above four text compression algorithms from the aspects of algorithms, CPUs, GPUs, and I / O reads and writes.

[0216] Accelerate sampling: Based on data partitioning, perform parallel data sampling and subset merging to control the dataset ratio and the total number of tokens.

[0217] (2) Scalability. After verifying the effectiveness of the framework, it is applied to larger-scale datasets (compressed from 800B tokens to 200B tokens). This progress demonstrates the high scalability of the method and the potential for efficiency improvement.

[0218] (3) Utilization improvement. A low-cost computing resource solution is adopted: Use multiple low-configuration CPU and GPU servers for data parallel processing; Optimize task allocation by flexibly scheduling and managing more than 1000 command processes; Use a shared disk storage form to read the partitioned dataset to ensure efficient data storage and access.

[0219] (4) Process automation tools. Given the repetitive labor and fragmented processing logic in the data preprocessing process, this embodiment constructs a data processing pipeline to standardize all processing processes. The entire data pipeline is defined by configuring program scripts, and most steps from raw text cleaning to data compression are automatically executed, significantly reducing the need for human intervention.

[0220] Through these improvement measures, the processing time is shortened by four times, and the performance of the trained model is maintained or even improved. By leveraging the advantages of AI hardware resources and software frameworks, significant computational acceleration is achieved in processing large-scale text datasets.

[0221] In this embodiment, by performing data compression on the corpus samples in the sampled corpus set based on data similarity, redundant data is effectively removed, improving the data quality. Then, parallel data sampling and data merging are performed on the compressed corpus set to further improve the efficiency and accuracy of data processing.

[0222] Exemplarily, to help understand the implementation process of the large model optimization method obtained by combining this embodiment with the above-mentioned Embodiment 1, please refer to Figure 4 , Figure 4 A brief flowchart of a large model optimization method is provided. Specifically: based on the original data, ratio matching and sampling are performed to obtain a sampled dataset, which can include 200B tokens and 800B tokens; data compression operations are performed, including similarity compression rate search, QuRating, and clustering; the data after compression processing will be used in the pre-training process of the large model, and the pre-training process includes continued pre-training and pre-training from scratch. The large model includes Llama3-8b and the small model scratch-1.5B, and the training datasets are the entire dataset and the knowledge-based dataset respectively; the trained large model will be evaluated for downstream tasks.

[0223] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the large model optimization method of this application. Based on this technical concept, more forms of simple transformations are within the protection scope of this application.

[0224] This application also provides a large model optimization device. Please refer to Figure 5 , the large model optimization device includes:

[0225] An acquisition module 10 for acquiring a sampled corpus set.

[0226] A compression module 20 for performing data compression on the corpus samples in the sampled corpus set based on data similarity to obtain a dataset.

[0227] The training module 30 is used to train the large model based on the dataset to obtain an optimized large language model.

[0228] The large model optimization device provided in this application adopts the large model optimization method in the above embodiment, and can solve the technical problem that data redundancy in large-scale text data affects the model performance. Compared with the prior art, the beneficial effects of the large model optimization device provided in this application are the same as those of the large model optimization method provided in the above embodiment, and other technical features in the large model optimization device are the same as the features disclosed in the method of the above embodiment, which will not be elaborated here.

[0229] Optionally, the compression module 20 is further configured to perform data compression on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set; perform parallel data sampling and data merging on the compressed corpus set to obtain a dataset.

[0230] Optionally, the compression module 20 is further configured to perform mutually exclusive compression on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set; perform bilingual quality assessment compression on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set; perform clustering compression on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set; perform duplicate removal compression on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set.

[0231] Optionally, the compression module 20 is further configured to calculate the similarity between the corpus sample pairs in the sampled corpus set to obtain the similarity between the corpus sample pairs; construct a similarity matrix based on the similarity between the corpus sample pairs; perform mutually exclusive screening on the corpus samples in the sampled corpus set based on the similarity matrix and a preset similarity threshold to obtain a compressed corpus set.

[0232] Optionally, the compression module 20 is further configured to compare the elements in the similarity matrix with a preset similarity threshold to obtain a comparison result; in the case where the comparison result is that the element is less than the preset similarity threshold, obtain the index of the element; perform mutually exclusive screening on the corpus samples in the sampled corpus set according to the index of the element to obtain a mutually exclusive set; calculate the compression rate based on the mutually exclusive set to obtain the current compression rate; in the case where the current compression rate reaches the target compression rate, use the mutually exclusive set as the compressed corpus set; in the case where the current compression rate does not reach the target compression rate, adjust the preset similarity threshold and re-execute the step of comparing the elements in the similarity matrix with the preset similarity threshold to obtain a comparison result.

[0233] Optionally, the compression module 20 is further configured to perform language detection and classification on the corpus samples in the sampled corpus set to obtain a first corpus sample and a second corpus sample, where the languages of the first corpus sample and the second corpus sample are different; perform quality evaluation on the first corpus sample and the second corpus sample respectively to obtain corresponding quality evaluation scores; sort the first corpus sample and the second corpus sample based on the quality evaluation scores to obtain a sorting result; perform bilingual mixed sampling on the corpus samples in the sampled corpus set according to the sorting result to obtain a compressed corpus set.

[0234] Optionally, the compression module 20 is further configured to perform label prediction on the corpus samples in the sampled corpus set based on a preset clustering model to obtain clustering labels corresponding to the corpus samples; cluster the corpus samples in the sampled corpus set according to the clustering labels to obtain a clustered corpus set; calculate the similarity between the corpus samples corresponding to each category based on the clustered corpus set; filter the corpus samples within the cluster based on the similarity to obtain a compressed corpus set.

[0235] Optionally, the compression module 20 is further configured to vectorize the corpus samples in the sampled corpus set and use any corpus sample vector in the sampled corpus set as a target query; calculate the similarity between the target query and other corpus sample vectors in the sampled corpus set respectively; remove the other corpus sample vectors whose similarity is greater than or equal to a preset similarity threshold as redundant corpus sample vectors to obtain a compressed corpus set.

[0236] Optionally, the compression module 20 is further configured to divide the compressed corpus set into multiple data blocks; perform parallel data sampling on the data blocks to obtain multiple corpus subsets; perform data merging on the corpus subsets to obtain a data set.

[0237] Optionally, the acquisition module 10 is further configured to randomly select corpus data from the corpus based on a preset ratio to obtain original corpus, where the corpus includes multiple corpus data sets; perform word segmentation on the original corpus to obtain the word-segmented original corpus; perform data sampling on the word-segmented original corpus to obtain a sampled corpus set.

[0238] Optionally, the optimization module 30 is further configured to train a large model based on the data set to obtain a trained large model; perform performance evaluation on the trained large model to obtain performance metrics; in the case where the performance metrics reach the corresponding metric thresholds, use the trained large model as an optimized large language model; in the case where the performance metrics do not reach the corresponding metric thresholds, adjust the training parameters of the large model and re-execute the step of training the large model based on the data set.

[0239] The present application provides a large model optimization device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the large model optimization method in the first embodiment above.

[0240] Refer to the following Figure 6 , which shows a schematic structural diagram of a large model optimization device suitable for implementing the embodiments of the present application. The large model optimization device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The large model optimization device shown is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present application.

[0241] As Figure 6As shown, the large model optimization device may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which may perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM: Random Access Memory) 1004. In the RAM 1004, various programs and data required for the operation of the large model optimization device are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems may be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the large model optimization device to communicate with other devices wirelessly or wiredly to exchange data. Although the figure shows a large model optimization device having various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or had alternatively.

[0242] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program codes for performing the methods shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network through the communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above functions defined in the methods of the embodiments disclosed in the present application are performed.

[0243] The large model optimization device provided by the present application adopts the large model optimization method in the above embodiments, and can solve the technical problem that data redundancy in large-scale text data affects the model performance. Compared with the prior art, the beneficial effects of the large model optimization device provided by the present application are the same as those of the large model optimization method provided by the above embodiments, and other technical features in the large model optimization device are the same as those disclosed in the method of the previous embodiment, and will not be elaborated here.

[0244] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in a suitable manner in any one or more embodiments or examples.

[0245] As described above, this is only the specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in this application, and all of them should be covered by the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

[0246] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the large model optimization method in the above embodiments.

[0247] The computer-readable storage medium provided by this application can be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory, optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this embodiment, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or combined with an instruction execution system, device, or device. The program code contained on the computer-readable storage medium can be transmitted by any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0248] The above computer-readable storage medium can be included in the large model optimization device; it can also exist alone and not be assembled into the large model optimization device.

[0249] The above computer-readable storage medium stores one or more programs, which, when executed by the large model optimization device, cause the large model optimization device to: obtain a sampled corpus set; perform data compression on the corpus samples in the sampled corpus set based on data similarity to obtain a data set; and train a large model based on the data set to obtain an optimized large language model.

[0250] Computer program code for performing the operations of this application may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, and C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., by connecting through an Internet service provider via the Internet).

[0251] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system for performing the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0252] The modules described in the embodiments of this application may be implemented in software or in hardware. In some cases, the name of the module does not constitute a limitation on the unit itself.

[0253] The readable storage medium provided by this application is a computer-readable storage medium, and the computer-readable storage medium stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned large model optimization method, which can solve the technical problem that data redundancy in large-scale text data affects the model performance. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by this application are the same as those of the large model optimization method provided by the above embodiment, and will not be elaborated here.

[0254] This application also provides a computer program product, including a computer program, and when the computer program is executed by a processor, it realizes the steps of the large model optimization method as described above.

[0255] The computer program product provided by this application can solve the technical problem that data redundancy in large-scale text data affects the model performance. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as those of the large model optimization method provided by the above embodiment, and will not be elaborated here.

[0256] The above are only partial embodiments of this application, and do not limit the patent scope of this application. Any equivalent structural transformation made under the technical concept of this application by using the content of the specification and drawings of this application, or directly / indirectly applied in other related technical fields, is included in the patent protection scope of this application.

[0257] The present invention discloses A1. A large model optimization method, the method includes:

[0258] Obtain a sampled corpus set;

[0259] Perform data compression on the corpus samples in the sampled corpus set based on data similarity to obtain a data set;

[0260] Train a large model based on the data set to obtain an optimized large language model.

[0261] A2. The method as described in A1, where the performing data compression on the corpus samples in the sampled corpus set based on data similarity to obtain a data set includes:

[0262] Perform data compression on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set;

[0263] Perform parallel data sampling and data merging on the compressed corpus set to obtain a data set.

[0264] A3. The method as described in A2, where the performing data compression on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set includes at least one of the following:

[0265] Mutually exclusive compression is performed on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set;

[0266] Bilingual quality assessment compression is performed on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set;

[0267] Clustering compression is performed on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set;

[0268] Duplicate removal compression is performed on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set.

[0269] A4. The method as described in A3, where mutually exclusive compression is performed on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set, includes:

[0270] Calculate the similarity between corpus sample pairs in the sampled corpus set to obtain the similarity between the corpus sample pairs;

[0271] Construct a similarity matrix based on the similarity between the corpus sample pairs;

[0272] Based on the similarity matrix and a preset similarity threshold, perform mutually exclusive screening on the corpus samples in the sampled corpus set to obtain a compressed corpus set.

[0273] A5. The method as described in A4, where mutually exclusive screening is performed based on the similarity matrix and a preset similarity threshold to obtain a compressed corpus set, includes:

[0274] Compare the elements in the similarity matrix with the preset similarity threshold to obtain a comparison result;

[0275] When the comparison result is that the element is less than the preset similarity threshold, obtain the index of the element;

[0276] Based on the index of the element, perform mutually exclusive screening on the corpus samples in the sampled corpus set to obtain a mutually exclusive set;

[0277] Calculate the compression rate based on the mutually exclusive set to obtain the current compression rate;

[0278] When the current compression rate reaches the target compression rate, use the mutually exclusive set as the compressed corpus set;

[0279] When the current compression rate does not reach the target compression rate, adjust the preset similarity threshold and re-execute the step of comparing the elements in the similarity matrix with the preset similarity threshold to obtain a comparison result.

[0280] A6. The method as described in A3, which performs bilingual quality assessment and compression on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set, includes:

[0281] Performing language detection and classification on the corpus samples in the sampled corpus set to obtain a first corpus sample and a second corpus sample, where the languages of the first corpus sample and the second corpus sample are different;

[0282] Performing quality assessment on the first corpus sample and the second corpus sample respectively to obtain corresponding quality assessment scores;

[0283] Sorting the first corpus sample and the second corpus sample based on the quality assessment scores to obtain a sorting result;

[0284] Performing bilingual mixed sampling on the corpus samples in the sampled corpus set according to the sorting result to obtain a compressed corpus set.

[0285] A7. The method as described in A3, which performs clustering compression on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set, includes:

[0286] Performing label prediction on the corpus samples in the sampled corpus set based on a preset clustering model to obtain clustering labels corresponding to each corpus sample;

[0287] Clustering the corpus samples in the sampled corpus set according to the clustering labels to obtain a clustered corpus set;

[0288] Performing similarity calculation based on the clustered corpus set to obtain the similarity between the corpus samples corresponding to each category;

[0289] Screening the corpus samples within the cluster based on the similarity to obtain a compressed corpus set.

[0290] A8. The method as described in A3, which performs deduplication compression on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set, includes:

[0291] Vectorizing the corpus samples in the sampled corpus set, and using any corpus sample vector in the sampled corpus set as a target query;

[0292] Calculating the similarity between the target query and other corpus sample vectors in the sampled corpus set respectively;

[0293] Removing other corpus sample vectors with a similarity greater than or equal to a preset similarity threshold as redundant corpus sample vectors to obtain a compressed corpus set.

[0294] A9. The method as described in A2, wherein the parallel data sampling and data merging of the compressed corpus set to obtain a data set includes:

[0295] Dividing the compressed corpus set into multiple data blocks;

[0296] Performing parallel data sampling on the data blocks to obtain multiple corpus subsets;

[0297] Performing data merging on the corpus subsets to obtain a data set.

[0298] A10. The method as described in A1, wherein the obtaining of the sampled corpus set includes:

[0299] Randomly selecting corpus data from a corpus based on a preset ratio to obtain original corpus, wherein the corpus includes multiple corpus data sets;

[0300] Performing word segmentation on the original corpus to obtain the word-segmented original corpus;

[0301] Performing data sampling based on the word-segmented original corpus to obtain a sampled corpus set.

[0302] A11. The method as described in A1, wherein the training of a large model based on the data set to obtain an optimized large language model includes:

[0303] Training a large model based on the data set to obtain a trained large model;

[0304] Performing performance evaluation on the trained large model to obtain performance metrics;

[0305] When the performance metrics reach the corresponding metric thresholds, using the trained large model as the optimized large language model;

[0306] When the performance metrics do not reach the corresponding metric thresholds, adjusting the training parameters of the large model and re-executing the step of training the large model based on the data set.

[0307] The present invention also discloses B12. A large model optimization device, the large model optimization device includes:

[0308] An acquisition module for acquiring a sampled corpus set;

[0309] A compression module for compressing corpus samples in the sampled corpus set based on data similarity to obtain a data set;

[0310] A training module for training a large model based on the data set to obtain an optimized large language model.

[0311] B13. The device as described in B12, wherein the compression module is further configured to perform data compression on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set;

[0312] Perform parallel data sampling and data merging on the compressed corpus set to obtain a data set.

[0313] B14. The device as described in B13, wherein the compression module is further configured to perform mutually exclusive compression on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set;

[0314] Perform bilingual quality assessment compression on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set;

[0315] Perform clustering compression on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set;

[0316] Perform deduplication compression on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set.

[0317] B15. The device as described in B14, wherein the compression module is further configured to calculate the similarity between corpus sample pairs in the sampled corpus set to obtain the similarity between the corpus sample pairs;

[0318] Construct a similarity matrix based on the similarity between the corpus sample pairs;

[0319] Perform mutually exclusive screening on the corpus samples in the sampled corpus set based on the similarity matrix and a preset similarity threshold to obtain a compressed corpus set.

[0320] B16. The device as described in B15, wherein the compression module is further configured to compare the elements in the similarity matrix with a preset similarity threshold to obtain a comparison result;

[0321] When the comparison result is that the element is less than the preset similarity threshold, obtain the index of the element;

[0322] Perform mutually exclusive screening on the corpus samples in the sampled corpus set according to the index of the element to obtain a mutually exclusive set;

[0323] Calculate the compression rate based on the mutually exclusive set to obtain the current compression rate;

[0324] When the current compression rate reaches the target compression rate, use the mutually exclusive set as the compressed corpus set;

[0325] In the case that the current compression ratio has not reached the target compression ratio, adjust the preset similarity threshold, and re-execute the step of comparing the elements in the similarity matrix with the preset similarity threshold to obtain a comparison result.

[0326] B17. The device as described in B14, wherein the compression module is further configured to perform language detection and classification on the corpus samples in the sampled corpus set to obtain a first corpus sample and a second corpus sample, where the languages of the first corpus sample and the second corpus sample are different;

[0327] Perform quality assessment on the first corpus sample and the second corpus sample respectively to obtain corresponding quality assessment scores;

[0328] Sort the first corpus sample and the second corpus sample based on the quality assessment scores to obtain a sorting result;

[0329] Perform bilingual mixed sampling on the corpus samples in the sampled corpus set according to the sorting result to obtain a compressed corpus set.

[0330] B18. The device as described in B14, wherein the compression module is further configured to perform label prediction on the corpus samples in the sampled corpus set based on a preset clustering model to obtain clustering labels corresponding to the respective corpus samples;

[0331] Cluster the corpus samples in the sampled corpus set according to the clustering labels to obtain a clustered corpus set;

[0332] Calculate the similarity between the corpus samples corresponding to each category based on the clustered corpus set;

[0333] Filter the corpus samples within the cluster based on the similarity to obtain a compressed corpus set.

[0334] The present invention also discloses C19. A large model optimization device, the large model optimization device includes: a memory, a processor, and a large model optimization program stored on the memory and executable on the processor, the large model optimization program is configured to implement the large model optimization method as described above.

[0335] The present invention also discloses D20. A storage medium, on which a large model optimization program is stored, and when the large model optimization program is executed by a processor, it implements the large model optimization method as described above.

Claims

1. An optimization method for large models, characterized in that, The method includes: Obtaining a sampled corpus set; Performing data compression on the corpus samples in the sampled corpus set based on data similarity to obtain a data set; Training a large model based on the data set to obtain an optimized large language model.

2. The method according to claim 1, characterized in that, The performing data compression on the corpus samples in the sampled corpus set based on data similarity to obtain a data set includes: Performing data compression on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set; Performing parallel data sampling and data merging on the compressed corpus set to obtain a data set.

3. The method according to claim 2, wherein The performing data compression on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set includes at least one of the following: Performing mutually exclusive compression on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set; Performing bilingual quality assessment compression on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set; Performing clustering compression on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set; Performing deduplication compression on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set.

4. The method according to claim 3, wherein The performing mutually exclusive compression on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set includes: Calculating the similarity between corpus sample pairs in the sampled corpus set to obtain the similarity between the corpus sample pairs; Constructing a similarity matrix based on the similarity between the corpus sample pairs; Performing mutually exclusive screening on the corpus samples in the sampled corpus set based on the similarity matrix and a preset similarity threshold to obtain a compressed corpus set.

5. The method according to claim 4, wherein The performing mutually exclusive screening based on the similarity matrix and a preset similarity threshold to obtain a compressed corpus set includes: Comparing the elements in the similarity matrix with the preset similarity threshold to obtain a comparison result; When the comparison result is that the element is less than the preset similarity threshold, obtaining the index of the element; Performing mutually exclusive screening on the corpus samples in the sampled corpus set according to the index of the element to obtain a mutually exclusive set; Calculating a compression rate based on the mutually exclusive set to obtain the current compression rate; When the current compression rate reaches the target compression rate, using the mutually exclusive set as the compressed corpus set; When the current compression rate does not reach the target compression rate, adjusting the preset similarity threshold and re-executing the step of comparing the elements in the similarity matrix with the preset similarity threshold to obtain a comparison result.

6. The method according to claim 3, wherein The performing bilingual quality assessment compression on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set includes: Performing language detection and classification on the corpus samples in the sampled corpus set to obtain a first corpus sample and a second corpus sample, where the languages of the first corpus sample and the second corpus sample are different; Performing quality assessment on the first corpus sample and the second corpus sample respectively to obtain corresponding quality assessment scores; Sort the first corpus sample and the second corpus sample based on the quality assessment scores to obtain a sorting result; Perform bilingual mixed sampling on the corpus samples in the sampled corpus set according to the sorting result to obtain a compressed corpus set.

7. The method according to claim 3, wherein The clustering compression of the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set includes: Perform label prediction on the corpus samples in the sampled corpus set based on a preset clustering model to obtain the clustering labels corresponding to each corpus sample; Cluster the corpus samples in the sampled corpus set according to the clustering labels to obtain a clustered corpus set; Calculate the similarity between the corpus samples corresponding to each category based on the clustered corpus set; Screen the corpus samples within the cluster based on the similarity to obtain a compressed corpus set.

8. An optimization device for large models, characterized in that, The large model optimization device includes: An acquisition module for acquiring a sampled corpus set; A compression module for performing data compression on the corpus samples in the sampled corpus set based on data similarity to obtain a data set; A training module for training a large model based on the data set to obtain an optimized large language model.

9. An optimization device for large models, characterized in that, The large model optimization device includes: a memory, a processor, and a large model optimization program stored on the memory and executable on the processor, the large model optimization program being configured to implement the large model optimization method according to any one of claims 1 to 7.

10. A storage medium, characterized in that, A large model optimization program is stored on the storage medium, and when the large model optimization program is executed by a processor, it implements the large model optimization method according to any one of claims 1 to 7.

Images