Large model optimization method and device, equipment and storage medium

Abstract

The application relates to the technical field of large language models, and discloses a large model optimization method and device, equipment and a storage medium. The method comprises the following steps: acquiring a sampling corpus set; performing data compression on corpus samples in the sampling corpus set based on data similarity to obtain a data set; and training a large model based on the data set to obtain an optimized large language model. According to the application, the redundant data in the training data of the large language model is removed through data compression according to the data similarity, the data quality is effectively improved, and then the model is trained through the compressed data, so that the model training efficiency and the model performance are effectively improved.

Description

Technical Field

[0001] This application relates to the field of large language model technology, and in particular to large model optimization methods, apparatus, devices and storage media. Background Technology

[0002] Incremental pre-training (or continued pre-training) of large models has become an important means of improving the performance of base models. Current practices face the problems of data redundancy and semantic repetition. This challenge not only affects the effectiveness of learning but also consumes significant computational resources and storage space. When processing large-scale text data, duplicate data can easily induce bias, leading to the "illusion" effect—that is, large models generate inaccurate or illogical outputs. Although existing techniques have demonstrated significant value in their respective fields, challenges remain in efficiently removing duplicate data, reducing resource consumption, and enhancing the knowledge retention capacity during the training of large models.

[0003] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a method, apparatus, device and storage medium for large-scale model optimization, which aims to solve the technical problem of data redundancy in large-scale text data affecting model performance.

[0005] To achieve the above objectives, this application proposes a large model optimization method, which includes:

[0006] Obtain the sampled corpus set;

[0007] Data samples in the sampled corpus are compressed based on data similarity to obtain a dataset.

[0008] The large model is trained based on the dataset to obtain an optimized large language model.

[0009] In one embodiment, the step of compressing the corpus samples in the sampled corpus based on data similarity to obtain a dataset includes:

[0010] Based on data similarity, the corpus samples in the sampled corpus set are compressed to obtain the compressed corpus set;

[0011] The compressed corpus is subjected to parallel data sampling and data merging to obtain a dataset.

[0012] In one embodiment, the data compression of the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set includes at least one of the following:

[0013] Based on data similarity, mutually exclusive compression is performed on the corpus samples in the sampled corpus set to obtain the compressed corpus set;

[0014] Based on data similarity, the corpus samples in the sampled corpus set are compressed for bilingual quality assessment to obtain the compressed corpus set;

[0015] Based on data similarity, the corpus samples in the sampled corpus are clustered and compressed to obtain the compressed corpus.

[0016] Based on data similarity, the corpus samples in the sampled corpus are deduplicated and compressed to obtain the compressed corpus.

[0017] In one embodiment, the step of mutually exclusive compression of corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set includes:

[0018] The similarity between corpus sample pairs in the sampled corpus set is calculated.

[0019] A similarity matrix is ​​constructed based on the similarity between the corpus sample pairs;

[0020] Based on the similarity matrix and the preset similarity threshold, mutually exclusive screening is performed on the corpus samples in the sampled corpus set to obtain the compressed corpus set.

[0021] In one embodiment, the step of performing mutually exclusive filtering based on the similarity matrix and a preset similarity threshold to obtain a compressed corpus set includes:

[0022] The elements in the similarity matrix are compared with a preset similarity threshold to obtain the comparison result;

[0023] If the comparison result indicates that the element is less than the preset similarity threshold, the index of the element is obtained;

[0024] Based on the index of the element, mutually exclusive filters are performed on the corpus samples in the sampled corpus set to obtain a mutually exclusive set;

[0025] The compression ratio is calculated based on the mutually exclusive set to obtain the current compression ratio;

[0026] If the current compression rate reaches the target compression rate, the mutually exclusive set will be used as the compressed corpus set.

[0027] If the current compression ratio does not reach the target compression ratio, adjust the preset similarity threshold and re-execute the step of comparing the elements in the similarity matrix with the preset similarity threshold to obtain the comparison result.

[0028] In one embodiment, the step of compressing the corpus samples in the sampled corpus set based on data similarity for bilingual quality assessment to obtain a compressed corpus set includes:

[0029] Language detection and classification are performed on the corpus samples in the sampled corpus set to obtain a first corpus sample and a second corpus sample, wherein the languages ​​of the first corpus sample and the second corpus sample are different;

[0030] The first and second corpus samples were evaluated for quality, and the corresponding quality evaluation scores were obtained.

[0031] The first and second corpus samples are sorted based on the quality assessment scores to obtain the sorting results;

[0032] Based on the sorting results, bilingual mixed sampling is performed on the corpus samples in the sampled corpus set to obtain a compressed corpus set.

[0033] In one embodiment, the step of clustering and compressing the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set includes:

[0034] Based on a pre-defined clustering model, the labels of the corpus samples in the sampled corpus set are predicted to obtain the clustering labels corresponding to each corpus sample;

[0035] The corpus samples in the sampled corpus set are clustered according to the clustering labels to obtain the clustered corpus set;

[0036] Similarity calculation is performed on the clustered corpus to obtain the similarity between corpus samples corresponding to each category.

[0037] Based on the similarity, the corpus samples within the cluster are filtered to obtain a compressed corpus set.

[0038] In one embodiment, the step of deduplicating and compressing the corpus samples in the sampled corpus based on data similarity to obtain a compressed corpus includes:

[0039] Vectorize the corpus samples in the sampled corpus set, and use any corpus sample vector in the sampled corpus set as the target query;

[0040] Calculate the similarity between the target query and the sample vectors of other corpora in the sampled corpus set;

[0041] Other corpus sample vectors with similarity greater than or equal to a preset similarity threshold are removed as redundant corpus sample vectors to obtain a compressed corpus set.

[0042] In one embodiment, the parallel data sampling and data merging of the compressed corpus to obtain a dataset includes:

[0043] The compressed corpus is divided into multiple data blocks;

[0044] Parallel data sampling is performed on the data block to obtain multiple subsets of the corpus;

[0045] The subset of the corpus is merged to obtain the dataset.

[0046] In one embodiment, obtaining the sampled corpus set includes:

[0047] Based on a preset ratio, corpus data is randomly selected from the corpus to obtain the original corpus, wherein the corpus includes multiple corpus data sets;

[0048] The original corpus is segmented into words to obtain the segmented original corpus;

[0049] Data sampling is performed on the original corpus after word segmentation to obtain a sampled corpus set.

[0050] In one embodiment, training a large model based on the dataset to obtain an optimized large language model includes:

[0051] The large model is trained based on the dataset to obtain the trained large model;

[0052] The performance of the trained large model is evaluated to obtain performance metrics.

[0053] If the performance index reaches the corresponding index threshold, the trained large model will be used as the optimized large language model.

[0054] If the performance metric fails to reach the corresponding threshold, adjust the training parameters of the large model and re-execute the step of training the large model based on the dataset.

[0055] Furthermore, to achieve the above objectives, this application also proposes a large model optimization apparatus, which includes:

[0056] The acquisition module is used to acquire the sampled corpus set;

[0057] The compression module is used to compress the corpus samples in the sampled corpus based on data similarity to obtain the dataset;

[0058] The training module is used to train the large model based on the dataset to obtain an optimized large language model.

[0059] In one embodiment, the compression module is further configured to compress the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set;

[0060] The compressed corpus is subjected to parallel data sampling and data merging to obtain a dataset.

[0061] In one embodiment, the compression module is further configured to perform mutually exclusive compression on the corpus samples in the sampled corpus set based on data similarity, to obtain a compressed corpus set;

[0062] Based on data similarity, the corpus samples in the sampled corpus set are compressed for bilingual quality assessment to obtain the compressed corpus set;

[0063] Based on data similarity, the corpus samples in the sampled corpus are clustered and compressed to obtain the compressed corpus.

[0064] Based on data similarity, the corpus samples in the sampled corpus are deduplicated and compressed to obtain the compressed corpus.

[0065] In one embodiment, the compression module is further configured to perform similarity calculation on the pairs of corpus samples in the sampled corpus set to obtain the similarity between the pairs of corpus samples;

[0066] A similarity matrix is ​​constructed based on the similarity between the corpus sample pairs;

[0067] Based on the similarity matrix and the preset similarity threshold, mutually exclusive screening is performed on the corpus samples in the sampled corpus set to obtain the compressed corpus set.

[0068] In one embodiment, the compression module is further configured to compare the elements in the similarity matrix with a preset similarity threshold to obtain a comparison result;

[0069] If the comparison result indicates that the element is less than the preset similarity threshold, the index of the element is obtained;

[0070] Based on the index of the element, mutually exclusive filters are performed on the corpus samples in the sampled corpus set to obtain a mutually exclusive set;

[0071] The compression ratio is calculated based on the mutually exclusive set to obtain the current compression ratio;

[0072] If the current compression rate reaches the target compression rate, the mutually exclusive set will be used as the compressed corpus set.

[0073] If the current compression ratio does not reach the target compression ratio, adjust the preset similarity threshold and re-execute the step of comparing the elements in the similarity matrix with the preset similarity threshold to obtain the comparison result.

[0074] In one embodiment, the compression module is further configured to perform language detection and classification on the corpus samples in the sampled corpus set to obtain a first corpus sample and a second corpus sample, wherein the languages ​​of the first corpus sample and the second corpus sample are different;

[0075] The first and second corpus samples were evaluated for quality, and the corresponding quality evaluation scores were obtained.

[0076] The first and second corpus samples are sorted based on the quality assessment scores to obtain the sorting results;

[0077] Based on the sorting results, bilingual mixed sampling is performed on the corpus samples in the sampled corpus set to obtain a compressed corpus set.

[0078] In one embodiment, the compression module is further configured to predict the labels of the corpus samples in the sampled corpus set based on a preset clustering model, so as to obtain the clustering labels corresponding to each corpus sample;

[0079] The corpus samples in the sampled corpus set are clustered according to the clustering labels to obtain the clustered corpus set;

[0080] Similarity calculation is performed on the clustered corpus to obtain the similarity between corpus samples corresponding to each category.

[0081] Based on the similarity, the corpus samples within the cluster are filtered to obtain a compressed corpus set.

[0082] In addition, to achieve the above objectives, this application also proposes a large model optimization apparatus, the apparatus comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the large model optimization method as described above.

[0083] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the large model optimization method described above.

[0084] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the large model optimization method described above.

[0085] This application provides a method for optimizing a large language model. The method involves first obtaining a sampled corpus; then compressing the corpus samples based on data similarity to obtain a dataset; and finally training a large language model using this dataset to obtain an optimized large language model. By compressing data based on data similarity to remove redundant data from the training data of the large language model, the method effectively improves data quality. Furthermore, training the model using the compressed data significantly improves training efficiency and model performance. Attached Figure Description

[0086] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0087] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0088] Figure 1 This is a flowchart illustrating the first embodiment of the large model optimization method of this application.

[0089] Figure 2 This is a flowchart illustrating Embodiment 2 of the large model optimization method of this application;

[0090] Figure 3 A schematic diagram illustrating the sample partitioning effect of different clustering parameters provided in an embodiment of the large model optimization method of this application;

[0091] Figure 4 A simplified flowchart is provided for one embodiment of the large model optimization method of this application;

[0092] Figure 5 This is a schematic diagram of the module structure of the large model optimization device in an embodiment of this application;

[0093] Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the large model optimization method of this application.

[0094] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0095] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0096] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0097] The main solution of this application embodiment is: to obtain a sampled corpus set; to compress the corpus samples in the sampled corpus set based on data similarity to obtain a dataset; and to train a large model based on the dataset to obtain an optimized large language model.

[0098] Incremental pre-training (or continued pre-training) of large models has become an important means of improving the performance of the base model. Current practices face the problems of data redundancy and semantic repetition. This challenge not only affects the effectiveness of learning but also consumes significant computational resources and storage space. When processing large-scale text data, repetitive data can easily induce bias, leading to the "illusion" effect—that is, large models generate inaccurate or illogical outputs.

[0099] This application improves data quality by compressing data based on data similarity to remove redundant data in the training data of large language models. This, in turn, improves model training efficiency and performance by using the compressed data for model training.

[0100] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or large model optimization device capable of performing the above functions. The following description uses a large model optimization device as an example to illustrate this embodiment and the subsequent embodiments.

[0101] Based on this, this application provides a large model optimization method, applied to a control host, with reference to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the large model optimization method of this application.

[0102] In this embodiment, the large model optimization method includes steps S10 to S30:

[0103] Step S10: Obtain the sampled corpus set.

[0104] It should be noted that the sampled corpus set refers to the corpus data set obtained by matching and sampling the original corpus data. It contains various corpus samples used for training large models. These corpus samples can be in the form of text, dialogue, article fragments, etc., and this embodiment does not impose specific limitations on them. The construction of the sampled corpus set is to ensure that the large model can learn from diverse data, thereby improving the model's generalization ability and performance.

[0105] In one feasible implementation, step S10 may include: randomly selecting corpus data from a corpus based on a preset ratio to obtain original corpus, wherein the corpus includes multiple corpus data sets; segmenting the original corpus into words to obtain segmented original corpus; and sampling data based on the segmented original corpus to obtain a sampled corpus set.

[0106] It should be noted that a corpus is a database storing a large amount of linguistic data, which can be used for training and optimizing large models. In this embodiment, the corpus can be regarded as a raw dataset containing multiple commonly used linguistic datasets. For example, the corpus may include more than 60 commonly used linguistic datasets, totaling 173TB. These data are matched and sampled according to a preset ratio to achieve the required data scale.

[0107] It should be understood that commonly used corpus datasets can include more than 60 types, such as government, wanjuan_zh_patent_high_quality, 360we nku, 360wenda, and 360tiku, with a total capacity of 173TB. To optimize the training process of large-scale language models, it is necessary to sample a specific size subset of data from this massive dataset, such as 20 billion tokens, to ensure the best balance between training efficiency and effectiveness.

[0108] It is worth noting that the data proportions and dataset sizes differ across different corpus datasets; for example, government and wanjua... The data sets n_zh_patent_high_quality, 360wenku, 360wenda, and 360tiku have data ratios of 0.00277, 0.00206, 0.10715, 0.0274573, and 0.0469119, respectively, with dataset sizes of 20G, 17G, 5.4T, 551G, and 74G. Specifically, government is a government-related text dataset containing corpora related to government policies, legal documents, and government reports; wanjuan_zh_patent_high_quality is a Chinese patent dataset containing patent text corpora; 360wenku is a shared Chinese document dataset, including academic papers, technical reports, and e-books; 360wenda is a Chinese question-and-answer dataset, including various questions and answers; and 360tiku is an educational resource dataset, including test questions, courseware, and lesson plans.

[0109] It should be understood that by sampling from the corpus according to a preset ratio, a customized dataset of the expected size, i.e., the original corpus, can be generated. By segmenting the text in the original corpus into words, converting the text into a series of token ID sequences, not only is the speed of subsequent model training improved, but the overall training quality is also enhanced. Data sampling of the segmented original corpus aims to select representative and diverse samples from the massive corpus data to construct a sampled corpus set. In this way, existing large-scale corpus resources can be efficiently utilized, accelerating and optimizing the learning process of large-scale language models.

[0110] Step S20: Compress the corpus samples in the sampled corpus set based on data similarity to obtain the dataset.

[0111] It should be noted that data compression aims to reduce data redundancy and increase knowledge density, thereby improving data quality and optimizing the model training process. In this embodiment, data compression is based on data similarity, that is, removing duplicate or highly similar data based on data similarity, and retaining representative and unique samples.

[0112] It should be understood that, in order to solve the problem of global semantic compression and minimize data redundancy, this embodiment may use a global sample mutual exclusion data compression algorithm for data compression, or it may use bilingual-QuRating, adaptive sampling-based kmenas clustering compression method, FAISS similarity deduplication compression algorithm, etc. This embodiment does not impose specific limitations on this.

[0113] Step S30: Train the large model based on the dataset to obtain an optimized large language model.

[0114] It should be noted that the compressed data will be used in the pre-training process of the large model. The trained large model will be evaluated on 14 standard test sets to verify its performance improvement.

[0115] It is understood that the large model in this implementation can be the LLaMA3-8B pedestal model, which is trained using the next token prediction method. The core technologies mainly focus on parameter tuning, performance optimization, and cost estimation.

[0116] In one feasible implementation, step S30 may include: training a large model based on the dataset to obtain a trained large model; evaluating the performance of the trained large model to obtain a performance index; if the performance index reaches the corresponding index threshold, using the trained large model as an optimized large language model; if the performance index does not reach the corresponding index threshold, adjusting the training parameters of the large model and re-executing the step of training the large model based on the dataset.

[0117] It should be noted that in this embodiment, the hardware configuration for large model pre-training is a cluster consisting of 8 H-800 servers (each with 8 H800 GPUs and 2TB of memory), and training 200B tokens takes 24 hours. If 4 H-800 servers are used, the training time will double.

[0118] Understandably, when using the LLaMA3-8B base model, the dataset can have two sizes: one containing 50B tokens and the other containing 200B tokens. The initial loss is 2.2, and the global batch size is set to 1024. Training objectives are achieved through techniques such as gradient accumulation, data parallelism, and model parallelism.

[0119] For the 50B token dataset, training was completed after 11,920 iterations, reducing the loss to 1.73, and taking approximately 25 hours using 8 computing nodes. In contrast, the 200B token dataset required 47,683 iterations to achieve a similar loss (1.69). During training at the same timescale, the loss values ​​for both datasets remained largely consistent. Comparing the results using only 1 / 4 of the data volume with the 200B token dataset shows a 0.06-point decrease in average score on the knowledge-based dataset, but a 0.63-point increase across all dataset types.

[0120] To compare the impact of different data volumes and training parameters on model performance, increasing the learning rate of the 50B token dataset by four times or reducing its batch size to one-quarter of its original size can alleviate the problem of differences in the total number of training steps or the magnitude of updates.

[0121] Furthermore, additional pre-training was performed on the 50B randomly sampled data, and it was compared with the selected 50B data. The results showed that when using the LLaMA3-8B model, the average score improved by 1.57 points on the knowledge-based dataset, while it increased by 0.5 points on all types of datasets.

[0122] The test results above not only reveal the impact of different scales and parameter settings on training efficiency, but also demonstrate that carefully selecting training data can significantly improve the model's performance on specific tasks.

[0123] Alternatively, the large model can also be a small Transformer model for scratch-1.3B. To evaluate the data compression scheme without interference and to improve experimental efficiency in the process, a small Transformer model for scratch-1.3B was trained and analyzed from scratch. This model is based on the Llama 3 architecture, and the parameter size was controlled to approximately 1.3B by adjusting the number of layers and vector dimensions. A 50B token dataset was randomly selected from the 200B token dataset as the initial training dataset v1. In addition, another different 50B dataset v2 was selected based on the 200B dataset using a deduplication strategy. The effectiveness of the data compression scheme was evaluated by comparing the performance of the model pre-training using these two datasets with different characteristics.

[0124] Specifically, the model structure and parameters are as follows: The basic architecture is based on Llama 3's Transformer layers; the number of parameters is controlled at 1.3B (i.e., scratch-1.3B). Pre-training parameters are: a learning rate of 2e-4, using a cosine annealing learning rate scheduling strategy. The final loss value when training on the selected dataset v2 is approximately 2.4, while the final loss obtained from training on the randomly selected dataset v1 is approximately 2.6. When attempting to adjust the learning rate to 0.01, it was found that the model training effect did not change significantly, and the final loss remained unchanged.

[0125] The effectiveness of different compression schemes was evaluated by comparing the performance of training with a deduplicated, carefully selected dataset (v2) versus a randomly selected dataset (v1). The results show that, at the same learning rate, using the optimized deduplicated dataset allows the model to reach a lower loss faster, demonstrating better convergence performance and a more efficient learning process. This experimental design not only verifies the effectiveness of specific data processing strategies in training small Transformer models but also improves overall experimental efficiency to some extent.

[0126] It's worth noting that during model training, performance metrics can include perplexity and loss. Perplexity reflects the model's uncertainty in predicting test data; a lower value indicates stronger predictive ability. Loss, on the other hand, quantifies the difference between the model's prediction and the true label; similarly, a lower loss value means the model's prediction is more accurate.

[0127] Optionally, accuracy is used in downstream task evaluation. Downstream task evaluation includes: 1) Knowledge ability evaluation: using the average value of knowledge-based datasets (mmlu, cmmlu, ceval, gpqa_diamond, arc-c); 2) General ability evaluation: using the average value of all datasets; 3) Experimental setup: full data vs. random sampling vs. selected compressed data.

[0128] The pre-training dataset volume was reduced to 25% of its original size while ensuring that the model performance remained essentially the same as before compression. Experimental results show that: (1) Compressing the pre-training data volume to 25% (from 200B tokens to 50B tokens) saved 3 / 4 of the model training time. (2) Continuing pre-training, 200B data vs. 50B selected data. The average score of Llama3-8b 50B data decreased by 0.06 points in the knowledge dataset, but increased by 0.63 points in all datasets. (3) Continuing pre-training, 50B randomly sampled data vs. 50B selected data. The average score of Llama3-8b knowledge dataset increased by 1.57 points, and the average score of all datasets increased by 0.5 points. (4) Pre-training from scratch: 50B randomly sampled data vs. 50B selected data. A large model of 1.5B was randomly initialized, and Scratch-1.5B was pre-trained from scratch. The average score of the knowledge dataset increased by 0.66 points, and the average score of all datasets increased by 0.11 points.

[0129] This embodiment provides a large-scale model optimization method. First, a sampled corpus set is obtained; then, the corpus samples in the sampled corpus set are compressed based on data similarity to obtain a dataset; finally, a large-scale model is trained based on this dataset to obtain an optimized large-scale language model. By compressing data based on data similarity to remove redundant data from the large-scale language model training data, the data quality is effectively improved. Furthermore, training the model using the compressed data effectively improves model training efficiency and model performance.

[0130] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 Step S20 further includes steps S201-S202:

[0131] Step S201: Based on data similarity, compress the data samples in the sampled corpus set to obtain the compressed corpus set.

[0132] It should be noted that, in order to solve the problem of global semantic compression and minimize data redundancy, this embodiment can use a global sample mutual exclusion data compression algorithm for data compression.

[0133] Alternatively, data compression in this embodiment can also employ bilingual QuRating, adaptive sampling-based kmenas clustering compression, FAISS similarity deduplication compression algorithm, etc. This embodiment does not impose specific limitations on these methods.

[0134] In one feasible implementation, step S201 may include at least one of steps A10 to A13:

[0135] Step A10: Perform mutually exclusive compression on the corpus samples in the sampled corpus set based on data similarity to obtain the compressed corpus set.

[0136] It should be noted that this implementation uses a global sample mutual exclusion data compression algorithm for mutual exclusion compression. This algorithm identifies and removes highly similar redundant samples by calculating the similarity between corpus samples, thereby retaining the most representative data. This method can not only significantly reduce the size of the dataset, but also effectively avoid overfitting during model training and improve the model's generalization ability.

[0137] In one feasible implementation, step A10 may include: calculating the similarity between pairs of corpus samples in the sampled corpus set to obtain the similarity between the pairs of corpus samples; constructing a similarity matrix based on the similarity between the pairs of corpus samples; and performing mutual exclusion screening on the corpus samples in the sampled corpus set based on the similarity matrix and a preset similarity threshold to obtain a compressed corpus set.

[0138] It should be noted that existing data deduplication methods rely on the FAISS database; however, they face the problem of not being able to adaptively search for optimal thresholds for different subsets, and they cannot achieve global sample mutual exclusion. In this implementation, mutually exclusive compression of samples is achieved globally through adaptive compression threshold search and sample selection algorithms. In terms of computation, the use of a data partitioning algorithm significantly improves GPU utilization efficiency.

[0139] Understandably, to improve data compression efficiency, the sampled corpus can be initially divided into several subsets before constructing the similarity matrix. Then, similarity calculations are performed within each subset to construct a similarity matrix, and mutual exclusion filtering is performed based on a preset similarity threshold. This ensures that each subset retains the most representative corpus samples while removing redundancy; this embodiment does not impose specific limitations on this approach.

[0140] It's worth noting that the similarity matrix is ​​a two-dimensional array, where each element represents the similarity between two corpus samples. Similarity can be calculated using algorithms such as cosine similarity, Jaccard similarity, or edit distance; the specific choice depends on the characteristics of the corpus samples and the application scenario. In this implementation, to more accurately measure the semantic similarity between corpus samples, a semantic similarity calculation method based on the BERT model can be used to capture the deep semantic relationships between corpus samples, thereby more accurately identifying and removing redundant samples.

[0141] It should be understood that after constructing the similarity matrix, a mutually exclusive selection of corpus samples can be performed by setting a preset similarity threshold. That is, for corpus sample pairs with similarity exceeding the threshold, only one is retained as a representative sample, thereby achieving data compression. This mutually exclusive selection method can effectively reduce redundant information in the dataset, improve data quality, and thus improve the efficiency and performance of model training. Optionally, the selection of the preset similarity threshold is crucial, as it directly affects the data compression effect and the final result of model training. To find the optimal preset similarity threshold, this embodiment further proposes an adaptive compression threshold search algorithm. This algorithm automatically selects the best preset similarity threshold by iteratively adjusting the preset similarity threshold and evaluating the model training effect of the compressed dataset under different thresholds.

[0142] In one feasible implementation, the step of performing mutually exclusive filtering based on the similarity matrix and a preset similarity threshold to obtain a compressed corpus set includes: comparing elements in the similarity matrix with the preset similarity threshold to obtain a comparison result; if the comparison result indicates that the element is less than the preset similarity threshold, obtaining the index of the element; performing mutually exclusive filtering on corpus samples in the sampled corpus set according to the index of the element to obtain a mutually exclusive set; calculating the compression ratio based on the mutually exclusive set to obtain a current compression ratio; if the current compression ratio reaches the target compression ratio, using the mutually exclusive set as the compressed corpus set; if the current compression ratio does not reach the target compression ratio, adjusting the preset similarity threshold and re-executing the step of comparing elements in the similarity matrix with the preset similarity threshold to obtain a comparison result.

[0143] It should be noted that in this embodiment, an exclusive set extraction algorithm is used to achieve efficient screening and compression of corpus samples. The goal of the exclusive set extraction algorithm is to refine the data set by eliminating rows (or columns in a symmetric matrix) corresponding to elements with pairwise similarity below a predefined threshold, so as to extract an exclusive sample set. Utilizing the properties of the symmetric matrix, this algorithm simplifies the calculation and effectively processes the input sample set represented as a two-dimensional matrix. This algorithm accepts two parameters: the matrix itself containing the samples and the similarity threshold. It outputs two key values: the compression ratio "ratio" and the set of retained row indices "keep set". This algorithm first constructs a similarity matrix that captures the pairwise similarity between samples. Subsequently, it filters out elements with similarity below the specified threshold. This is achieved by comparing each element of the similarity matrix with the threshold and using the "(inner < Threshold).nonzero()" operation to determine the positions of the elements that do not meet the criteria. Here, "inner" represents the pre-computed similarity matrix, and "nonzero()" returns the indices of the elements that do not exceed the threshold. To determine the retained rows, this algorithm only focuses on the row indices (recognizing that the column indices are redundant in a symmetric matrix). It extracts the first dimension of the index array, converts it into a set "keep set" to eliminate duplicates, and calculates the compression ratio based on the number of retained rows. This method ensures that only the most representative exclusive samples are retained, thus promoting effective data set compression while retaining key information.

[0144] The implementation of the exclusive set extraction algorithm is as follows:

[0145] Input: matrix, threshold

[0146] Output: ratio, keep_set

[0147] 1: inner = matrix * matrix^T

[0148] 2: keeps = (inner < threshold).nonzero()

[0149] 3: keeps_set = set(keeps[:, 0])

[0150] 4: return len(keeps_set) / batch, keeps_set

[0151] It is worth noting that by determining the optimal preset similarity threshold through an adaptive compression threshold search algorithm, the effect of data compression can be significantly improved, and the performance of model training can be further optimized.

[0152] Optionally, the adaptive compression threshold search algorithm employs a binary search method. This algorithm continuously adjusts the preset similarity threshold and evaluates the model training performance of the compressed dataset after each adjustment, thereby gradually approaching the optimal preset similarity threshold. The core of the binary search method lies in its ability to halve the search range in each iteration, thus quickly converging to the optimal solution. This efficient search strategy enables the adaptive compression threshold search algorithm to quickly find the optimal preset similarity threshold with limited computational resources, thereby optimizing both data compression performance and model training performance.

[0153] In practical implementation, an initial preset similarity threshold range can be set first, and then a binary search method can be used to search within this range. In each iteration, the midpoint of the current search range is calculated as the new preset similarity threshold, and the sampled corpus is mutually exclusive-filtered and compressed based on this threshold. Subsequently, the compressed dataset is used for model training, and the model training effect is evaluated. Based on the evaluation results, the search range is adjusted, and the next iteration continues until a preset search termination condition is reached, such as the search range being less than the preset threshold or the improvement in model training effect being less than the preset threshold. Finally, the preset similarity threshold obtained in the last iteration is taken as the optimal solution and used in subsequent data compression and model training processes.

[0154] It's worth noting that a binary search method is used to adaptively determine the optimal threshold within a specified subset of a given sample matrix, defined by the "start" and "end" indices, such that the ratio of retained elements is close to a predefined target ratio ("target ratio"). The algorithm initializes the "keeps set" variable to accumulate row indices identified by the "removesim" function as below the current threshold in each iteration. The core of the algorithm lies in the binary search loop (lines 2-10), which iteratively narrows the search interval until its width falls below a predefined "precision" threshold, ensuring high accuracy. In each iteration, the midpoint of the current interval ("mid") is used as a candidate threshold. This threshold is applied to a subset of the matrix ("matrix[start:end,]") using the "remove sim" function, and the algorithm calculates the proportion of elements below the threshold ("keep_portion") and the corresponding set of indices. Based on the comparison between the "retained portion" and the "target ratio," the algorithm dynamically adjusts the boundaries of the search interval ("left" and "right"). Once the termination criterion is met—that is, the interval width is less than the "precision" threshold—the left boundary ("left") is taken as the threshold closest to but not exceeding the "target_ratio," following the inherent properties of binary search. The algorithm then uses this final threshold to call the "remove_sim" function again, obtaining the final index set "final_keeps_set." To prepare the output, the algorithm sorts the indices in "final_keeps_set" (line 12) to maintain consistency and facilitate subsequent processing. Given that indices are subset-dependent, the algorithm converts them into absolute indices by offsetting each index by the "start" value, thus reflecting their position in the entire matrix. Finally, the algorithm returns the sorted absolute index set "sorted_keep_set," which represents the optimal row selection that satisfies the target ratio criterion.

[0155] The adaptive compression threshold search algorithm is implemented as follows:

[0156] Input:matrix,start,end,left,right,precision,target_ratio

[0157] Output: sorted_keep_set

[0158] 1: keeps_set = None

[0159] 2:while left+precision <right do

[0160] 3: mid = left + (right - left) / 2

[0161] 4:keep_portion,keeps_set=remove_sim(matrix[start:end,],threshold=mid)

[0162] 5: if keep_portion <target_ratio then

[0163] 6:left = mid

[0164] 7:else

[0165] 8:right=mid

[0166] 9:end if

[0167] 10:end while

[0168] 11:_,final_keeps_set=remove_sim(matrix[start:end,],threshold=left)

[0169] 12:sorted_keep_set=sorted(final_keeps_set)+start

[0170] 13: return sorted_keep_set

[0171] It is worth noting that by employing the aforementioned adaptive compression threshold search algorithm and mutually exclusive set extraction algorithm, efficient compression and optimization of the sampled corpus can be achieved, thereby improving data quality and model training efficiency. This optimization method not only significantly reduces the size of the dataset and redundant information but also effectively avoids overfitting during model training, improving the model's generalization ability and prediction accuracy.

[0172] Step A11: Compress the corpus samples in the sampled corpus set based on data similarity to obtain the compressed corpus set.

[0173] It should be noted that this implementation uses the bilingual-QuRating data compression method for data compression. The bilingual-QuRating method combines bilingual alignment and a scoring mechanism. It performs bilingual alignment on the corpus samples, assigns a score based on the alignment quality, and then selects corpus samples based on the score. This method helps ensure the diversity and quality of the dataset, further improving the model's performance on cross-language tasks.

[0174] Understandably, the original Qurating model was primarily trained on English, resulting in lower overall scores for Chinese compared to English, thus creating a bias in the ratio of Chinese to English corpora. To address this issue, this implementation uses a data partitioning mechanism based on bilingual classification modeling to resolve the imbalance in the Chinese-English sampling ratio. The overall workflow consists of: data source / language detection and classification, data partitioning, parallel computation, separate bilingual scoring and ranking, and a bilingual mixed sampling mechanism. Furthermore, the original model was limited to a single-machine operation mode and could not handle large-scale data. This implementation, by combining data partitioning methods, adapts the model to scenarios capable of handling large-scale data, significantly improving overall processing efficiency.

[0175] In one feasible implementation, step A11 may include: performing language detection and classification on the corpus samples in the sampled corpus set to obtain a first corpus sample and a second corpus sample, wherein the first corpus sample and the second corpus sample use different languages; performing quality assessment on the first corpus sample and the second corpus sample respectively to obtain corresponding quality assessment scores; sorting the first corpus sample and the second corpus sample based on the quality assessment scores to obtain a sorting result; and performing bilingual mixed sampling on the corpus samples in the sampled corpus set according to the sorting result to obtain a compressed corpus set.

[0176] It should be noted that the first and second corpus samples represent corpora of different languages, such as Chinese and English. By independently scoring and ranking the Chinese and English corpora, the quality of each language corpus can be fully considered during the mixed sampling process. In the bilingual mixed sampling stage, different sampling ratios or priorities can be set according to the ranking results to correct the bias in the original Qurating model regarding the ratio of Chinese and English corpora. This method not only maintains the diversity of the dataset but also ensures a balanced distribution of Chinese and English corpora, thereby improving the model's generalization ability and performance on cross-language tasks. Furthermore, by combining data partitioning methods and parallel computing techniques, the processing efficiency and scalability of the bilingual-QuRating data compression method can be further improved, enabling it to be better applied to the compression and optimization of large-scale multilingual datasets.

[0177] Understandably, language detection and classification of corpus samples can be achieved using a classifier, which can be trained based on machine learning algorithms and is capable of accurately identifying the language type of the corpus samples. The formula for language detection and classification is:

[0178] X en X zh =F lang (X)

[0179] Where X represents all samples, en represents English, zh represents Chinese, and Z represents Chinese. zh Z is the first corpus sample. en For the second corpus sample, F lang (X) is a classifier that divides samples into Chinese and English.

[0180] The quality assessment score for the first corpus sample is:

[0181] Y zh =Rating(X 2h )

[0182] Among them, Y zh Rating() represents the quality assessment score of the first corpus sample.

[0183] The quality assessment score for the second corpus sample is:

[0184] Y en =Rating(X en )

[0185] Among them, Y en Rating() represents the quality assessment score of the second corpus sample.

[0186] It is worth noting that in the bilingual mixed sampling process, by setting a reasonable sampling ratio or priority, the distribution of Chinese and English corpora in the dataset can be effectively balanced, thus solving the imbalance problem of the original Qurating model in the ratio of Chinese and English. The sampling strategy can be dynamically adjusted according to the quality assessment scores of the Chinese and English corpora. For example, when the quality of the Chinese corpus is generally high, the sampling ratio of the Chinese corpus can be appropriately increased; conversely, when the quality of the English corpus is superior, the sampling priority of the English corpus can be increased. This flexible sampling mechanism not only helps to maintain the diversity and quality of the dataset, but also significantly improves the model's generalization ability and prediction accuracy in cross-language tasks.

[0187] Optionally, to optimize the performance of the bilingual-QuRating data compression method, machine learning and deep learning techniques can be combined to iteratively optimize the scoring model and sampling strategy. By continuously introducing new corpus data and training samples, the accuracy and robustness of the scoring model can be continuously improved. Simultaneously, by fine-tuning the sampling strategy, the effectiveness and efficiency of data compression can be further enhanced. This continuous optimization and improvement approach helps ensure that the bilingual-QuRating data compression method can adapt to constantly changing data environments and task requirements, thereby providing users with more efficient and reliable data compression and optimization solutions.

[0188] Step A12: Cluster and compress the corpus samples in the sampled corpus set based on data similarity to obtain the compressed corpus set.

[0189] It should be noted that this embodiment uses the Kmenas clustering compression method based on adaptive sampling for data compression. This method performs cluster analysis on the corpus samples, groups similar corpus samples into one class, and retains only the representative samples of each class, thereby achieving data compression.

[0190] It should be understood that the clustering process first calculates the similarity between corpus samples, and then constructs a clustering tree or uses other clustering algorithms to group the corpus samples based on the similarity. The size of the compressed dataset can be controlled by setting the number of clusters or a clustering threshold. After obtaining the clustering results, one or more of the most representative samples from each cluster are selected as retained samples, while the remaining samples are deleted or merged. This method can preserve the key information and diversity of the dataset while significantly reducing the size and redundancy of the dataset.

[0191] It's worth noting that during the clustering compression stage, an adaptive sampling strategy can be incorporated to dynamically adjust the sampling ratio and priority based on the clustering results and data distribution, thereby further improving the effectiveness and efficiency of data compression. The core of adaptive sampling lies in its ability to intelligently select representative samples based on the density of clusters and the characteristics of data distribution. In densely clustered regions, the number of samples can be appropriately reduced because similar samples provide higher information redundancy; while in more dispersed regions, the number of samples should be increased to ensure the diversity and integrity of the dataset.

[0192] In one feasible implementation, step A12 may include: predicting labels for corpus samples in the sampled corpus set based on a preset clustering model to obtain clustering labels corresponding to each corpus sample; clustering the corpus samples in the sampled corpus set according to the clustering labels to obtain a clustered corpus set; calculating similarity based on the clustered corpus set to obtain the similarity between corpus samples corresponding to each category; and filtering the corpus samples within the clusters based on the similarity to obtain a compressed corpus set.

[0193] It should be noted that a k-means clustering model is trained by randomly sampling a certain proportion of samples, and then the label prediction operation is performed on the test samples, with adaptive sampling. Existing clustering methods only select clustering parameters to train the model based on experience, while this implementation introduces the function of adaptive sampling parameter adjustment after predicting cluster labels, and combines clustering dimensionality reduction visualization to realize an optimized clustering scheme, such as... Figure 3 As shown, Figure 3 A schematic diagram illustrating the sample partitioning effect for different clustering parameters.

[0194] Understandably, the preset clustering model can be a k-means clustering model, which iteratively optimizes clustering parameters and sampling strategies to gradually approach the optimal clustering effect. In each iteration, the model first predicts labels for the corpus samples based on the current clustering parameters, and then clusters the samples based on the prediction results. Next, the model calculates the similarity between samples within each cluster and filters the samples based on similarity, retaining the most representative samples. By continuously adjusting the clustering parameters and sampling strategies, the model can gradually optimize the clustering effect and achieve efficient data compression. Furthermore, combined with clustering dimensionality reduction visualization technology, the clustering results and data distribution characteristics can be intuitively displayed, providing strong support for further optimization of the clustering scheme. This method not only improves the efficiency and quality of data compression but also provides users with a more intuitive and visual evaluation of data compression effects.

[0195] It is worth noting that the formula for cluster labels is:

[0196] Y = kmeans(X,m)

[0197] Where X = [x1, x2, x3, ..., xn] represents the set of all samples, Y = [y1, y2, y3, ..., yn] represents the label of each sample, and m represents the maximum number of cluster categories.

[0198] The formula for adjusting the parameters in adaptive sampling is:

[0199]

[0200] Where F() represents the adaptive sampling function, Y δ This represents the control over the number of samples taken in each category.

[0201] Step A13: Based on data similarity, deduplicatize and compress the corpus samples in the sampled corpus set to obtain the compressed corpus set.

[0202] It should be noted that this implementation uses a compression method based on FAISS similarity deduplication for data compression. This method utilizes the FAISS (Facebook AI Similarity Search) library to achieve efficient similarity calculation and deduplication. FAISS is a library developed by Facebook specifically for handling large-scale vector similarity search and clustering problems. During data compression, feature vectors of the corpus samples are first calculated and input into the FAISS library. Then, FAISS's similarity search function is used to find similar corpus samples. To improve deduplication efficiency, a similarity threshold can be set; only when the similarity between two samples exceeds this threshold are they considered duplicates. After identifying duplicate samples, only one is retained as a representative sample, and the rest are deleted. This method can significantly reduce redundant information in the dataset while retaining key information. It is worth noting that during the deduplication process, domain knowledge and contextual information can be combined to further optimize the deduplication strategy. For example, in certain specific domains, even if the text content of two samples is not exactly the same, if they convey similar information and meaning, they can be considered duplicate samples for deduplication. This deduplication strategy, which incorporates domain knowledge, can further improve the effectiveness and accuracy of data compression. By employing the aforementioned compression method based on FAISS similarity deduplication, efficient deduplication and compression of the sampled corpus can be achieved, thereby further improving data quality and model training efficiency.

[0203] In a feasible implementation, step A13 may include: vectorizing the corpus samples in the sampled corpus set, and taking any corpus sample vector in the sampled corpus set as the target query; calculating the similarity between the target query and other corpus sample vectors in the sampled corpus set respectively; removing other corpus sample vectors with similarity greater than or equal to a preset similarity threshold as redundant corpus sample vectors to obtain a compressed corpus set.

[0204] It should be noted that compression based on FAISS similarity deduplication is a greedy compression scheme. By setting a fixed filtering threshold, samples similar to the target query can be removed one by one, and the above operation can be performed repeatedly in the remaining sample set until the remaining sample set no longer changes, thus obtaining the final compressed corpus.

[0205] As we can understand, vectorization is the process of converting corpus samples into numerical feature vectors, transforming text data into a form that machine learning models can process. During vectorization, methods such as the bag-of-words model, TF-IDF, and word embeddings (e.g., Word2Vec, BERT) can be used to extract features from the corpus samples. These feature vectors capture the lexical, grammatical, and semantic information within the corpus samples, thus providing strong support for subsequent similarity calculations and deduplication operations. Through vectorization, each corpus sample in the sampled corpus can be converted into a high-dimensional vector, and these vectors exhibit certain distributional characteristics in the vector space.

[0206] It's worth noting that similar corpus samples are closer together in the vector space, while dissimilar corpus samples are farther apart. This characteristic allows for efficient similarity calculation and deduplication. When calculating similarity, methods such as cosine similarity and Euclidean distance can be used to measure the degree of similarity between two vectors. When the similarity between two vectors exceeds a preset threshold, their corresponding corpus samples are considered duplicates and deduplication is performed. By continuously iterating and optimizing this process, redundant information in the dataset can be gradually reduced while retaining key information, thus achieving efficient deduplication and compression of the sampled corpus. This method not only improves the efficiency and quality of data compression but also provides more accurate and reliable data support for subsequent model training and evaluation.

[0207] Step S202: Perform parallel data sampling and data merging on the compressed corpus set to obtain a dataset.

[0208] It's important to note that during the parallel data sampling phase, distributed computing frameworks such as Apache Spark or Hadoop can be employed to split the compressed corpus into multiple subsets and execute sampling operations in parallel on different computing nodes. This fully leverages the computing power of multi-core processors and large-scale computing clusters, significantly improving the speed and efficiency of data sampling. Simultaneously, during the sampling process, the sampling ratio and strategy can be dynamically adjusted based on the data distribution characteristics and business needs to ensure the representativeness and diversity of the sampling results. After sampling is complete, the sampling results from each subset can be merged to obtain the final dataset. During the data merging phase, attention must be paid to data deduplication and consistency processing to avoid data redundancy and inconsistencies. By employing the aforementioned parallel data sampling and data merging methods, the efficiency and accuracy of data processing can be further improved.

[0209] In one feasible implementation, step S202 may include: dividing the compressed corpus into multiple data blocks; performing parallel data sampling on the data blocks to obtain multiple corpus subsets; and merging the corpus subsets to obtain a dataset.

[0210] It's important to note that various strategies can be employed during data merging to ensure data integrity and consistency. For example, hash functions can be used to detect and remove duplicate data items while maintaining the original data order. Furthermore, for situations where data is missing or anomaly-prone, interpolation, mean imputation, or machine learning-based methods can be used for data repair and cleaning to improve the dataset's quality and usability. By carefully designing and optimizing the parallel data sampling and merging process, modern computing resources can be fully utilized to achieve efficient, accurate, and reliable data processing.

[0211] It is worth noting that, in order to improve the processing efficiency of large-scale training data, this implementation proposes a data compression acceleration computing framework. This framework aims to achieve efficient data compression and significantly accelerate computing speed by optimizing the data statistics, sampling, embeddings generation and compression processes.

[0212] (1) Framework Overview: The pre-trained data compression framework includes a series of core algorithms and technical processes:

[0213] Accelerate text statistics: Perform statistical analysis on the number of samples, tokens, and files in a large-scale corpus.

[0214] Accelerate Embeddings Generation: Employ multi-machine, multi-GPU parallel text embedding transformation to generate high-quality semantic representations.

[0215] Accelerated Compression: Improve the running speed of the four text compression algorithms mentioned above from the aspects of algorithm, CPU, GPU, and I / O read / write utilization.

[0216] Accelerated sampling: Parallel data sampling and subset merging are performed based on data partitioning to control the dataset allocation and the total number of tokens.

[0217] (2) It is scalable, and after validating the effectiveness of the framework, it was applied to a larger dataset (compressed from 800B tokens to 200B tokens). This advancement demonstrates the method's high scalability and potential for efficiency improvement.

[0218] (3) Improved utilization. A low-cost computing resource solution was adopted: multiple low-configuration CPU and GPU servers were used for parallel data processing; task allocation was optimized by flexibly scheduling and managing more than 1,000 command processes; and the partitioned dataset was read using shared disk storage to ensure efficient data storage and access.

[0219] (4) Process Automation Tools. Given the repetitive labor and fragmented processing logic involved in data preprocessing, this implementation constructs a data processing pipeline to standardize all processing steps. By configuring program scripts to define the entire data pipeline and automatically executing most steps from raw text cleaning to data compression, the need for human intervention is significantly reduced.

[0220] These improvements reduced processing time by four times while maintaining or even improving the performance of the trained model. By leveraging the advantages of AI hardware resources and software frameworks, significant computational acceleration was achieved in processing large-scale text datasets.

[0221] In this embodiment, data compression is performed on the corpus samples in the sampled corpus based on data similarity, which effectively removes redundant data and improves data quality. Then, parallel data sampling and data merging are performed on the compressed corpus, which further improves the efficiency and accuracy of data processing.

[0222] For example, to help understand the implementation flow of the large model optimization method obtained by combining this embodiment with the above embodiment one, please refer to... Figure 4 , Figure 4 A simplified flowchart of a large model optimization method is provided. Specifically: based on the original data, matching and sampling are performed to obtain a sampled dataset, which may include 200B tokens and 800B tokens; data compression operations are performed, including similarity compression rate search, QuRating, and clustering; the compressed data is used for the pre-training process of the large model, which includes continued pre-training and de novo pre-training. The large model includes Llama3-8b and the small model scratch-1.5B, and the training datasets are the entire dataset and the knowledge-based dataset, respectively; the trained large model is then evaluated on downstream tasks.

[0223] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the large model optimization method of this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0224] This application also provides a large model optimization device; please refer to [reference needed]. Figure 5 The large model optimization device includes:

[0225] Module 10 is used to acquire the sampled corpus set.

[0226] Compression module 20 is used to compress the corpus samples in the sampled corpus set based on data similarity to obtain a dataset.

[0227] Training module 30 is used to train the large model based on the dataset to obtain an optimized large language model.

[0228] The large model optimization apparatus provided in this application, employing the large model optimization method described in the above embodiments, can solve the technical problem of data redundancy in large-scale text data affecting model performance. Compared with the prior art, the beneficial effects of the large model optimization apparatus provided in this application are the same as those of the large model optimization method described in the above embodiments, and other technical features in the large model optimization apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0229] Optionally, the compression module 20 is further configured to compress the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set; and to perform parallel data sampling and data merging on the compressed corpus set to obtain a dataset.

[0230] Optionally, the compression module 20 is further configured to perform mutually exclusive compression on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set; perform bilingual quality assessment compression on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set; perform cluster compression on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set; and perform deduplication compression on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set.

[0231] Optionally, the compression module 20 is further configured to calculate the similarity between pairs of corpus samples in the sampled corpus set to obtain the similarity between the pairs of corpus samples; construct a similarity matrix based on the similarity between the pairs of corpus samples; and perform mutual exclusion screening on the corpus samples in the sampled corpus set based on the similarity matrix and a preset similarity threshold to obtain a compressed corpus set.

[0232] Optionally, the compression module 20 is further configured to compare the elements in the similarity matrix with a preset similarity threshold to obtain a comparison result; if the comparison result indicates that the element is less than the preset similarity threshold, obtain the index of the element; perform mutual exclusion filtering on the corpus samples in the sampled corpus set according to the index of the element to obtain a mutually exclusive set; calculate the compression ratio based on the mutually exclusive set to obtain the current compression ratio; if the current compression ratio reaches the target compression ratio, use the mutually exclusive set as the compressed corpus set; if the current compression ratio does not reach the target compression ratio, adjust the preset similarity threshold and re-execute the step of comparing the elements in the similarity matrix with the preset similarity threshold to obtain a comparison result.

[0233] Optionally, the compression module 20 is further configured to perform language detection and classification on the corpus samples in the sampled corpus set to obtain a first corpus sample and a second corpus sample, wherein the languages ​​of the first corpus sample and the second corpus sample are different; to perform quality assessment on the first corpus sample and the second corpus sample respectively to obtain corresponding quality assessment scores; to sort the first corpus sample and the second corpus sample based on the quality assessment scores to obtain a sorting result; and to perform bilingual mixed sampling on the corpus samples in the sampled corpus set according to the sorting result to obtain a compressed corpus set.

[0234] Optionally, the compression module 20 is further configured to predict the labels of the corpus samples in the sampled corpus set based on a preset clustering model to obtain the clustering labels corresponding to each corpus sample; cluster the corpus samples in the sampled corpus set according to the clustering labels to obtain the clustered corpus set; calculate the similarity based on the clustered corpus set to obtain the similarity between the corpus samples corresponding to each category; and filter the corpus samples within the clusters based on the similarity to obtain the compressed corpus set.

[0235] Optionally, the compression module 20 is further configured to vectorize the corpus samples in the sampled corpus set, and use any corpus sample vector in the sampled corpus set as the target query; calculate the similarity between the target query and other corpus sample vectors in the sampled corpus set respectively; and remove other corpus sample vectors with similarity greater than or equal to a preset similarity threshold as redundant corpus sample vectors to obtain the compressed corpus set.

[0236] Optionally, the compression module 20 is further configured to divide the compressed corpus into multiple data blocks; perform parallel data sampling on the data blocks to obtain multiple corpus subsets; and merge the corpus subsets to obtain a dataset.

[0237] Optionally, the acquisition module 10 is further configured to randomly select corpus data from the corpus based on a preset ratio to obtain original corpus, wherein the corpus includes multiple corpus data sets; segment the original corpus into words to obtain segmented original corpus; and sample data based on the segmented original corpus to obtain a sampled corpus set.

[0238] Optionally, the optimization module 30 is further configured to train a large model based on the dataset to obtain a trained large model; evaluate the performance of the trained large model to obtain a performance index; if the performance index reaches the corresponding index threshold, use the trained large model as an optimized large language model; if the performance index does not reach the corresponding index threshold, adjust the training parameters of the large model and re-execute the step of training the large model based on the dataset.

[0239] This application provides a large model optimization device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the large model optimization method in Embodiment 1 above.

[0240] The following is for reference. Figure 6 The diagram illustrates a structural schematic suitable for implementing the large model optimization device of the embodiments of this application. The large model optimization device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The large model optimization device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0241] like Figure 6As shown, the large model optimization device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the large model optimization device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the large model optimization device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows a large model optimization device with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0242] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0243] The large model optimization device provided in this application, employing the large model optimization method described in the above embodiments, can solve the technical problem of data redundancy in large-scale text data affecting model performance. Compared with the prior art, the beneficial effects of the large model optimization device provided in this application are the same as those of the large model optimization method described in the above embodiments, and other technical features of this large model optimization device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0244] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0245] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0246] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the large model optimization method described in the above embodiments.

[0247] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0248] The aforementioned computer-readable storage medium may be included in the large model optimization device; or it may exist independently and not be assembled into the large model optimization device.

[0249] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by a large model optimization device, cause the large model optimization device to: acquire a sampled corpus set; compress the corpus samples in the sampled corpus set based on data similarity to obtain a dataset; and train a large model based on the dataset to obtain an optimized large language model.

[0250] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0251] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0252] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0253] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., computer programs) for executing the above-described large model optimization method. This addresses the technical problem of data redundancy in large-scale text data affecting model performance. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the large model optimization method provided in the above embodiments, and will not be elaborated upon here.

[0254] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the large model optimization method described above.

[0255] The computer program product provided in this application can solve the technical problem of data redundancy in large-scale text data, which affects model performance. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the large model optimization method provided in the above embodiments, and will not be repeated here.

[0256] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

[0257] This invention discloses A1. A large model optimization method, the method comprising:

[0258] Obtain the sampled corpus set;

[0259] Data samples in the sampled corpus are compressed based on data similarity to obtain a dataset.

[0260] The large model is trained based on the dataset to obtain an optimized large language model.

[0261] A2. As described in A1, the step of compressing the corpus samples in the sampled corpus set based on data similarity to obtain a dataset includes:

[0262] Based on data similarity, the corpus samples in the sampled corpus set are compressed to obtain the compressed corpus set;

[0263] The compressed corpus is subjected to parallel data sampling and data merging to obtain a dataset.

[0264] A3. The method as described in A2, wherein the data compression of the corpus samples in the sampled corpus set based on data similarity to obtain the compressed corpus set includes at least one of the following:

[0265] Based on data similarity, mutually exclusive compression is performed on the corpus samples in the sampled corpus set to obtain the compressed corpus set;

[0266] Based on data similarity, the corpus samples in the sampled corpus set are compressed for bilingual quality assessment to obtain the compressed corpus set;

[0267] Based on data similarity, the corpus samples in the sampled corpus are clustered and compressed to obtain the compressed corpus.

[0268] Based on data similarity, the corpus samples in the sampled corpus are deduplicated and compressed to obtain the compressed corpus.

[0269] A4. The method described in A3, wherein the mutual exclusion compression of the corpus samples in the sampled corpus set based on data similarity to obtain the compressed corpus set includes:

[0270] The similarity between corpus sample pairs in the sampled corpus set is calculated.

[0271] A similarity matrix is ​​constructed based on the similarity between the corpus sample pairs;

[0272] Based on the similarity matrix and the preset similarity threshold, mutually exclusive screening is performed on the corpus samples in the sampled corpus set to obtain the compressed corpus set.

[0273] A5. The method described in A4, wherein the mutual exclusion filtering based on the similarity matrix and a preset similarity threshold to obtain the compressed corpus set includes:

[0274] The elements in the similarity matrix are compared with a preset similarity threshold to obtain the comparison result;

[0275] If the comparison result indicates that the element is less than the preset similarity threshold, the index of the element is obtained;

[0276] Based on the index of the element, mutually exclusive filters are performed on the corpus samples in the sampled corpus set to obtain a mutually exclusive set;

[0277] The compression ratio is calculated based on the mutually exclusive set to obtain the current compression ratio;

[0278] If the current compression rate reaches the target compression rate, the mutually exclusive set will be used as the compressed corpus set.

[0279] If the current compression ratio does not reach the target compression ratio, adjust the preset similarity threshold and re-execute the step of comparing the elements in the similarity matrix with the preset similarity threshold to obtain the comparison result.

[0280] A6. As described in A3, the step of compressing the corpus samples in the sampled corpus set based on data similarity for bilingual quality assessment to obtain a compressed corpus set includes:

[0281] Language detection and classification are performed on the corpus samples in the sampled corpus set to obtain a first corpus sample and a second corpus sample, wherein the languages ​​of the first corpus sample and the second corpus sample are different;

[0282] The first and second corpus samples were evaluated for quality, and the corresponding quality evaluation scores were obtained.

[0283] The first and second corpus samples are sorted based on the quality assessment scores to obtain the sorting results;

[0284] Based on the sorting results, bilingual mixed sampling is performed on the corpus samples in the sampled corpus set to obtain a compressed corpus set.

[0285] A7. The method described in A3, wherein clustering and compressing the corpus samples in the sampled corpus set based on data similarity to obtain the compressed corpus set includes:

[0286] Based on a pre-defined clustering model, the labels of the corpus samples in the sampled corpus set are predicted to obtain the clustering labels corresponding to each corpus sample;

[0287] The corpus samples in the sampled corpus set are clustered according to the clustering labels to obtain the clustered corpus set;

[0288] Similarity calculation is performed on the clustered corpus to obtain the similarity between corpus samples corresponding to each category.

[0289] Based on the similarity, the corpus samples within the cluster are filtered to obtain a compressed corpus set.

[0290] A8. As described in A3, the step of deduplicating and compressing the corpus samples in the sampled corpus based on data similarity to obtain a compressed corpus includes:

[0291] Vectorize the corpus samples in the sampled corpus set, and use any corpus sample vector in the sampled corpus set as the target query;

[0292] Calculate the similarity between the target query and the sample vectors of other corpora in the sampled corpus set;

[0293] Other corpus sample vectors with similarity greater than or equal to a preset similarity threshold are removed as redundant corpus sample vectors to obtain a compressed corpus set.

[0294] A9. The method as described in A2, wherein the parallel data sampling and data merging of the compressed corpus to obtain a dataset includes:

[0295] The compressed corpus is divided into multiple data blocks;

[0296] Parallel data sampling is performed on the data block to obtain multiple subsets of the corpus;

[0297] The subset of the corpus is merged to obtain the dataset.

[0298] A10. As described in A1, the step of obtaining the sampled corpus set includes:

[0299] Based on a preset ratio, corpus data is randomly selected from the corpus to obtain the original corpus, wherein the corpus includes multiple corpus data sets;

[0300] The original corpus is segmented into words to obtain the segmented original corpus;

[0301] Data sampling is performed on the original corpus after word segmentation to obtain a sampled corpus set.

[0302] A11. The method as described in A1, wherein training the large model based on the dataset to obtain an optimized large language model includes:

[0303] The large model is trained based on the dataset to obtain the trained large model;

[0304] The performance of the trained large model is evaluated to obtain performance metrics.

[0305] If the performance index reaches the corresponding index threshold, the trained large model will be used as the optimized large language model.

[0306] If the performance metric fails to reach the corresponding threshold, adjust the training parameters of the large model and re-execute the step of training the large model based on the dataset.

[0307] The present invention also discloses B12. A large model optimization device, the large model optimization device comprising:

[0308] The acquisition module is used to acquire the sampled corpus set;

[0309] The compression module is used to compress the corpus samples in the sampled corpus based on data similarity to obtain the dataset;

[0310] The training module is used to train the large model based on the dataset to obtain an optimized large language model.

[0311] B13. The apparatus as described in B12, wherein the compression module is further configured to compress the corpus samples in the sampled corpus set based on data similarity, to obtain a compressed corpus set;

[0312] The compressed corpus is subjected to parallel data sampling and data merging to obtain a dataset.

[0313] B14. The apparatus as described in B13, wherein the compression module is further configured to perform mutually exclusive compression on the corpus samples in the sampled corpus set based on data similarity, to obtain a compressed corpus set;

[0314] Based on data similarity, the corpus samples in the sampled corpus set are compressed for bilingual quality assessment to obtain the compressed corpus set;

[0315] Based on data similarity, the corpus samples in the sampled corpus are clustered and compressed to obtain the compressed corpus.

[0316] Based on data similarity, the corpus samples in the sampled corpus are deduplicated and compressed to obtain the compressed corpus.

[0317] B15. The apparatus as described in B14, wherein the compression module is further configured to perform similarity calculation on pairs of corpus samples in the sampled corpus set to obtain the similarity between pairs of corpus samples;

[0318] A similarity matrix is ​​constructed based on the similarity between the corpus sample pairs;

[0319] Based on the similarity matrix and the preset similarity threshold, mutually exclusive screening is performed on the corpus samples in the sampled corpus set to obtain the compressed corpus set.

[0320] B16. The apparatus as described in B15, wherein the compression module is further configured to compare the elements in the similarity matrix with a preset similarity threshold to obtain a comparison result;

[0321] If the comparison result indicates that the element is less than the preset similarity threshold, the index of the element is obtained;

[0322] Based on the index of the element, mutually exclusive filters are performed on the corpus samples in the sampled corpus set to obtain a mutually exclusive set;

[0323] The compression ratio is calculated based on the mutually exclusive set to obtain the current compression ratio;

[0324] If the current compression rate reaches the target compression rate, the mutually exclusive set will be used as the compressed corpus set.

[0325] If the current compression ratio does not reach the target compression ratio, adjust the preset similarity threshold and re-execute the step of comparing the elements in the similarity matrix with the preset similarity threshold to obtain the comparison result.

[0326] B17. The apparatus as described in B14, wherein the compression module is further configured to perform language detection and classification on the corpus samples in the sampled corpus set to obtain a first corpus sample and a second corpus sample, wherein the languages ​​of the first corpus sample and the second corpus sample are different;

[0327] The first and second corpus samples were evaluated for quality, and the corresponding quality evaluation scores were obtained.

[0328] The first and second corpus samples are sorted based on the quality assessment scores to obtain the sorting results;

[0329] Based on the sorting results, bilingual mixed sampling is performed on the corpus samples in the sampled corpus set to obtain a compressed corpus set.

[0330] B18. The apparatus as described in B14, wherein the compression module is further configured to predict the labels of the corpus samples in the sampled corpus set based on a preset clustering model, and obtain the clustering labels corresponding to each corpus sample;

[0331] The corpus samples in the sampled corpus set are clustered according to the clustering labels to obtain the clustered corpus set;

[0332] Similarity calculation is performed on the clustered corpus to obtain the similarity between corpus samples corresponding to each category.

[0333] Based on the similarity, the corpus samples within the cluster are filtered to obtain a compressed corpus set.

[0334] The present invention also discloses C19. A large model optimization device, the large model optimization device comprising: a memory, a processor, and a large model optimization program stored in the memory and executable on the processor, the large model optimization program being configured to implement the large model optimization method as described above.

[0335] The present invention also discloses D20. A storage medium storing a large model optimization program, wherein the large model optimization program, when executed by a processor, implements the large model optimization method as described above.

Claims

1. A large model optimization method, characterized in that, The method includes: Obtain the sampled corpus set; Data samples in the sampled corpus are compressed based on data similarity to obtain a dataset. The large model is trained based on the dataset to obtain an optimized large language model; The data dataset obtained by compressing the corpus samples in the sampled corpus based on data similarity includes: Based on data similarity, the corpus samples in the sampled corpus set are compressed to obtain the compressed corpus set; The compressed corpus is subjected to parallel data sampling and data merging to obtain a dataset; The data compression of corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set includes at least: Based on data similarity, mutually exclusive compression is performed on the corpus samples in the sampled corpus set to obtain the compressed corpus set; The method of mutually exclusive compression of corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set includes: The similarity between corpus sample pairs in the sampled corpus set is calculated. A similarity matrix is ​​constructed based on the similarity between the corpus sample pairs; The elements in the similarity matrix are compared with a preset similarity threshold to obtain the comparison result; If the comparison result indicates that the element is less than the preset similarity threshold, the index of the element is obtained; Based on the index of the element, mutually exclusive filters are performed on the corpus samples in the sampled corpus set to obtain a mutually exclusive set; The compression ratio is calculated based on the mutually exclusive set to obtain the current compression ratio; If the current compression rate reaches the target compression rate, the mutually exclusive set will be used as the compressed corpus set. If the current compression ratio does not reach the target compression ratio, adjust the preset similarity threshold and re-execute the step of comparing the elements in the similarity matrix with the preset similarity threshold to obtain the comparison result.

2. The method as described in claim 1, characterized in that, The data compression of corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set includes at least one of the following: Based on data similarity, the corpus samples in the sampled corpus set are compressed for bilingual quality assessment to obtain the compressed corpus set; Based on data similarity, the corpus samples in the sampled corpus are clustered and compressed to obtain the compressed corpus. Based on data similarity, the corpus samples in the sampled corpus are deduplicated and compressed to obtain the compressed corpus.

3. The method as described in claim 2, characterized in that, The method of compressing the corpus samples in the sampled corpus based on data similarity to obtain a compressed corpus includes: Language detection and classification are performed on the corpus samples in the sampled corpus set to obtain a first corpus sample and a second corpus sample, wherein the languages ​​of the first corpus sample and the second corpus sample are different; The first and second corpus samples were evaluated for quality, and the corresponding quality evaluation scores were obtained. The first and second corpus samples are sorted based on the quality assessment scores to obtain the sorting results; Based on the sorting results, bilingual mixed sampling is performed on the corpus samples in the sampled corpus set to obtain a compressed corpus set.

4. The method as described in claim 2, characterized in that, The clustering and compression of corpus samples in the sampled corpus based on data similarity yields a compressed corpus, including: Based on a pre-defined clustering model, the labels of the corpus samples in the sampled corpus set are predicted to obtain the clustering labels corresponding to each corpus sample; The corpus samples in the sampled corpus set are clustered according to the clustering labels to obtain the clustered corpus set; Similarity calculation is performed on the clustered corpus to obtain the similarity between corpus samples corresponding to each category. Based on the similarity, the corpus samples within the cluster are filtered to obtain a compressed corpus set.

5. A large model optimization device, characterized in that, The large model optimization device includes: The acquisition module is used to acquire the sampled corpus set; The compression module is used to compress the corpus samples in the sampled corpus based on data similarity to obtain the dataset; The training module is used to train the large model based on the dataset to obtain an optimized large language model; The compression module is further configured to compress the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set; to perform parallel data sampling and data merging on the compressed corpus set to obtain a dataset; the step of compressing the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set includes at least: performing mutually exclusive compression on the corpus samples in the sampled corpus set based on data similarity to obtain a compressed corpus set; The compression module is further configured to: calculate the similarity between pairs of corpus samples in the sampled corpus set to obtain the similarity between the pairs of corpus samples; construct a similarity matrix based on the similarity between the pairs of corpus samples; compare the elements in the similarity matrix with a preset similarity threshold to obtain a comparison result; if the comparison result is that the element is less than the preset similarity threshold, obtain the index of the element; perform mutual exclusion filtering on the corpus samples in the sampled corpus set according to the index of the element to obtain a mutually exclusive set; calculate the compression ratio based on the mutually exclusive set to obtain the current compression ratio; if the current compression ratio reaches the target compression ratio, use the mutually exclusive set as the compressed corpus set; if the current compression ratio does not reach the target compression ratio, adjust the preset similarity threshold and re-execute the step of comparing the elements in the similarity matrix with the preset similarity threshold to obtain a comparison result.

6. A large model optimization device, characterized in that, The large model optimization device includes: a memory, a processor, and a large model optimization program stored in the memory and executable on the processor, the large model optimization program being configured to implement the large model optimization method as described in any one of claims 1 to 4.

7. A storage medium, characterized in that, The storage medium stores a large model optimization program, which, when executed by a processor, implements the large model optimization method as described in any one of claims 1 to 4.

Images