Large model training data set optimization method and system

Abstract

The invention provides a large model training data set optimization method and system, and relates to the field of data set optimization. According to the method, an experimental design method is combined, the advantages of an orthogonal experimental scheme design method and a near-orthogonal Latin hypercube design method are adopted, and the design process of typical large model training data optimization meeting near orthogonality and near uniformity is simplified; according to the method, through a pre-constructed 20-factor and 9-factor near-orthogonal and near-uniform basic table meeting near-orthogonal and near-uniform requirements, full arrangement of typical factor data items of all factors in a large model training data set is constructed, and data with the same factors as the typical factor data items are selected from an original training data set and used for standard and training. The number of data items of a large model training data set is greatly reduced, the design work of the training data set is optimized, and the training efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of dataset optimization, and in particular to a method and system for optimizing a large model training dataset. Background Art

[0002] Currently, large-scale pre-training models face dual bottlenecks in their rapid development: first, ultra-large-scale training datasets lead to an exponential increase in computing power requirements. A single training task requires the deployment of a computing cluster consisting of thousands of GPUs, and the storage requirements exceed the petabyte level. Second, the quality of data annotation fluctuates significantly, with redundant samples accounting for more than 30%, seriously restricting the efficiency of model convergence.

[0003] It is worth noting that when big model technology penetrates into enterprise-level scenarios, if various industry themes compete to develop customized vertical big models, it will trigger a geometric growth in computing power demand and training cycles, resulting in serious resource mismatch. Summary of the Invention

[0004] (1) Technical problems solved

[0005] In response to the shortcomings of the existing technology, the present invention provides a large model training data set optimization method and system, which solves the technical problem of excessive computing power requirements caused by the huge size of the existing large model training data set.

[0006] (2) Technical solution

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0008] In a first aspect, the present invention provides a method for optimizing a large model training data set, comprising:

[0009] S1. Obtain the factor data and factor ranges of the large model training data set W and establish the large model training data factor set H;

[0010] S2. Determine whether the factor value range of each factor in the factor set H can be converted into 9 value situations. If so, proceed to S3; otherwise, end.

[0011] S3. Determine whether the number of factors is less than 20. If so, update the set H to an empty set and proceed directly to S4. Otherwise, select 20 factors from the factor set H and update the set H = H - {h1, ..., h20}, and proceed to S4. Where h1, ..., h20 represents 20 factors selected from the factor set H.

[0012] S4. Determine whether the number of selected factors m is less than 3. If so, perform a full permutation of the m factors and 9 factors to obtain the data item U i, where i represents the number of cycles, and then proceed to S9; Is the number of factors m greater than 3 and less than 20? If so, select the same number of columns from the 20-factor, 9-factor nearly orthogonal, nearly uniform basic table, and then proceed to S5; Otherwise, directly use the 20-factor, 9-factor nearly orthogonal, nearly uniform basic table, and then proceed to S5;

[0013] S5. Determine the mapping relationship between the factors and the columns selected from the 20-factor 9-factor nearly orthogonal and nearly uniform basic table;

[0014] S6. Determine the mapping relationship between each variable value and the numerical value in each column;

[0015] S7. According to the mapping relationship, replace the values ​​in the selected 20 factors and 9 factors near-orthogonal and near-uniform basic table with all factor values, and output 512 groups of data items U of the 20 factors selected to meet the near-orthogonality and near-uniformity. i ;

[0016] S8. Determine whether the set H is empty. If not, go to S3; if empty, go to S9.

[0017] S9, the U generated in each cycle i Cartesian product or nested construction of orthogonal Latin hypercube is performed to form a typical data set U, and typical data items d are selected, which satisfy d∈W∩U, to form a training data set D;

[0018] Among them, the basic table of the 20-factor 9-factor near-orthogonal and near-uniform design is a table with 20 columns and 512 rows. The basic table of the 20-factor 9-factor near-orthogonal and near-uniform design has near-orthogonality and near-uniformity.

[0019] Preferably, the method for constructing the basic table of the 20-factor 9-factor discrete variable nearly orthogonal and nearly uniform design includes:

[0020] Step 1: Given a 512*1 vector;

[0021] Step 2: Add a 512*1 vector to form a 512*2 matrix. Calculate the orthogonality of the two vectors. If the result is less than 0.05, add the vector of the 512*1 vector. Otherwise, find a new 512*1 vector and calculate it again.

[0022] Step 3: Add a 512*1 vector to form a 512*3 matrix and calculate the orthogonality of the two vectors. Here, we need to calculate the orthogonality of three two-vectors. If the results are all less than 0.05, keep the matrix. Otherwise, start again to find another 512*1 vector and calculate it. And so on, find and build a 512*20 matrix.

[0023] Step 4: Use the two-dimensional plane distribution diagram to determine the uniformity of the design content. If it is uniform, retain the set of 512*20 matrices, otherwise start again from step 1.

[0024] Preferably, the near-orthogonality of the basic table of the 20-factor 9-factor discrete variable near-orthogonal and near-uniform design is verified by calculating the maximum absolute value of the correlation coefficient, specifically including:

[0025] The maximum absolute value of the correlation coefficient is recorded as ρ 相关系数绝对值的最大值 , the absolute value of the correlation coefficient is recorded as |ρ ij |, for any two different column vectors X i and X j , and i≠j, then the maximum absolute value of the correlation coefficient is:

[0026]

[0027] Among them, ρ 相关系数绝对值的最大值 The smaller the value of , the better the orthogonality. When the value is less than or equal to 0.05, it is considered to be a nearly orthogonal design. i and X j ρ 相关系数绝对值的最大值 = 0, indicating orthogonality; l represents the column vector column vector X i and X j The first line of .

[0028] Preferably, the mapping relationship between the determination factors and the columns selected from the 20-factor and 9-factor nearly orthogonal and nearly uniform basic table includes:

[0029] Suppose there is a set A with m variable factors, where A = {1, ..., m}, and m columns are selected from the 20-factor 9-factor nearly orthogonal and nearly uniform basic table to form a set B, where B = {1, ..., m}, and the relationship between sets A and B is f:A→B. f(a1)≠f(a2), and b∈B∧a∈A→f(a)=b; establish variable mapping relationship { i ,b j >}, where a i ∈A,b j ∈B,i={1,…,m},j={1,…,m}.

[0030] Preferably, determining the mapping relationship between each variable value and the numerical value in each column includes:

[0031] a i ={a i1 ,a i2 ,a i3 ,a i4 ,a i5 ,a​i6 ,a i7 ,a i8 ,a i9}, b j ={b j1 ,b j2 ,b j3 ,b j4 ,b j5 ,b j6 ,b j7 ,b j8 ,b j9}, i={1,…,

[0032] m}, j = {1, ..., m}, determine the mapping relationship between each variable value and the corresponding column of the 9-factor basic table, { i1 ,b j1 >} or { i1 ,b j2 >} or { i1 ,b j3 >} or { i1 ,b j4 >} or { i1 ,b j5 >} or { i1 ,b j6 >} or { i1 ,b j7 >} or { i1 ,b j8 >} or { i1 ,b j9 >}.

[0033] In a second aspect, the present invention provides a large model training data set optimization system, comprising:

[0034] A large model training data set acquisition module is used to execute S1, obtain the factor data and factor ranges of the large model training data set W, and establish a large model training data factor set H;

[0035] The first judgment module is used to execute S2 and judge whether the factor value range of each factor in the factor set H can be converted into 9 value situations. If so, enter S3; otherwise, end;

[0036] The second judgment module is used to execute S3 and determine whether the number of factors is less than 20. If so, the set H is updated to be empty and the process directly proceeds to S4. Otherwise, 20 factors are selected from the factor set H and the set H = H - {h1, ..., h20} is updated, and the process proceeds to S4. Where h1, ..., h20 represents 20 factors selected from the factor set H.

[0037] ​​​​​​​​​The third judgment module is used to execute S4 and judge whether the number of selected factors m is less than 3. If so, the m factors and 9 factors are fully arranged to obtain the data item U i , where i represents the number of cycles, and then proceed to S9; Is the number of factors m greater than 3 and less than 20? If so, select the same number of columns from the 20-factor, 9-factor nearly orthogonal, nearly uniform basic table, and then proceed to S5; Otherwise, directly use the 20-factor, 9-factor nearly orthogonal, nearly uniform basic table, and then proceed to S5;

[0038] A first mapping relationship determination module is configured to execute S5, determine the mapping relationship between the factors and the columns selected from the 20-factor and 9-factor nearly orthogonal and nearly uniform basic table;

[0039] A second mapping relationship determination module, configured to execute S6, determining a mapping relationship between each variable value and a numerical value in each column;

[0040] The first output module is used to execute S7, replace the values ​​in the selected 20 factors and 9 factors near-orthogonal and near-uniform basic table with all factor values ​​according to the mapping relationship, and output 512 groups of data items U of the 20 factors selected to meet the near-orthogonality and near-uniformity. i ;

[0041] The fourth judgment module is used to judge whether the set H is empty. If not, the process goes to S3; if it is empty, the process goes to S9;

[0042] The second output module is used to execute S9 and convert the U generated in each cycle into i Cartesian product or nested construction of orthogonal Latin hypercube is performed to form a typical data set U, and typical data items d are selected, which satisfy d∈W∩U, to form a training data set D;

[0043] Among them, the basic table of the 20-factor 9-factor near-orthogonal and near-uniform design is a table with 20 columns and 512 rows. The basic table of the 20-factor 9-factor near-orthogonal and near-uniform design has near-orthogonality and near-uniformity.

[0044] Preferably, the method for constructing the basic table of the 20-factor 9-factor discrete variable nearly orthogonal and nearly uniform design includes:

[0045] Step 1: Given a 512*1 vector;

[0046] Step 2: Add a 512*1 vector to form a 512*2 matrix. Calculate the orthogonality of the two vectors. If the result is less than 0.05, add the vector of the 512*1 vector. Otherwise, find a new 512*1 vector and calculate it again.

[0047] Step 3: Add a 512*1 vector to form a 512*3 matrix and calculate the orthogonality of the two vectors. Here, we need to calculate the orthogonality of three two-vectors. If the results are all less than 0.05, keep the matrix. Otherwise, start again to find another 512*1 vector and calculate it. And so on, find and build a 512*20 matrix.

[0048] Step 4: Use the two-dimensional plane distribution diagram to determine the uniformity of the design content. If it is uniform, retain the set of 512*20 matrices, otherwise start again from step 1.

[0049] Preferably, the mapping relationship between the determination factors and the columns selected from the 20-factor and 9-factor nearly orthogonal and nearly uniform basic table includes:

[0050] Suppose there is a set A with m variable factors, where A = {1, ..., m}, and m columns are selected from the 20-factor 9-factor nearly orthogonal and nearly uniform basic table to form a set B, where B = {1, ..., m}, and the relationship between sets A and B is f:A→B. f(a1)≠f(a2), and b∈B∧a∈A→f(a)=b; establish variable mapping relationship { i ,b j >}, where a i ∈A,b j ∈B,i={1,…,m},j={1,…,m}.

[0051] In a third aspect, the present invention provides a computer-readable storage medium storing a computer program for optimizing a large model training dataset, wherein the computer program enables a computer to execute the large model training dataset optimization method as described above.

[0052] In a fourth aspect, the present invention provides an electronic device, comprising:

[0053] One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for executing the large model training data set optimization method as described above.

[0054] (3) Beneficial effects

[0055] The present invention provides a method and system for optimizing large model training datasets. Compared with existing technologies, it has the following advantages:

[0056] ​This method combines the advantages of orthogonal experimental design methods and near-orthogonal Latin hypercube design methods, and simplifies the design process for experimental data that meets near-orthogonality and near-uniformity requirements. By pre-constructing a 20-factor, 9-factor near-orthogonal, near-uniform basic table that meets the near-orthogonality and near-uniformity requirements, it provides a typical large-model training data set optimization scheme that meets near-orthogonality and near-uniformity when the amount of large-model training data is large, covering all the characteristics of the large-model training data set. This greatly simplifies the number of data items in the large-model training data set, optimizes the training data set design process, and improves training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0058] Figure 1 A block diagram of a large model training data set optimization method according to an embodiment of the present invention;

[0059] Figure 2 A two-dimensional plane distribution effect diagram is designed for the uniformity of 20 groups of discrete variables with a variation range of 9 ({1,2,3,4,5,6,7,8,9}). DETAILED DESCRIPTION

[0060] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0061] The embodiments of the present application provide a method and system for optimizing large model training data sets, thereby solving the technical problem of excessive computing power requirements caused by the huge size of existing large model training data sets, and reducing the data size while retaining data characteristics.

[0062] The technical solution in the embodiments of the present application is to solve the above technical problems, and the overall idea is as follows:

[0063] As big model technology rapidly penetrates enterprise scenarios, the rush to develop customized vertical big models across various industries will lead to exponential growth in computing power requirements and training cycles, exacerbating resource mismatches. Therefore, it's crucial to build a comprehensive solution system encompassing training data compression, computing efficiency improvements, dynamic computing power scheduling, storage optimization, and data quality enhancement. By establishing a data quality assessment system to address this at the source, not only can the demand for labeled data be reduced by 40%-60%, but computing and storage overhead can also be reduced by over 30%, expanding the application space for large-scale pre-trained models. In building this system, the optimized design of large-model training datasets is a key step, directly impacting the overall effectiveness of the solution. The effectiveness of the optimized design of large-model training datasets directly depends on the performance of the experimental design method.

[0064] Currently, commonly used experimental design methods include completely randomized design, factorial design, orthogonal design, Latin hypercube design, uniform design, and near-orthogonal Latin hypercube design. These methods are important tools for optimizing datasets, but they all have limitations that hinder the efficient optimization of large-scale model training datasets. For example, while orthogonal design can eliminate correlations and interactions between factors, when dealing with large samples, the experimental results fail to fully cover the entire experimental space, resulting in insufficient representativeness. Furthermore, the solutions generated by this method are limited to a preset level and cannot be guaranteed to be optimal within the experimental range. While orthogonal design allows for the direct use of readily available orthogonal tables, saving on data table design effort, it still cannot compensate for the lack of representativeness of the experimental sample space. Latin hypercube design is complex to construct, and while it can reduce variable correlation, it also suffers from the problem of insufficient representativeness of the experimental sample space. Near-orthogonal Latin hypercube design addresses the orthogonality and uniformity deficiencies of orthogonal, Latin hypercube, and uniform designs, but its complex computational process increases the difficulty of practical application.

[0065] In summary, these existing experimental design methods require a lot of calculation and verification work during the application process, which brings inconvenience to the design and application of large model training data set optimization.

[0066] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0067] The embodiment of the present invention provides a method for optimizing a large model training data set, such as Figure 1 As shown, the following steps are included:

[0068] S1. Obtain the factor data and factor ranges of the large model training data set W and establish the large model training data factor set H;

[0069] S2. Determine whether the factor value range of each factor in the factor set H can be converted into 9 value situations. If so, proceed to S3; otherwise, end.

[0070] S3. Determine whether the number of factors is less than 20. If so, update the set H to be empty and proceed directly to S4. Otherwise, select 20 factors (h1,…,h20) from the factor set H and update the set H (H = H - {h1,…,h20}), and proceed to S4.

[0071] S4. Determine whether the number of selected factors m is less than 3. If so, perform a full permutation of the m factors and 9 factors to obtain the data item U i (i represents the number of cycles, the first time is 1), go to S9; whether the number of factors m is greater than 3 and less than 20, if so, select the same number of columns in sequence from the 20-factor 9-factor nearly orthogonal and nearly uniform basic table, go to S5; otherwise, directly use the 20-factor 9-factor nearly orthogonal and nearly uniform basic table, go to S5; it should be noted that, in this step, the reason for judging whether the number of selected factors m is less than 3 is that when the number of factors is less than or equal to 3, the number of samples obtained by full permutation is less than 512. Therefore, if it is less than 3, this basic table is not needed, and the permutations and combinations can be performed directly, which not only generates a small number of samples but also covers the entire space.

[0072] S5. Determine the mapping relationship between the factors and the columns selected from the 20-factor 9-factor nearly orthogonal and nearly uniform basic table;

[0073] S6. Determine the mapping relationship between each variable value and the numerical value in each column;

[0074] S7. According to the mapping relationship, replace the values ​​in the selected 20 factors and 9 factors near-orthogonal and near-uniform basic table with all factor values, and output 512 groups of data items U of the 20 factors selected to meet the near-orthogonality and near-uniformity. i ;

[0075] S8. Determine whether the set H is empty. If not, go to S3; if empty, go to S9.

[0076] S9, the U generated in each cycle i (i≥1) performs Cartesian product (which can ensure orthogonality but cannot ensure uniformity) or nested construction of orthogonal Latin hypercube (which ensures both orthogonality and uniformity) to form a typical data set U, selects typical data items d that satisfy d∈W∩U, and forms a training data set D for training or labeling.

[0077] The base table for the 20-factor, 9-factor near-orthogonal, near-uniform design exhibits excellent near-orthogonality, ensuring the independence of the design schemes, and good near-uniformity, indicating the representativeness of each scheme within the experimental design space. Therefore, the experimental data generated using this table provide good coverage of the experimental input variables.

[0078] The embodiment of the present invention combines the advantages of the orthogonal experimental design method and the nearly orthogonal Latin hypercube design method, and simplifies the design process of experimental data that meets the nearly orthogonal and nearly uniform requirements. By using a pre-constructed 20-factor, 9-factor nearly orthogonal, nearly uniform basic table that meets the nearly orthogonal and nearly uniform requirements, when the number of enumerations of the experiment exceeds 512 times (512<9^3, that is, when the number of variables exceeds 2, the number of experimental design schemes enumerated will exceed 512. This method can control the experimental design schemes to 512, reducing the number of design schemes while ensuring experimental coverage), a design scheme that only requires 512 times and meets the nearly orthogonal and nearly uniform requirements is provided, covering all the characteristics of multi-factor 9-factor enumeration data. This greatly optimizes the experimental design work and improves the efficiency of scheme design.

[0079] It should be noted that the experimental scheme obtained can be used for simulation experiments or engineering experiments.

[0080] The construction method of the basic table of 20-factor 9-factor discrete variable nearly orthogonal and nearly uniform design is as follows:

[0081] Step 1: Given a 512*1 vector;

[0082] Step 2: Add a 512*1 vector to form a 512*2 matrix. Calculate the orthogonality of the two vectors. If the result is less than 0.05, add the vector of the 512*1 vector. Otherwise, find a new 512*1 vector and calculate it again.

[0083] Step 3: Add a 512*1 vector to form a 512*3 matrix and calculate the orthogonality of the two vectors. Here, we need to calculate the orthogonality of three two-vectors. If the results are all less than 0.05, keep the matrix. Otherwise, start again to find another 512*1 vector and calculate it. And so on, find and build a 512*20 matrix.

[0084] Step 4: Use the 2D distribution graph to determine the uniformity of the design. If uniform, retain the 512x20 matrix. Otherwise, restart from step 1. The uniformity criterion is as follows: if the 2D distribution graph between any two variables does not approximate an "X" shape, that is, there are no areas without sampling points, then the design is uniform.

[0085] It should be noted that the basic table of the 20-factor 9-factor discrete variable nearly orthogonal and nearly uniform design in the embodiment of the present invention is shown in Table 1:

[0086] Table 1 Basic table of 20-factor 9-factor discrete variable nearly orthogonal and nearly uniform design

[0087]

[0088]

[0089]

[0090]

[0091]

[0092]

[0093]

[0094]

[0095]

[0096]

[0097]

[0098]

[0099]

[0100] Methods for proving the near-orthogonality of Table 1 include:

[0101] The near-orthogonality of this near-orthogonal and near-uniform basic table is verified by calculating the maximum absolute value of the correlation coefficient. The maximum absolute value of the correlation coefficient is denoted as ρ 相关系数绝对值的最大值 , the absolute value of the correlation coefficient is recorded as |ρ ij |, for any two different column vectors X i and X j , and i≠j, then the maximum absolute value of the correlation coefficient is:

[0102]

[0103] Among them, ρ 相关系数绝对值的最大值 The smaller the value of , the better the orthogonality. When the value is less than or equal to 0.05, it is considered to be a nearly orthogonal design. i and X j ρ 相关系数绝对值的最大值 = 0, it means orthogonal. l represents the column vector column vector X i and X j The first line of .

[0104] According to the maximum value calculation method of the absolute value of the correlation coefficient, the maximum value set of the absolute value of the correlation coefficient of all columns in the basic table is obtained, that is, ρ = {0.034558662, 0.03473167, 0.031254537, 0.033067408, 0.032750242, 0.035536293, 0.029980049, 0.031932812, 0 .032598396, 0.033804627, 0.032050843, 0.028592505, 0.035518458, 0.032419739, 0.032972227, 0.025396651, 0.026062323, 0.029864059, 0.02962178, 0.02939236}. Where i represents the maximum absolute value of the correlation coefficient between the i-th column and other columns in the base table, where 1≤i≤20. Therefore, it can be obtained that the columns in this matrix table have good near-orthogonality.

[0105] The uniformity proof includes: the dispersion and uniformity of the experimental design data can be intuitively demonstrated by designing the uniformity of 20 discrete variables with a range of 9 ({1, 2, 3, 4, 5, 6, 7, 8, 9}). Figure 2 shown.

[0106] The following is a detailed description through specific embodiments:

[0107] This example is for a large model training data set with 20 factors and a variation factor of 9 for each factor variable. The design process is as follows:

[0108] S201, obtaining 20 experimental variable factor data and the value range of the variable change factor of the large model training data set;

[0109] S202: Determine whether the value range of each variable change factor can be converted into 9 value conditions. If yes, execute S203; otherwise, end.

[0110] S203, determine the mapping relationship between the experimental variable factors (set A) and the columns in the selected basic table (set B) { i ,b j >}, where a i ∈A,b j ∈B,i={1,…,20},j={1,…,20};

[0111] S204, determine the mapping relationship between each variable value and the value in each column { <a​i1 ,b j1 >} or { i1 ,b j2 >} or { i1 ,b j3 >} or { i1 ,b j4 >} or { i1 ,b j5 >} or { i1 ,b j6 >} or { i1 ,b j7 >} or { i1 ,b j8 >} or { i1 ,b j9 >}, where a i ={a i1 ,a i2 ,a i3 ,a i4 ,a i5 ,a i6 ,a i7 ,a i8 ,a i9}, b j ={b j1 ,b j2 ,b j3 ,b j4 ,b j5 ,b j6 ,b j7 ,b j8 ,b j9}, i={1,…,20}, j={1,…,20};

[0112] S205, according to the mapping relationship, replace the values ​​in the selected 20-factor 9-factor nearly orthogonal and nearly uniform basic table with all experimental variable values, and output 512 groups of typical data selection schemes for large model training data sets that meet nearly orthogonal and nearly uniform properties. The two-dimensional distribution effect of the data in the typical data selection scheme is as follows: Figure 2 As shown in the figure, select the dataset that is the intersection of this solution dataset and the large model training dataset.

[0113] An embodiment of the present invention further provides a large model training data set optimization system, comprising:

[0114] A large model training data set acquisition module is used to execute S1, obtain the factor data and factor ranges of the large model training data set W, and establish a large model training data factor set H;

[0115] ​​​​​​​​The first judgment module is used to execute S2 and judge whether the factor value range of each factor in the factor set H can be converted into 9 value situations. If so, enter S3; otherwise, end;

[0116] The second judgment module is used to execute S3 and determine whether the number of factors is less than 20. If so, the set H is updated to be empty and the process directly proceeds to S4. Otherwise, 20 factors are selected from the factor set H and the set H = H - {h1, ..., h20} is updated, and the process proceeds to S4. Where h1, ..., h20 represents 20 factors selected from the factor set H.

[0117] The third judgment module is used to execute S4 and judge whether the number of selected factors m is less than 3. If so, the m factors and 9 factors are fully arranged to obtain the data item U i , where i represents the number of cycles, and then proceed to S9; Is the number of factors m greater than 3 and less than 20? If so, select the same number of columns from the 20-factor, 9-factor nearly orthogonal, nearly uniform basic table, and then proceed to S5; Otherwise, directly use the 20-factor, 9-factor nearly orthogonal, nearly uniform basic table, and then proceed to S5;

[0118] A first mapping relationship determination module is configured to execute S5, determine the mapping relationship between the factors and the columns selected from the 20-factor and 9-factor nearly orthogonal and nearly uniform basic table;

[0119] A second mapping relationship determination module, configured to execute S6, determining a mapping relationship between each variable value and a numerical value in each column;

[0120] The first output module is used to execute S7, replace the values ​​in the selected 20 factors and 9 factors near-orthogonal and near-uniform basic table with all factor values ​​according to the mapping relationship, and output 512 groups of data items U of the 20 factors selected to meet the near-orthogonality and near-uniformity. i ;

[0121] The fourth judgment module is used to judge whether the set H is empty. If not, the process goes to S3; if it is empty, the process goes to S9;

[0122] The second output module is used to execute S9 and convert the U generated in each cycle into i Cartesian product or nested construction of orthogonal Latin hypercube is performed to form a typical data set U, and typical data items d are selected, which satisfy d∈W∩U to form a training data set D for training or labeling.

[0123] Among them, the basic table of the 20-factor 9-factor near-orthogonal and near-uniform design is a table with 20 columns and 512 rows. The basic table of the 20-factor 9-factor near-orthogonal and near-uniform design has near-orthogonality and near-uniformity.

[0124] It can be understood that the large model training dataset optimization system provided in an embodiment of the present invention corresponds to the above-mentioned large model training dataset optimization method. The explanation, examples, beneficial effects, etc. of the relevant contents can refer to the corresponding contents in the large model training dataset optimization method, and will not be repeated here.

[0125] An embodiment of the present invention also provides a computer-readable storage medium storing a computer program for optimizing a large model training dataset, wherein the computer program enables a computer to execute a large model training dataset optimization method as described above.

[0126] An embodiment of the present invention further provides an electronic device, including:

[0127] one or more processors;

[0128] Memory; and

[0129] One or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for executing a large model training data set optimization method as described above.

[0130] In summary, compared with the existing technology, the present invention has the following beneficial effects:

[0131] This embodiment of the present invention combines the advantages of the orthogonal experimental design method and the near-orthogonal Latin hypercube design method, and simplifies the design process for experimental data that meets near-orthogonality and near-uniformity requirements. By pre-constructing a 20-factor, 9-factor near-orthogonal, near-uniform basic table that meets the near-orthogonality and near-uniformity requirements, it provides a typical large-model training dataset optimization scheme that meets near-orthogonality and near-uniformity when the amount of large-model training data is large, covering all the characteristics of the large-model training dataset. This greatly simplifies the number of data items in the large-model training dataset, optimizes the training dataset design process, and improves training efficiency.

[0132] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0133] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for optimizing a large model training data set, characterized in that: include: S1. Obtain the factor data and factor ranges of the large model training data set W and establish the large model training data factor set H; S2. Determine whether the factor value range of each factor in the factor set H can be converted into 9 value situations. If so, proceed to S3; otherwise, end. S3. Determine whether the number of factors is less than 20. If so, update the set H to an empty set and proceed directly to S4. Otherwise, select 20 factors from the factor set H and update the set H = H - {h1, ..., h20}, and proceed to S4. Where h1, ..., h20 represents 20 factors selected from the factor set H. S4. Determine whether the number of selected factors m is less than 3. If so, perform a full permutation of the m factors and 9 factors to obtain the data item U i , where i represents the number of cycles, and then proceed to S9; Is the number of factors m greater than 3 and less than 20? If so, select the same number of columns from the 20-factor, 9-factor nearly orthogonal, nearly uniform basic table, and then proceed to S5; Otherwise, directly use the 20-factor, 9-factor nearly orthogonal, nearly uniform basic table, and then proceed to S5; S5. Determine the mapping relationship between the factors and the columns selected from the 20-factor 9-factor nearly orthogonal and nearly uniform basic table; S6. Determine the mapping relationship between each variable value and the numerical value in each column; S7. According to the mapping relationship, replace the values ​​in the selected 20 factors and 9 factors near-orthogonal and near-uniform basic table with all factor values, and output 512 groups of data items U of the 20 factors selected to meet the near-orthogonality and near-uniformity. i ; S8. Determine whether the set H is empty. If not, go to S3; if empty, go to S9. S9, the U generated in each cycle i Cartesian product or nested construction of orthogonal Latin hypercube is performed to form a typical data set U, and typical data items d are selected, which satisfy d∈W∩U, to form a training data set D; Among them, the basic table of the 20-factor 9-factor near-orthogonal and near-uniform design is a table with 20 columns and 512 rows. The basic table of the 20-factor 9-factor near-orthogonal and near-uniform design has near-orthogonality and near-uniformity.

2. The large model training data set optimization method according to claim 1, characterized in that: The method for constructing the basic table of the 20-factor 9-factor discrete variable nearly orthogonal and nearly uniform design includes: Step 1: Given a 512*1 vector; Step 2: Add a 512*1 vector to form a 512*2 matrix. Calculate the orthogonality of the two vectors. If the result is less than 0.05, add the vector of the 512*1 vector. Otherwise, find a new 512*1 vector and calculate it again. Step 3: Add a 512*1 vector to form a 512*3 matrix and calculate the orthogonality of the two vectors. Here, we need to calculate the orthogonality of three two-vectors. If the results are all less than 0.05, keep the matrix. Otherwise, start again to find another 512*1 vector and calculate it. And so on, find and build a 512*20 matrix. Step 4: Use the two-dimensional plane distribution diagram to determine the uniformity of the design content. If it is uniform, retain the set of 512*20 matrices, otherwise start again from step 1.

3. The large model training data set optimization method according to claim 1, characterized in that: The near-orthogonality of the basic table of the near-orthogonal and near-uniform design of the 20 factors and 9 factors discrete variables is verified by calculating the maximum absolute value of the correlation coefficient, specifically including: The maximum absolute value of the correlation coefficient is recorded as ρ 相关系数绝对值的最大值 , the absolute value of the correlation coefficient is recorded as |ρ ij |, for any two different column vectors X i and X j , and i≠j, then the maximum absolute value of the correlation coefficient is: Among them, ρ 相关系数绝对值的最大值 The smaller the value of , the better the orthogonality. When the value is less than or equal to 0.05, it is considered to be a nearly orthogonal design. i and X j ρ 相关系数绝对值的最大值 = 0, indicating orthogonality; l represents the column vector column vector X i and X j The first line of .

4. The large model training data set optimization method according to claim 1, characterized in that: The mapping relationship between the determination factors and the columns selected from the 20-factor 9-factor nearly orthogonal and nearly uniform basic table includes: Suppose there is a set A with m variable factors, where A = {1, ..., m}, and m columns are selected from the 20-factor 9-factor nearly orthogonal and nearly uniform basic table to form a set B, where B = {1, ..., m}, and the relationship between sets A and B is f:A→B. f(a1)≠f(a2), and b∈B∧a∈A→f(a)=b; establish variable mapping relationship { i ,b j >}, where a i ∈A,b j ∈B,i={1,…,m},j={1,…,m}.​ 5. The large model training data set optimization method according to any one of claims 1 to 4, characterized in that: Determining the mapping relationship between each variable value and the numerical value in each column includes: a i ={a i1 ,a i2 ,a i3 ,a i4 ,a i5 ,a i6 ,a i7 ,a i8 ,a i9 }, b j ={b j1 ,b j2 ,b j3 ,b j4 ,b j5 ,b j6 ,b j7 ,b j8 ,b j9 }, i = {1, ..., m}, j = {1, ..., m}, determine the mapping relationship between each variable value and the corresponding column of the 9-factor basic table, { i1 ,b j1 >} or { i1 ,b j2 >} or { i1 ,b j3 >} or { i1 ,b j4 >} or { i1 ,b j5 >} or { i1 ,b j6 >} or { i1 ,b j7 >} or { i1 ,b j8 >} or { i1 ,b j9 >}.​​​​​​​​​ 6. A 20-factor 9-factor experimental design system, characterized by: include: A large model training data set acquisition module is used to execute S1, obtain the factor data and factor ranges of the large model training data set W, and establish a large model training data factor set H; The first judgment module is used to execute S2 and judge whether the factor value range of each factor in the factor set H can be converted into 9 value situations. If so, enter S3; otherwise, end; The second judgment module is used to execute S3 and determine whether the number of factors is less than 20. If so, the set H is updated to be empty and the process directly proceeds to S4. Otherwise, 20 factors are selected from the factor set H and the set H = H - {h1, ..., h20} is updated, and the process proceeds to S4. Where h1, ..., h20 represents 20 factors selected from the factor set H. The third judgment module is used to execute S4 and judge whether the number of selected factors m is less than 3. If so, the m factors and 9 factors are fully arranged to obtain the data item U i , where i represents the number of cycles, and then proceed to S9; Is the number of factors m greater than 3 and less than 20? If so, select the same number of columns from the 20-factor, 9-factor nearly orthogonal, nearly uniform basic table, and then proceed to S5; Otherwise, directly use the 20-factor, 9-factor nearly orthogonal, nearly uniform basic table, and then proceed to S5; A first mapping relationship determination module is configured to execute S5, determine the mapping relationship between the factors and the columns selected from the 20-factor and 9-factor nearly orthogonal and nearly uniform basic table; A second mapping relationship determination module, configured to execute S6, determining a mapping relationship between each variable value and a numerical value in each column; The first output module is used to execute S7, replace the values ​​in the selected 20 factors and 9 factors near-orthogonal and near-uniform basic table with all factor values ​​according to the mapping relationship, and output 512 groups of data items U of the 20 factors selected to meet the near-orthogonality and near-uniformity. i ; The fourth judgment module is used to judge whether the set H is empty. If not, the process goes to S3; if it is empty, the process goes to S9; The second output module is used to execute S9 and convert the U generated in each cycle into i Cartesian product or nested construction of orthogonal Latin hypercube is performed to form a typical data set U, and typical data items d are selected, which satisfy d∈W∩U, to form a training data set D; Among them, the basic table of the 20-factor 9-factor near-orthogonal and near-uniform design is a table with 20 columns and 512 rows. The basic table of the 20-factor 9-factor near-orthogonal and near-uniform design has near-orthogonality and near-uniformity.

7. The large model training data set optimization system according to claim 6, characterized in that: The method for constructing the basic table of the 20-factor 9-factor discrete variable nearly orthogonal and nearly uniform design includes: Step 1: Given a 512*1 vector; Step 2: Add a 512*1 vector to form a 512*2 matrix. Calculate the orthogonality of the two vectors. If the result is less than 0.05, add the vector of the 512*1 vector. Otherwise, find a new 512*1 vector and calculate it again. Step 3: Add a 512*1 vector to form a 512*3 matrix and calculate the orthogonality of the two vectors. Here, we need to calculate the orthogonality of three two-vectors. If the results are all less than 0.05, keep the matrix. Otherwise, start again to find another 512*1 vector and calculate it. And so on, find and build a 512*20 matrix. Step 4: Use the two-dimensional plane distribution diagram to determine the uniformity of the design content. If it is uniform, retain the set of 512*20 matrices, otherwise start again from step 1.

8. The large model training data set optimization system according to claim 7, characterized in that: The mapping relationship between the determination factors and the columns selected from the 20-factor 9-factor nearly orthogonal and nearly uniform basic table includes: Suppose there is a set A with m variable factors, where A = {1, ..., m}, and m columns are selected from the 20-factor 9-factor nearly orthogonal and nearly uniform basic table to form a set B, where B = {1, ..., m}, and the relationship between sets A and B is f:A→B. f(a1)≠f(a2), and b∈B∧a∈A→f(a)=b; establish variable mapping relationship { i ,b j >}, where a i ∈A,b j ∈B,i={1,…,m},j={1,…,m}.​ 9. A computer-readable storage medium, characterized in that It stores a computer program for optimizing a large model training dataset, wherein the computer program enables a computer to execute the large model training dataset optimization method according to any one of claims 1 to 5.

10. An electronic device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for executing the large model training data set optimization method as described in any one of claims 1 to 5.

Images