Large model data set processing method and device, equipment and storage medium

Abstract

The invention provides a large model data set processing method and device, equipment and a storage medium. The method comprises the following steps: acquiring a pre-constructed large model algorithm and a first learning material set required by large model training, and pre-simulating a second learning material set with a target factor relationship and a simulation algorithm required by simulating the second learning material set; adding a simulation algorithm to a large model algorithm, and adding the second learning material set to the first learning material set; performing factor relation labeling on each learning material in the updated first learning material set based on the updated large model algorithm; and determining a processing strategy aiming at the updated first learning material set and / or a modification strategy aiming at the updated large model algorithm in response to the marking result indicating whether the target factor relationship appears or not. The method has a more accurate big data information screening effect and a large model correction anchoring effect, the time and the computing power of large model training iterative learning are obviously reduced, and the labor cost required by information screening is reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a large model data set processing method, device, equipment and storage medium. Background Art

[0002] In recent years, with the popularization and vigorous development of artificial intelligence (AI), various applications combining artificial intelligence have also developed rapidly. Generally speaking, the traditional way of using artificial intelligence requires users to train the model first, and then put the trained model into the inference system, and deploy the trained model through the inference system to realize specific AI applications.

[0003] Existing AI models have problems of one kind or another. For example, large-scale language models (LLMs) often require manual judgment, self-written programs, or mechanical external design, or use artificial simulation data to feed existing large models. However, the manual part may be inaccurate or the judgment criteria of each person may be different, which makes the answers vary greatly. For example, the generated content is inconsistent with the user input, contradicts the previously generated content, or is inconsistent with the known world knowledge. Even a large model that has been well evaluated before may directly affect its application performance in downstream tasks after iteration or drift.

[0004] Therefore, a solution for processing large model data sets is continued to improve model training performance, so as to meet the needs of practical applications. Summary of the invention

[0005] The embodiments of the present disclosure at least provide a large model data set processing method, device, equipment and storage medium, which performs correction through learning data determined by the target factor relationship, thereby achieving a more accurate big data data screening effect, and further having a better large model correction anchoring effect, thereby reducing training iteration learning time.

[0006] In a first aspect, an embodiment of the present disclosure provides a method for processing a large model data set, comprising:

[0007] Obtaining a pre-built large model algorithm and a first learning data set required for large model training, as well as a second learning data set pre-simulated with a target factor relationship and a simulation algorithm required to simulate the second learning data set;

[0008] Adding the simulation algorithm to the large model algorithm to obtain an updated large model algorithm, and adding the second learning data set to the first learning data set to obtain an updated first learning data set;

[0009] Based on the updated large model algorithm, factor relationship annotation is performed on each learning material in the updated first learning material set to determine the annotation result;

[0010] In response to whether the annotation result indicates the presence of the target factor relationship, a processing strategy for the updated first learning data set and / or a modification strategy for the updated large model algorithm are determined.

[0011] In a possible implementation manner, a processing strategy for the updated first learning data set is determined in one of the following ways:

[0012] In the case where the annotation result indicates the presence of the target factor relationship, using the updated first learning data set as input data for large model training;

[0013] When the annotation result indicates that the target factor relationship does not appear, and a compatible factor relationship associated with the target factor relationship appears, a new second learning data set is added to the updated first learning data set until the annotation result shows the target factor relationship.

[0014] In a possible implementation, the method further includes:

[0015] obtaining a third learning data set to be analyzed;

[0016] Inputting the third learning data set into the final large model algorithm to determine whether the target factor relationship appears; the final large model algorithm is obtained by large model iterative training using at least one of the processed learning data obtained by processing the processing strategy of the updated first learning data set and the modified large model algorithm obtained by the modification strategy of the updated large model algorithm;

[0017] The third learning material set is screened based on the judgment result.

[0018] In a possible implementation, the target factor relationship includes one or more of the following relationships:

[0019] The functional relationship between the independent variable and the dependent variable,

[0020] A clustering factor relationship having at least one clustering factor;

[0021] The time series relationship or frequency relationship between different learning materials;

[0022] The time series relationship or frequency relationship between clustering factors within the learning data.

[0023] In a possible implementation, when the target factor relationship is the grouping factor relationship, the method further includes:

[0024] Performing clustering verification based on a statistical clustering verification method to determine whether the target factor relationship meets the specified clustering status;

[0025] In the case where the grouping state is determined to be met based on the statistical grouping verification method, the grouping distribution diagram or geometric distribution form of the target factor relationship is confirmed with the second learning data set, and then screened again; or,

[0026] By analyzing and searching the second learning data set and the first learning data set, a cluster distribution diagram or a geometric distribution form in which all data distributions conform to the target factor relationship is determined.

[0027] In a possible implementation manner, when the target factor relationship is the time series relationship, the method further includes:

[0028] Add random factor relationships outside the Fourier change interval that meet the target factor relationship to confirm the target factor time series relationship;

[0029] Grouping the second learning data set by target factors or factor relationships based on the target time series relationship;

[0030] Based on the statistical clustering verification method, the clustered factors are clustered again to confirm the new target factor time series relationship; and / or, based on the target factor, the various clustering results are summed up or mixed to determine the new clustering factor relationship.

[0031] In a possible implementation manner, after acquiring the second learning material set, the method further includes:

[0032] performing random processing on all or part of the learning materials in the second learning material set to obtain a processed second learning material set;

[0033] The first learning data set is updated based on the processed second learning data set.

[0034] In a possible implementation manner, when there are multiple target factor relationships, the method further includes:

[0035] Determining a common factor relationship between a plurality of the target factor relationships;

[0036] The convolutional layers included in the large model algorithm are screened based on the common factor relationship, and the large model training is performed based on the screened convolutional layers.

[0037] In a possible implementation, screening each convolutional layer included in the large model algorithm based on the common factor relationship includes:

[0038] For each convolution layer included in the large model algorithm, based on the statistical test of the relationship between the same type of convolution layer and the target factor, determine the extreme value or obtain the point value after Fourier transformation; screen by the determined extreme value or the obtained point value;

[0039] And or, for the external learning data to be corrected, convolutional layer screening or data conversion is performed based on the statistical verification of the relationship between the same type of convolutional layer and the target factor.

[0040] In a possible implementation, the method further includes:

[0041] Obtaining calibration indicators for data set calibration;

[0042] The correction index is added to the second learning data set to achieve correction of the correction index during the large model training process.

[0043] In a second aspect, the present disclosure further provides a large model data set processing device, comprising:

[0044] An acquisition module, used to acquire a pre-built large model algorithm and a first learning data set required for large model training, as well as a second learning data set pre-simulated with a target factor relationship and a simulation algorithm required to simulate the second learning data set;

[0045] An adding module, used for adding the simulation algorithm to the large model algorithm to obtain an updated large model algorithm, and adding the second learning data set to the first learning data set to obtain an updated first learning data set;

[0046] A labeling module, used to label the factor relationship of each learning material in the updated first learning material set based on the updated large model algorithm, and determine the labeling result;

[0047] A processing module is used to determine a processing strategy for the updated first learning data set and / or a modification strategy for the updated large model algorithm in response to whether the labeling result indicates the presence of the target factor relationship.

[0048] In a third aspect, the present disclosure further provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, a large model data set processing method as described in any one of the first aspect and its various embodiments is performed.

[0049] In a fourth aspect, the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, executes a large model data set processing method as described in any one of the first aspect and its various embodiments.

[0050] By adopting the above-mentioned large model data set processing method, device, equipment and storage medium, after obtaining the pre-built large model algorithm and the first learning data set required for large model training, as well as the second learning data set pre-simulated with the target factor relationship and the simulation algorithm required to simulate the second learning data set, the large model algorithm and the learning data set can be updated, and then the factor relationship of each learning material in the updated first learning data set can be labeled based on the updated large model algorithm, so as to determine the processing strategy for the updated first learning data set according to the labeling results, and / or the modification strategy for the updated large model algorithm. The first learning data set obtained by the processing strategy is a learning data set that has been corrected by the target factor relationship, achieving a more accurate large data screening effect. At the same time, a better large model correction anchoring effect can be achieved by modifying the large model algorithm, which significantly shortens the subsequent large model training iterative learning time and computing power, and reduces the manpower cost required for data screening.

[0051] Other advantages of the present disclosure will be explained in more detail in conjunction with the following description and drawings.

[0052] It should be understood that the above description is only an overview of the technical solution of the present disclosure, so that the technical means of the present disclosure can be generally understood and then implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present disclosure more obvious and easy to understand, the specific implementation methods of the present disclosure are specifically described below with examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the drawings only illustrate certain embodiments of the present disclosure and therefore should not be regarded as limiting the scope of protection. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work. Moreover, the same reference numerals are used to represent the same components throughout the drawings. In the drawings:

[0054] Figure 1 A flowchart of a large model data set processing method provided by an embodiment of the present disclosure is shown;

[0055] Figure 2A schematic diagram of a large model data set processing device provided by an embodiment of the present disclosure is shown;

[0056] Figure 3 A schematic diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0057] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0058] In the description of the embodiments of the present disclosure, it should be understood that terms such as "including" or "having" are intended to indicate the presence of the disclosed features, numbers, steps, behaviors, components, parts, or a combination thereof in the specification, and do not exclude the possibility of the presence of one or more other features, numbers, steps, behaviors, components, parts, or a combination thereof.

[0059] Unless otherwise specified, “ / ” means or. For example, A / B can mean A or B. The “and / or” in this article is merely a way to describe the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0060] The terms "first", "second", etc. are used only to distinguish the same or similar technical features for the convenience of description, and should not be understood as indicating or implying the relative importance or quantity of these technical features. Thus, the features defined by "first", "second", etc. may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present disclosure, unless otherwise specified, the term "plurality" means two or more than two.

[0061] Research has found that the development of existing large AI models has deviated from the scaling rule. Taking large-scale language models (LLMs) as an example, some of the bottlenecks come from the fact that data often requires manual judgment or the use of artificially simulated data to feed existing large models, but the manual part may be inaccurate or the judgment criteria of each person may be different. Artificial data simulation may bring in artificial simulation factors, such as making the answers vary greatly, generating content that is inconsistent with user input, inconsistent with previously generated content, or inconsistent with known world knowledge. Even a large model that was previously well evaluated may directly affect its application performance in downstream tasks after iteration or drift.

[0062] On the other hand, in the past, AI used self-written programs or mechanical external designs to identify or operate based on specified factor relationships, improve the recognition of theoretical data, avoid drift, and reduce the need and errors of manual screening. Compared with the underlying large models based on traditional statistical methods, it can provide more accurate clustering screening and time series relationship identification, but it cannot integrate the effects of existing large model learning iterations.

[0063] In order to at least partially solve one or more of the above-mentioned problems and other potential problems, the present disclosure provides at least one large model data set processing solution, which uses target factor relationship simulation data to allow the large model to learn, uses target factor relationship to confirm the large model learning, and uses self-written programs or mechanical external designs to further screen out content that cannot be confirmed by statistical relationships alone. Big data data that is different from specific simulations can quickly establish a large model Agent, which can be used to add and calibrate the large model Agent; it can also be used as a timely judgment or abnormal warning of the input and output data of the specific factor relationship Agent, especially when the current Agent model has some problems of insufficient data or too few factors. The target factor relationship can be specified to quickly screen out the excessive situation, which is helpful for the application of the Agent model in the terminal and reduces hallucination output; or it can be applied in the case where the existing data is insufficient and the relevant specified factor large model data needs to be screened.

[0064] At the same time, the processing mode provided by the embodiments of the present disclosure does not conflict with other large-model data screening methods, such as the existing competitive learning of manually confirming positive and negative cases, or the reinforcement learning of generating data with large models, which can be used together and complement each other. In addition, the embodiments of the present disclosure implement automated screening, further improve the theoretical accuracy of data identification, and improve the large-model learning effect of the data set after screening and learning identification. It has a higher precision measurement function, effectively promotes the progress of human culture and technological development, and the confirmation of related target factor relationships and the re-learning of screened data are expected to further improve the learning results of the large model. Furthermore, the embodiments of the present disclosure also provide a matching accelerated convolution and correction method to accelerate large-model iterations and save computing power and labor costs.

[0065] To facilitate understanding of this embodiment, a large model data set processing method disclosed in the embodiment of the present disclosure is first introduced in detail. The execution subject of the large model correction method provided in the embodiment of the present disclosure is generally an electronic device with certain computing capabilities, and the electronic device includes, for example, a server or other processing device. In some possible implementations, the large model data set processing method can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0066] See also Figure 1, which shows a flow chart of a large model data set processing method provided by an embodiment of the present disclosure, the method comprises the following steps S101 to S104:

[0067] S101: Acquire a pre-built large model algorithm and a first learning data set required for large model training, as well as a second learning data set pre-simulated with a target factor relationship and a simulation algorithm required for simulating the second learning data set;

[0068] S102: adding the simulation algorithm to the large model algorithm to obtain an updated large model algorithm, and adding the second learning data set to the first learning data set to obtain an updated first learning data set;

[0069] S103: labeling the factor relationship of each learning material in the updated first learning material set based on the updated large model algorithm, and determining the labeling result;

[0070] S104: In response to whether the labeling result indicates the presence of a target factor relationship, determining a processing strategy for the updated first learning data set and / or a modification strategy for the updated large model algorithm.

[0071] In order to facilitate understanding of the large model data set processing method provided by the embodiment of the present disclosure, the application scenario of the method is briefly introduced below. The large model data set processing method provided by the embodiment of the present disclosure can be mainly used in the field of artificial intelligence technology, for example, it can be applied to various artificial intelligence sub-fields such as image recognition and face detection, without specific restrictions. Especially in the large model training stage, the processed learning data set or the modified large model algorithm has better model training performance, so that it can adapt to more complex and changeable downstream application scenarios.

[0072] For different application scenarios, the simulation algorithms actually added here are also different. For example, it can be to add a specific factor relationship fitting algorithm, detailed to specific parameters such as gravity 1 / 2*9.8*t^0.5 or battery wave equation, or to specify parameters in the current large model algorithm.

[0073] In practical applications, there may be more than two target factor relationships: the target factor relationships may share or be related to more than one target factor or target factor relationship (or related relationship), and after confirming the individual target factor relationships, the shared target factor or target factor relationship is confirmed.

[0074] When there is a common target factor, confirm whether the target factors are identical, linear (linear relationship, curve relationship) or related. For non-continuous and sequential target factors, you can also choose to use the clustering method to confirm. If not, confirm the compatible algorithm, modify and add the algorithm module, and increase the simulation data for learning.

[0075] Assuming that the common target factor is Y or the target factor relationship is Y=cy, first confirm the other target factor relationships, then confirm the common target factor or target factor relationship (or related relationship). For example, y=Z^2 or y'=2lnZ is a possible compatible factor relationship, but it will appear in the parameter part of the non-specialized algorithm, which may be a similar value. At this time, after confirming that Y and y have a linear relationship or Y and 2^y' have a linear relationship, complete the confirmation and correction, and change the parameters to the parameters that the target factor relationship and compatible relationship should have.

[0076] For more complex target factor relationships and compatible factor relationships, all factor relationships can be written in a specified algorithm, or they can be split and combined, such as confirming some target factor relationships separately, and then confirming the common target factors or target factor relationships (or correlation relationships); confirm whether the target factors are identical, linear (linear relationship, curve relationship) or related, and the relevant non-continuous and sequential target factors can also be confirmed by clustering method. If not, the compatible algorithm is confirmed, the algorithm module is modified and added, and the simulation data for learning is added. The confirmation of the compatible algorithm is to confirm whether the algorithm parameters after various splits and combinations are the same or similar to the parameters of the built-in compatible algorithm, and replace its parameters.

[0077] Next, an example of a splitting and combining implementation is given. The entire set of equations, or the co-constructed set of equations that are split and combined can be used as a compatible algorithm. For example, there are many different compatible algorithms for electromagnetic wave functions. However, after splitting and combining, different data formats, data sources, or convolutional layers may appear. The factor relationship characteristics after splitting can be used to find compatible factor relationships.

[0078] In practical applications, when there is an overall target factor relationship, compatible factor relationships are more likely to occur, and after splitting, factor units or convolution and grouping methods are more likely to appear, causing factor value changes, but this method can complete the calibration of the target factor relationship algorithm as much as possible. If the specialized algorithm is not significant (i.e., it is excluded from the calculation of the large model factor value) after iteration again, it means drift, or when using a compatible algorithm, there may be drift, then the large model needs to be recalibrated.

[0079] When it is determined that the second learning data set with the simulation algorithm is added, the large model algorithm label can be used to check whether the target factor relationship appears. Here, the second learning data set that meets the target factor relationship is added to the first learning data set required for large model training, and it is iterated to see whether the target factor relationship appears; if the target factor relationship does not appear, the learning data set label is used to confirm whether a compatible factor relationship appears, or whether the algorithm module needs to be modified, or the learning data set is added and iterated again until the target factor relationship appears.

[0080] If the confirmation of all target factor relationships cannot be completed in the end, it is necessary to confirm whether there are contradictions or errors in the composition of the target factor relationships, but the correction of the large model after iteration can still be partially completed. In subsequent iterations, if the last correction is a compatible algorithm, or the target factor relationship specialization algorithm is not significant (that is, it is excluded from the calculation of the large model factor value), it is necessary to calibrate again.

[0081] In the embodiment of the present disclosure, the processing strategy for the updated first learning data set can be to use the updated first learning data set as input data for large model training when the annotation result indicates the presence of the target factor relationship; or to add a new second learning data set to the updated first learning data set until the target factor relationship appears in the annotation result when the annotation result indicates that the target factor relationship does not appear and a compatible factor relationship associated with the target factor relationship appears.

[0082] Here, when at least one of the processed learning data obtained by the processing strategy of the updated first learning data set and the modified large model algorithm obtained by the modification strategy of the updated large model algorithm is subjected to large model iterative training to obtain the final large model algorithm, the large model verification can be performed using the third learning data set to be analyzed.

[0083] Here, the big data set to be analyzed or identified (i.e., the third learning data set) is labeled with the large model after algorithm correction to confirm whether the same target factor relationship appears in the simulation data set; the target factor relationship here includes but is not limited to the functional relationship between the independent variable and the dependent variable, the clustering factor relationship with at least one clustering factor; the time series relationship or frequency relationship between different learning data; the time series relationship or frequency relationship between the clustering factors within the learning data.

[0084] In practical applications, if there is no built-in large model fitting program, the corresponding test can be used to confirm whether the target factor relationship appears or is within the target factor correlation interval. Then the data can be classified, or the non-compliant data can be removed, or warnings can be issued.

[0085] When the embodiment of the present disclosure is actually applied to the grouping factor relationship, the data can be updated according to the following steps:

[0086] Step 1: Perform clustering verification based on the statistical clustering verification method to determine whether the target factor relationship meets the specified clustering status;

[0087] Step 2: When the clustering status is determined to be met based on the statistical clustering verification method, the clustering distribution diagram or geometric distribution form of the target factor relationship is confirmed with the second learning data set, and then screened again; alternatively, through the analysis and search of the second learning data set and the first learning data set, it is determined that the distribution of all data meets the clustering distribution diagram or geometric distribution form of the target factor relationship.

[0088] When there is a clustering relationship in the target factor relationship, there may be more than one clustering factor. The clustering relationship can correspond to the statistical test of the distribution diagram of the clustering factor, or further specialized precise data formats, such as clustering tables, multidimensional matrices, images, mappings, Cayley diagrams, mapping relationships, multiplication tables, etc. These clustering factors may be common or non-common factors, common or non-common factor relationships of other target factor relationships, but there will be correlations, so the clustering factors can be used as independent learning data items (for example, in the three-phase diagram simulation learning data set, the corresponding temperature and pressure items are added to the learning data set);

[0089] When there are common factors in other target factor relationships, after confirming whether the corresponding target factor relationship is met in each group, the clustering factor can be confirmed (it is a common factor in other target factor relationships, so the label can be shared). Then, various statistical clustering verification methods can be used to perform clustering verification to confirm whether it meets the specified clustering status. If there are no common factors in other target factor relationships, the clustering factor is a non-common factor. At this time, this clustering factor can be added to the learning data, or there is a simulated data set with the target factor relationship of this clustering factor, or after the statistical clustering verification, the simulated data set and the original big data data set can be analyzed and searched by clustering method to determine the clustering factor label.

[0090] Among them, in the process of determining the clustering factor labels by the clustering method, there are many different ways of clustering methods and statistical tests that can be used. Here, numbers represent the values ​​or ranges of clustering condition factors. Through clustering, the fitted factor relationships are the same or different, and they are slowly eliminated and confirmed until one or a group of "factors" are obtained. For example, the corresponding learning data set or the big data set is added first, and then the non-correlated factor relationships are excluded. Compare the data clusters containing (1, 2, 3, 4, 5) with the clusters containing (6, 7, 8, 9) to find out the factor relationships that have changed (such as T test, breakpoint test), and then compare the clusters containing (2, 4, 6, 8) with the clusters containing (1, 3, 5, 7, 9), and even perform statistical tests on the set containing 1 factor and the set containing 2 factors. Statistical tests (such as factor clustering tests and breakpoint tests) can find out the changing factors.

[0091] In the learning data set taking the three-phase relationship of water as an example, the clustering relationship between temperature and pressure is corresponding. If the simulation data originally has density, thermal conductivity function, and refraction function, after fitting the aforementioned target factor relationship, the label of the clustering factor temperature can be found. At this time, the pressure label can be manually added to the learning data set, or the detectable and observable target factor relationship related to pressure (in the simulation data) can be simulated. After the clustering is confirmed, the clustering relationship can be further confirmed to confirm whether it matches the three-phase diagram.

[0092] In the case of further confirmation and correction of the clustering "target factor relationship distribution diagram", the "clustering target factor relationship diagram" can use pictures, three-dimensional Bayesian tests or various relationship diagrams and mapping diagrams in group theory. Although it is called a graph here, it can be single-dimensional or multi-dimensional. After considering the relationship between compatible factors of compatible target factor relationships (such as 2.1.2Y=cy), Gaussian transformation, group theory, topology and other methods (twisting, flipping, combination, rotation, exchange, permutation, subgroup, product and quotient, regularization, normalization, folding, adjacent, conjugate, flipping, rotation, homomorphism, modular operation, direct product, coprime) are used alone or in combination to transform the "clustering target factor relationship diagram", or fit it after various preset transformations, or perform Fourier transformation (inverse transformation) and then statistical test fitting to complete the confirmation of the target factor relationship. If the simulation data set or large model data set contains data that cannot be matched, confirm whether there are contradictions, errors, no solutions, or missing relationships among the target factors (for example, the three-phase water was later found to have more than 10 states), or whether to remove the simulation data set or large model data set.

[0093] Here, we can directly confirm whether they are identical or similar, or calculate the correlation by testing the fit, such as Bayesian sampling test, which uses random sampling points to confirm the correlation, or Bayesian area test, which moves the changed mapping graph and calculates the change in the overlapping area to confirm. The latter is also applicable to the "factor relationship distribution diagram" after Fourier transform or inverse Fourier transform (which may be a cluster distribution diagram or frequency distribution diagram).

[0094] Different groups correspond to a set of target factor relationships (density function, thermal conductivity function, etc.). After fitting the target factor relationships respectively, the factor relationship labels of different groups are statistically tested to confirm whether they meet the grouping status.

[0095] At this time, Bayesian sampling test is carried out, and the above-mentioned target factor relationship data in the sampling simulation data and the large model data are matched with the "target factor relationship diagram" to further confirm the clustering relationship.

[0096] When the embodiment of the present disclosure is actually applied to a time series relationship, the clustering verification can be performed according to the following steps:

[0097] Step 1: Add random factor relationships outside the Fourier change interval that meet the target factor relationship to confirm the target factor time series relationship;

[0098] Step 2: clustering the second learning data set by target factors or factor relationships based on the target time series relationship;

[0099] Step 3: Re-group the grouped factors based on the statistical grouping verification method to confirm the new target factor time series relationship; and / or, add up or mix the grouping results based on the target factor to determine the new grouping factor relationship.

[0100] Time series relationship refers to a group of more than one time series data, in which there is a cluster target factor relationship. It can be the target factor relationship of each cluster after grouping by different time, geography, subject (people, time, place, and things), or other cluster relationships; it can also be combined into a new cluster target factor relationship after remixing and segmenting different clusters for confirmation and analysis (such as resuming the time series data into data of every 5 minutes, hours, days, and months, or resumming the geographical data by district, city, and province, and the new target factor relationship of each cluster is derived from the target factor relationship of each original cluster).

[0101] In practical applications, the clustering relationship here is mainly the correlation relationship of external input data or target factor relationship. At this time, the target factor change or target factor relationship change in the time series data is the target factor relationship; first confirm the target factor or target factor relationship, and then perform image comparison or image comparison after Fourier transformation. (Example 1)

[0102] The Fourier transform relationship can be the transformation of time series data, images or multi-dimensional data. The relationship of the factor characteristics and the topological transformation target factors in the parameter search is the relationship after the Fourier transform. The Fourier transform or inverse transform relationship can be the transformation of time series data, sound, image or multi-dimensional data. It can be the Fourier transform of the grouped data. After the Fourier transform or inverse transform, it can be directly fitted or related test.

[0103] The large model data set processing method provided by the embodiment of the present disclosure performs random processing on all or part of the learning data in the second learning data set after acquiring the second learning data set to obtain a processed second learning data set; and then updates the first learning data set based on the processed second learning data set.

[0104] For example, for the specific application of car identification, the color of the car can be randomized, or it can be randomized according to the proportion of the license plate data issued in the area. Another example is the car flow. After Fourier transformation, the Fourier transform data of the on-site data can be compared. The flow outside the simulation interval can be added as randomness to change the simulation data, which helps the identification of the simulation interval.

[0105] In the actual large model training process, the embodiment of the present disclosure can also combine the common factor relationship to screen the convolution layer, which can be determined by the following steps:

[0106] Step 1: Determine the common factor relationships among multiple target factor relationships;

[0107] Step 2: Screen the convolutional layers included in the large model algorithm based on the common factor relationship, and perform large model training based on the screened convolutional layers.

[0108] Among them, for each convolution layer included in the large model algorithm, based on the statistical verification of the relationship between the same type of convolution layer and the target factor, the extreme value is determined or the point value is obtained after Fourier transformation; then screening is carried out through the determined extreme value or the obtained point value; and for external learning data to be corrected, convolution layer screening or data conversion can also be carried out based on the statistical verification of the relationship between the same type of convolution layer and the target factor.

[0109] Here, the statistical test of the relationship between the same type of convolutional layer and the target factor can calculate the extreme value, or the point value that appears after Fourier transformation, and give priority to confirming the convolutional layer; or use the common factors of several target factor relationships to confirm the convolutional layer that is suitable for each target factor relationship.

[0110] In a specific embodiment, the convolution of linear data can be divided into [1][2][3][4][5][6][7][8][9]

[10]

[11]

[12] ... Simple Fourier transform can be performed on [1][2][3][4][5][6][7][8][9]

[10]

[11]

[12] .... Prioritize the ones with high proportions after transformation (such as frequencies 3 and 10) and perform [1] + [2] + [3] convolution and [1] + [2] + [3] +.... +

[10] convolution to see which convolution neural fitting is more suitable.

[0111] If there are two sets of target factor relationships that have shared factors (i.e., have a common factor relationship) and may be different convolutional layers, the convolutional layer that can fit the common factor relationship can be preferentially selected.

[0112] In practical applications, the large model data set processing method provided by the embodiment of the present disclosure can also be calibrated through the following steps:

[0113] Step 1: obtaining a calibration index for calibrating a data set;

[0114] Step 2: Add the correction index to the second learning data set to calibrate the correction index during the training of the large model.

[0115] Here, correction indicators (such as local landmarks and focus shift image data) can be added to the simulation data, which can allow the large model to learn to detect the correction or convolution of the color temperature and focus of the data.

[0116] In order to facilitate understanding of the large model data set processing method provided by the embodiments of the present disclosure, some specific embodiments are used for detailed description below.

[0117] Example 1: The target factor relationship is a simulated frequency function of vehicles passing through a certain intersection

[0118] The target factor relationship is a vehicle frequency diagram (matrix) after Fourier transformation, and the learning material is a time series image of a simulated intersection taking into account traffic lights, weather, time and date. The learning material may include third-party data, and the image may include specific landmarks of the location.

[0119] After collecting local data and performing Fourier transformation, other frequency functions outside the simulated frequency function (interval) can be included to correct the simulated data. The colors of passing vehicles can be randomly dispersed, that is, non-correlated factors can be randomized.

[0120] The target factor function is the vehicle simulation frequency function, and the target factor relationship function is the vehicle frequency graph after the approximate Fourier transform, which is fitted with Bayesian statistics (matrix), and the Fourier transform is used for accelerated convolution.

[0121] The target factor relationship here can be, for example, vehicle speed and upper limit (pointing to function relationship), car passing frequency (pointing to grouping factor relationship), traffic light time (pointing to time series relationship), vehicle type, license plate recognition factor relationship, etc., to produce simulation data.

[0122] During production, the random factor frequency outside the target factor relationship interval can be increased by vehicle speed or frequency (since the two are related, adding one is sufficient), and the vehicle type, color, etc. that do not affect safety are randomly or locally sampled in the overall data. Data on local camera focus and landmarks are also added to the learning data.

[0123] Here, the target factor relationship algorithm is written, the clustering method is confirmed, and it is added to the time series data of the number of vehicles that have passed the simulation data as needed.

[0124] Among them, convolution or correction can be accelerated by landmarks, or accelerated convolution selection can be adopted.

[0125] After the large model is calibrated, the camera data at the intersection can be imported, and the target factor relationship fitted by the large model and the Fourier transform data of the time series after grouping can be used for Bayesian testing and real-time fitting to see if there are any abnormalities.

[0126] Example 2: Robot walking learning material screening

[0127] Walking simulation data (simulation data of sensor positions of each joint), image and joint position scans, data of position sensors of each limb, tension sensors, etc. (large model data set data).

[0128] Calibration is performed based on the target factor relationship of the simulation data. For example, the simulation data is input with the swing frequency of each foot and hand, and the computer simulates the movement under the influence of gravity on different slopes. Confirm whether there is a relationship such as gravity; perform Fourier transformation on the relative center of gravity of the joints to determine whether there is a target factor relationship and perform data screening.

[0129] In practical applications, the relationship between the joint control factors of the simulation data is used, such as taking the center of gravity as the origin, and the relative orthocenter, shoulder line and line of sight perpendicular translation line as the three axes of the three-dimensional coordinates; the distribution maps of other limbs and joints are drawn separately according to the four quadrants or positions of the feet off the ground, and the data are screened again based on this graph.

[0130] At this time, the convolution layer selection can be performed for the position sensor and tension sensor data after the screening. It can be used for daily calibration before use, or if a specific sensor or a similar sensor is replaced, it can be used for iterative learning with the screened data to determine whether the target factor relationship appears in the unaffected part of the other data, and then the convolution layer selection can be performed for the data affected by the replacement of the sensor.

[0131] It is known that the large model data set processing method provided by the embodiment of the present disclosure mainly has the following technical features:

[0132] (1) By simulating the learning data set with the target factor relationship and the algorithm module written with the target factor relationship, the algorithm module has high specificity and speciality, and can confirm whether the target factor relationship, the "target factor relationship graph" relationship, or the compatibility relationship with the aforementioned relationship appears, and complete the large model correction (automatically or semi-automatically adding algorithms or programs); it can also use this large model to label big data and provide automated big data or large model data set screening with the same target factor relationship confirmation method.

[0133] Among them, the target factor relationship and factor value are used for correction, model anchoring and large model data set screening.

[0134] In addition, correction methods for three types of target factor relationships are also provided. Taking them as basic units, the target factor relationships can be disassembled, combined, and transformed, providing operational methods for multi-dimensional, multi-layered, and multi-type target factor relationships.

[0135] (2) The disclosed embodiments also include conversion of various compatible factor relationships of target factor relationships, algorithm module confirmation methods; and operable conversion confirmation methods for mutually shared factors or factor relationships. This method allows external calibration or internal embedding of a large model, and can be incorporated into the large model after further iteration, giving full play to the advantages of the large model to screen big data (for example, the simulation data set is text, but it can also screen graphic data)

[0136] (3) Provide a systematic simulation data optimization solution (how and when to add random factors), and provide convolution correction and accelerated fitting methods for the relationship between hardware, software and target factors. A matching accelerated convolution layer selection method is provided. Therefore, the training results based on simulation data can play a role in model correction and better learning effects. When using it for big data screening, non-target factors can be included, and hardware, data format and other issues can be eliminated, thereby achieving better big data data analysis results.

[0137] Compared with the related art, the embodiment of the present disclosure provides a data set processing solution for screening and confirming large data with a large model algorithm. In practical applications, a semi-automatic or fully automated method can be used to add data verification and transformation methods to the large model (or Agent model), so that after iterative fitting of the simulated data, the target factor relationship can be used to confirm whether the large model calibration is completed, and this method can be used to further screen the large data or the specified data set. For example, it can be used to find data that meets or does not meet a specific target factor relationship; for another example, it can be used as real-time data monitoring, or to confirm whether it exceeds the scope of the Agent's existing data.

[0138] The disclosed embodiment uses the target factor relationship and factor value, as well as the corresponding simulation data for correction. Compared with the method of using factor value, data screening, and re-learning alone, it has a better large model correction anchoring effect, reduces iterative learning time, and can also achieve a better and more accurate large data data screening effect; in addition, it also provides a systematic simulation data optimization solution, and also provides a convolution correction for hardware and software, and a similar convolution layer selection method for target factor relationship correction, which can accelerate the fitting of the target factor relationship; and, by using the target factor relationship to screen big data, it can screen the target factor relationship more accurately, such as using scientific functions and corresponding simulation data correction, and then further deducing through the large model, automatically searching and confirming the big data to be analyzed; finally, it can also be applied to other large model needs. For example, it can be used to screen the learning data (re)invested in the large model; in the terminal large model Agent, when the hardware and software and data are insufficient, if this method is used to screen user questions that exceed the range of the built-in data, it can avoid illusions or erroneous answers, etc., so that it is more applicable.

[0139] In the description of this specification, the description with reference to the terms "some possible embodiments", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure, and the above terms do not necessarily represent the same embodiment or example. Moreover, the described specific features, structures, materials or characteristics may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0140] About the method flow chart of the embodiment of the present disclosure, some operations are described as different steps performed in a certain order. Such a flow chart is illustrative and not restrictive. Some steps described in this article can be grouped together and performed in a single operation, or some steps can be divided into multiple sub-steps, and some steps can be performed in an order different from that shown in this article. The various steps shown in the flow chart can be implemented in any way by any circuit structure and / or tangible mechanism (for example, by software running on a computer device, hardware (for example, a logical function implemented by a processor or a chip), etc., and / or any combination thereof).

[0141] Those skilled in the art will appreciate that, in the method described in the above specific implementation, the writing order of each step does not mean a strict execution order, and the specific execution order of each step should be determined by its function and possible internal logic.

[0142] Based on the same inventive concept, the embodiments of the present disclosure also provide a large model data set processing device corresponding to the large model data set processing method. Since the principle of solving the problem by the device in the embodiments of the present disclosure is similar to the large model data set processing method mentioned above in the embodiments of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0143] Reference Figure 2 , which is a schematic diagram of a large model data set processing device provided by an embodiment of the present disclosure, the device includes: an acquisition module 201, an addition module 202, a labeling module 203 and a processing module 204; wherein,

[0144] An acquisition module 201 is used to acquire a pre-built large model algorithm and a first learning data set required for large model training, as well as a second learning data set pre-simulated with a target factor relationship and a simulation algorithm required for simulating the second learning data set;

[0145] An adding module 202 is used to add the simulation algorithm to the large model algorithm to obtain an updated large model algorithm, and to add the second learning data set to the first learning data set to obtain an updated first learning data set;

[0146] A labeling module 203 is used to label the factor relationship of each learning material in the updated first learning material set based on the updated large model algorithm to determine the labeling result;

[0147] The processing module 204 is used to determine a processing strategy for the updated first learning data set and / or a modification strategy for the updated large model algorithm in response to whether the labeling result indicates the presence of the target factor relationship.

[0148] By adopting the above-mentioned large model data set processing device, after obtaining the pre-built large model algorithm and the first learning data set required for large model training, as well as the second learning data set pre-simulated with the target factor relationship and the simulation algorithm required to simulate the second learning data set, the large model algorithm and the learning data set can be updated, and then the factor relationship of each learning data in the updated first learning data set is annotated based on the updated large model algorithm, so as to determine the processing strategy for the updated first learning data set according to the annotation results, and / or the modification strategy for the updated large model algorithm. The first learning data set obtained by the processing strategy is a learning data set that has been corrected by the target factor relationship, achieving a more accurate large data screening effect. At the same time, a better large model correction anchoring effect can be achieved by modifying the large model algorithm, which significantly shortens the subsequent large model training iterative learning time and computing power, and reduces the manpower cost required for data screening.

[0149] In a possible implementation manner, the processing module 204 is specifically configured to determine a processing strategy for the updated first learning data set in one of the following ways:

[0150] When the labeling result indicates the presence of a target factor relationship, the updated first learning data set is used as input data for training the large model;

[0151] When the labeling result indicates that the target factor relationship does not appear, and a compatible factor relationship associated with the target factor relationship appears, a new second learning data set is added to the updated first learning data set until the labeling result shows the target factor relationship.

[0152] In a possible implementation, the processing module 204 is further configured to:

[0153] obtaining a third learning data set to be analyzed;

[0154] Inputting the third learning data set into the final large model algorithm to determine whether the target factor relationship appears; the final large model algorithm is obtained by iteratively training the large model using at least one of the processed learning data obtained by processing the processing strategy of the updated first learning data set and the modified large model algorithm obtained by the modification strategy of the updated large model algorithm;

[0155] The third learning material set is screened based on the judgment result.

[0156] In a possible implementation, the target factor relationship includes one or more of the following relationships:

[0157] The functional relationship between the independent variable and the dependent variable,

[0158] A clustering factor relationship having at least one clustering factor;

[0159] The time series relationship or frequency relationship between different learning materials;

[0160] The time series relationship or frequency relationship between clustering factors within the learning data.

[0161] In a possible implementation manner, when the target factor relationship is a grouping factor relationship, the adding module 202 is further used to:

[0162] Perform clustering verification based on statistical clustering verification methods to determine whether the target factor relationship meets the specified clustering status;

[0163] When the clustering status is determined to be met based on the statistical clustering verification method, the clustering distribution diagram or geometric distribution form of the target factor relationship is confirmed with the second learning data set, and then screened again; or,

[0164] Through the analysis and search of the second learning data set and the first learning data set, it is determined that the cluster distribution diagram or geometric distribution form of all data distributions conforms to the relationship between the target factors.

[0165] In a possible implementation, when the target factor relationship is a time series relationship, the processing module 204 is further configured to:

[0166] Add random factor relationships outside the Fourier change interval that meet the target factor relationship to confirm the target factor time series relationship;

[0167] The second learning data set is grouped by target factors or factor relationships based on the target time series relationship;

[0168] Based on the statistical clustering verification method, the clustered factors are clustered again to confirm the new target factor time series relationship; and / or, based on the target factor, the various clustering results are summed up or mixed to determine the new clustering factor relationship.

[0169] In a possible implementation, after acquiring the second learning material set, the processing module 204 is further configured to:

[0170] performing random processing on all or part of the learning materials in the second learning material set to obtain a processed second learning material set;

[0171] The first learning data set is updated based on the processed second learning data set.

[0172] In a possible implementation, when there are multiple target factor relationships, the processing module 204 is further configured to:

[0173] Determine the common factor relationships among multiple target factor relationships;

[0174] The convolutional layers included in the large model algorithm are screened based on the common factor relationship, and the large model training is performed based on the screened convolutional layers.

[0175] In a possible implementation, the processing module 204 is used to screen the convolutional layers included in the large model algorithm based on the common factor relationship according to the following steps:

[0176] For each convolution layer included in the large model algorithm, based on the statistical test of the relationship between the same type of convolution layer and the target factor, the extreme value is determined or the point value is obtained through Fourier transformation; the determined extreme value or the obtained point value is used for screening;

[0177] And or, for the external learning data to be corrected, convolutional layer screening or data transformation is performed based on the statistical verification of the relationship between the same type of convolutional layer and the target factor.

[0178] In a possible implementation, the processing module 204 is further configured to:

[0179] Obtaining calibration indicators for data set calibration;

[0180] The correction index is added to the second learning data set to achieve correction of the correction index during the training process of the large model.

[0181] It should be noted that the device in the embodiment of the present disclosure can implement each process of the embodiment of the aforementioned method and achieve the same effects and functions, which will not be repeated here.

[0182] The present disclosure also provides an electronic device, such as Figure 3 , which is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure, includes: a processor 301, a memory 302, and a bus 303. The memory 302 stores machine-readable instructions executable by the processor 301 (for example, Figure 2 The machine-readable instructions corresponding to the acquisition module 201, the addition module 202, the marking module 203 and the processing module 204 in the device are executed, etc. When the electronic device is running, the processor 301 communicates with the memory 302 through the bus 303, and the machine-readable instructions are executed by the processor 301 to perform the following processing:

[0183] Obtaining a pre-built large model algorithm and a first learning data set required for large model training, as well as a second learning data set pre-simulated with a target factor relationship and a simulation algorithm required to simulate the second learning data set;

[0184] Adding the simulation algorithm to the large model algorithm to obtain an updated large model algorithm, and adding the second learning data set to the first learning data set to obtain an updated first learning data set;

[0185] Based on the updated large model algorithm, factor relationship annotation is performed on each learning material in the updated first learning material set to determine the annotation result;

[0186] In response to whether the annotation result indicates the presence of the target factor relationship, a processing strategy for the updated first learning data set and / or a modification strategy for the updated large model algorithm are determined.

[0187] The present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the large model data set processing method described in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0188] The present disclosure also provides a computer program product that carries a program code. The program code includes instructions that can be used to execute the steps of the large model data set processing method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

[0189] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0190] Each embodiment in the present disclosure is described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and computer-readable storage medium embodiments, since they are basically similar to the method embodiments, their descriptions are simplified, and the relevant parts can be referred to the partial description of the method embodiments.

[0191] The apparatus, equipment and computer-readable storage medium provided in the embodiments of the present disclosure correspond one-to-one to the method. Therefore, the apparatus, equipment and computer-readable storage medium also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the apparatus, equipment and computer-readable storage medium will not be repeated here.

[0192] It should be understood by those skilled in the art that the embodiments of the present disclosure can be implemented in the form of methods and apparatus (devices or systems), or computer-readable storage media. Therefore, the present disclosure can be implemented in a complete hardware implementation, a complete software implementation, or an implementation that combines software and hardware. Moreover, the present disclosure can be implemented in the form of a computer-readable storage medium implemented on one or more computer-usable storage media (including but not limited to disk storage, read-only compact disk storage (CD-ROM), optical storage, etc.) containing computer-usable program code.

[0193] The present disclosure is described with reference to the flowchart and / or block diagram of the method, device (equipment or system) and computer-readable storage medium according to the embodiment of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, and the combination of the process and / or box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0194] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device, wherein the instruction device implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0195] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0196] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0197] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash memory (Flash RAM). The memory is an example of a computer-readable medium.

[0198] Computer-readable media include permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory, read-only memory, electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. In addition, although the operations of the disclosed method are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. In addition, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple sub-steps.

[0199] Although the spirit and principle of the present disclosure have been described above with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined. The present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims.

Claims

1. A method for processing a large model data set, characterized in that: include: Obtaining a pre-built large model algorithm and a first learning data set required for large model training, as well as a second learning data set pre-simulated with a target factor relationship and a simulation algorithm required to simulate the second learning data set; Adding the simulation algorithm to the large model algorithm to obtain an updated large model algorithm, and adding the second learning data set to the first learning data set to obtain an updated first learning data set; Based on the updated large model algorithm, factor relationship annotation is performed on each learning material in the updated first learning material set to determine the annotation result; In response to whether the annotation result indicates the presence of the target factor relationship, a processing strategy for the updated first learning data set and / or a modification strategy for the updated large model algorithm are determined.

2. The method according to claim 1, characterized in that Determine a processing strategy for the updated first learning data set in one of the following ways: In the case where the annotation result indicates the presence of the target factor relationship, using the updated first learning data set as input data for large model training; When the annotation result indicates that the target factor relationship does not appear, and a compatible factor relationship associated with the target factor relationship appears, a new second learning data set is added to the updated first learning data set until the annotation result shows the target factor relationship.

3. The method according to claim 1, characterized in that The method further comprises: obtaining a third learning data set to be analyzed; Inputting the third learning data set into the final large model algorithm to determine whether the target factor relationship appears; the final large model algorithm is obtained by large model iterative training using at least one of the processed learning data obtained by processing the processing strategy of the updated first learning data set and the modified large model algorithm obtained by the modification strategy of the updated large model algorithm; The third learning material set is screened based on the judgment result.

4. The method according to claim 1, characterized in that: The target factor relationship includes one or more of the following relationships: about the functional relationship between the independent and dependent variables; A clustering factor relationship having at least one clustering factor; The time series relationship or frequency relationship between different learning materials; The time series relationship or frequency relationship between clustering factors within the learning data.

5. The method according to claim 4, characterized in that In the case where the target factor relationship is the grouping factor relationship, the method further includes: Performing clustering verification based on a statistical clustering verification method to determine whether the target factor relationship meets the specified clustering status; In the case where the grouping state is determined to be met based on the statistical grouping verification method, the grouping distribution diagram or geometric distribution form of the target factor relationship is confirmed with the second learning data set, and then screened again; or, By analyzing and searching the second learning data set and the first learning data set, a cluster distribution diagram or a geometric distribution form in which all data distributions conform to the target factor relationship is determined.

6. The method according to claim 4, characterized in that In the case where the target factor relationship is the time series relationship, the method further includes: Add random factor relationships outside the Fourier change interval that meet the target factor relationship to confirm the target factor time series relationship; Grouping the second learning data set by target factors or factor relationships based on the target time series relationship; Based on the statistical clustering verification method, the clustered factors are clustered again to confirm the new target factor time series relationship; and / or, based on the target factor, the various clustering results are summed up or mixed to determine the new clustering factor relationship.

7. The method according to any one of claims 1 to 6, characterized in that After acquiring the second learning material set, the method further includes: performing random processing on all or part of the learning materials in the second learning material set to obtain a processed second learning material set; The first learning data set is updated based on the processed second learning data set.

8. The method according to any one of claims 1 to 6, characterized in that In the case where there are multiple target factor relationships, the method further includes: Determining a common factor relationship between a plurality of the target factor relationships; The convolutional layers included in the large model algorithm are screened based on the common factor relationship, and the large model training is performed based on the screened convolutional layers.

9. The method according to claim 8, characterized in that The screening of each convolutional layer included in the large model algorithm based on the common factor relationship includes: For each convolution layer included in the large model algorithm, based on the statistical test of the relationship between the same type of convolution layer and the target factor, determine the extreme value or obtain the point value after Fourier transformation; screen by the determined extreme value or the obtained point value; And or, for the external learning data to be corrected, convolutional layer screening or data conversion is performed based on the statistical verification of the relationship between the same type of convolutional layer and the target factor.

10. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Obtaining calibration indicators for data set calibration; The correction index is added to the second learning data set to achieve correction of the correction index during the large model training process.

11. A large model data set processing device, characterized in that: include: An acquisition module, used to acquire a pre-built large model algorithm and a first learning data set required for large model training, as well as a second learning data set pre-simulated with a target factor relationship and a simulation algorithm required to simulate the second learning data set; An adding module, used for adding the simulation algorithm to the large model algorithm to obtain an updated large model algorithm, and adding the second learning data set to the first learning data set to obtain an updated first learning data set; A labeling module, used to label the factor relationship of each learning material in the updated first learning material set based on the updated large model algorithm, and determine the labeling result; A processing module is used to determine a processing strategy for the updated first learning data set and / or a modification strategy for the updated large model algorithm in response to whether the labeling result indicates the presence of the target factor relationship.

12. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the large model data set processing method as described in any one of claims 1 to 10 is performed.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the large model data set processing method as described in any one of claims 1 to 10 is executed.

Images