Large model generation data quality determination method and system

Abstract

The invention discloses a large model generation data quality determination method and system. The method comprises the following steps: obtaining generation data and source data of a large model; inputting the generated data into a standard data model to obtain a first quality evaluation grade; obtaining a second quality evaluation grade according to the similar training data of the large model and the corresponding similar generation data; generating a data utility and a third quality evaluation grade of the data according to the similar history; determining a primary data quality grade and risk data of the generated data; after the generated data is put into the application scene, the generated data quality grade is obtained according to the generated data utility and the primary data quality grade; determining abnormal generation data, and adjusting model parameters of the large model according to data defects; the model precision is improved, the generation process of the generated data is improved, the quality of the generated data before being put into the application scene is evaluated, the overall data effectiveness of the generated data is improved, and the data quality of the generated data put into the application scene is guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of data quality assessment, and in particular to a method and system for determining the quality of data generated by a large model. Background Art

[0002] In recent years, with the development of the Internet industry, the scale of data generated has gradually expanded, and how to ensure the quality of this data has become an important issue in the computer field; at the same time, with the gradual improvement of machine learning technology, more and more models are used to solve various problems, and the quality of the large amount of generated data generated by the application of models has also become an important issue that people need to consider.

[0003] At present, the Chinese invention patent with application number 201811525693.0 discloses a data quality determination method, device, electronic device and storage medium. Although the quality of the data to be evaluated is determined by the quality of various data related to the data to be evaluated in the previous time period and in the same application scenario, and the effectiveness feedback of the data to be evaluated in the current time period and in the same application scenario, the data quality is not evaluated before the data is put into use, nor is the unqualified data processed, data defects are obtained, and the process of generating data is adjusted according to the data defects. Summary of the invention

[0004] The technical problem solved by the present invention is that the prior art does not evaluate the data quality before the data is put into use, nor does it process unqualified data, obtain data defects, and adjust the process of generating data according to the data defects.

[0005] In order to solve the above technical problems, in a first aspect, the present invention provides a method for determining the quality of data generated by a large model, comprising the following steps: Step S1, obtaining the generated data and source data of the large model; Step S2, inputting the generated data into a standard data model to obtain a first quality assessment level; Step S3, comparing the source data with the training data for building the large model, obtaining similar training data and corresponding similar generated data, and obtaining a second quality assessment level according to the similarity between the similar generated data and the generated data; Step S4, obtaining historical generation data of the large model, comparing the generation data with the historical generation data, obtaining similar historical generation data and data utility of the similar historical generation data, and obtaining a third quality assessment level according to the data utility; Step S5, determining the primary data quality level of the generated data according to different weights according to the first quality assessment level, the second quality assessment level and the third quality assessment level, and acquiring risk data in the generated data according to the primary data quality level; Step S6, after the generated data is put into the application scenario, the utility of the generated data is obtained, and the quality level of the generated data is obtained according to the utility of the generated data and the quality level of the primary data; Step S7, determining low-quality generated data according to the generated data quality level, and adjusting the model parameters of the large model according to the data defects of the low-quality generated data; As a preferred solution of the method for determining the quality of large model generation data described in the present invention, wherein: The step S2 specifically includes the following steps: Step S201, inputting the generated data into a standard data model; Step S202, using a standard data model to obtain an integrity assessment value, a consistency assessment value, and a timeliness assessment value of the generated data; Step S203, determining the first quality assessment level of the generated data according to the integrity assessment value, the consistency assessment value and the timeliness assessment value of the generated data according to certain weights, and different generated data correspond to different weight ratios according to different degrees of importance of integrity, consistency and timeliness to the generated data; The process of establishing the standard data model includes: Obtaining a generated data set, annotating the generated data set with integrity assessment values, consistency assessment values, and timeliness assessment values, and inputting the annotated generated data set into a machine learning model for training, using the generated data as input and the integrity assessment values, consistency assessment values, and timeliness assessment values ​​as output for model training; As a preferred solution of the large model generation data quality determination system described in the present invention, wherein: The step S3 specifically comprises the following steps: Step S301, comparing the source data with the training data for building a large model, obtaining the training data with the highest similarity to the source data, and defining it as similar training data; Step S302, obtaining the similarity between the source data and the similar training data, which is defined as a first similarity, and obtaining similar generated data corresponding to the similar training data; Step S303, comparing the similar generated data with the generated data to obtain a difference value between the similar generated data and the generated data, which is defined as a first difference value; Step S304, obtaining a generation accuracy of generated data according to the first difference value and the first similarity, and obtaining a corresponding second quality assessment level according to the generation accuracy, wherein the generation accuracy and the second quality assessment level are in a one-to-one correspondence; As a preferred solution of the method for determining the quality of large model generation data described in the present invention, wherein: The step S4 specifically comprises the following steps: Step S401, comparing the generated data with the historical generated data, obtaining the historical generated data with the highest similarity, and defining it as similar historical generated data; Step S401, defining the similarity between the similar historical generated data and the generated data as a second similarity; Step S403, calling data feedback information of the similar history generation data, and obtaining data utility of the similar history generation data according to the data feedback information, wherein the data feedback information includes user satisfaction and data call times; Step S404, obtaining data utility of the generated data according to the second similarity and the data utility of the similar historical generated data, and determining a second quality assessment level of the generated data according to the data utility of the generated data; Compare the generated data with the historical generated data, and obtain the historical generated data with the highest similarity, including: Compare each data item of the generated data with each data item of each historical generated data, obtain the similarity of each data item according to the difference of each data item, calculate the similarity of each data item corresponding to each historical generated data according to a certain weight, obtain the similarity of the generated data and each historical generated data, and arrange the similarities in ascending order, obtain the historical generated data with the highest similarity, and define it as similar historical generated data; The step of obtaining the data utility of generated data according to the second similarity and the data utility of the similar historical generated data includes: The calculation expression for obtaining the data utility of generated data is: , Among them, P is the data utility of generating data, H is the second similarity, Data utility for generating data for similar histories, is the model error rate, is the difference in the application scenario; As a preferred solution of the method and system for determining the quality of large model generation data described in the present invention, wherein: The step S5 specifically comprises the following steps: Step S501, calculating the first quality assessment level, the second quality assessment level and the third quality assessment level according to certain weights to obtain the primary data quality level; Step S502: if the primary data quality level of the generated data is less than or equal to 0.5, the generated data is defined as risk data, the risk data is marked and the activity of putting the risk data into application scenarios is suspended; If the primary data quality level of the generated data is greater than 0.5, the generated data is defined as qualified; As a preferred solution of the method and system for determining the quality of large model generation data described in the present invention, wherein: The step S6 specifically comprises the following steps: Step S601, after the generated data is put into the application scenario, the data feedback information of the generated data is retrieved after one application cycle, and the utility of the generated data is obtained according to the product of user satisfaction and the number of data calls; Step S602, determining a fourth quality assessment level of the generated data according to the utility of the generated data; Step S603, superimposing the fourth quality assessment level and the primary data quality level to obtain a generated data quality level; As a preferred solution of the method for determining the quality of large model generation data described in the present invention, wherein: The step S7 specifically includes the following steps: Step S701, if the quality level of the generated data is less than or equal to 2, define the generated data as low-quality generated data, and obtain the first quality assessment level, the second quality assessment level, the third quality assessment level and the fourth quality assessment level of the low-quality generated data respectively; Step S702, obtaining data defects of the low-quality generated data according to the first quality assessment level, the second quality assessment level, the third quality assessment level and the fourth quality assessment level, wherein the data defects include standard defects, training defects and utility defects, and the standard defects include integrity defects, consistency defects and timeliness defects; Step S703, adjusting the model parameters of the large model according to the data defects of the low-quality generated data; As a preferred solution of the method for determining the quality of large model generation data described in the present invention, wherein: The step S702 specifically includes the following steps: Step S7021: if the first quality assessment level is less than or equal to 1, then obtaining the integrity value, consistency value, and timeliness value of the generated data, and obtaining the standard defect of the generated data according to the integrity value, consistency value, and timeliness value; Step S7022: If the second quality assessment level is less than or equal to 1, the FreaAI method is used to filter out data slices with insufficient generation accuracy in the generated data, define the data slices as low-precision data areas, and obtain training defects of the generated data by comparing the low-precision data areas with the standard data areas; Step S7023, the third quality assessment level and the fourth quality assessment level are superimposed and calculated according to a certain weight to obtain a data utility level. If the data utility level is less than or equal to 1, a utility defect is obtained according to data feedback information of similar historical generated data and data feedback information of generated data; In a second aspect, a large model generation data quality determination system includes a data acquisition module, a quality assessment module, and a model adjustment module; The data acquisition module is used to obtain the generation data of the large model in a timely manner; The quality assessment module is used to obtain the corresponding first quality assessment level, second quality assessment level, third quality assessment level and fourth quality assessment level according to the generated data, and obtain the primary data quality level according to the first quality assessment level, the second quality assessment level and the third quality assessment level, and obtain the generated data quality level according to the primary data quality level and the fourth quality assessment level; The model adjustment module is used to obtain data defects according to the first quality assessment level, the second quality assessment level, the third quality assessment level and the fourth quality assessment level, and adjust the model parameters of the large model according to the data defects; As a preferred solution of the large model generation data quality determination system described in the present invention, wherein: The quality assessment module includes a primary quality assessment unit and a generated quality assessment unit; The primary quality assessment unit is used to obtain a first quality assessment level, a second quality assessment level and a third quality assessment level according to the generated data of the large model, and obtain the primary data quality level according to the first quality assessment level, the second quality assessment level and the third quality assessment level according to a certain weight; The generated data quality assessment unit is used to obtain the generated data utility according to the data feedback information of the generated data, determine the fourth quality assessment level according to the generated data utility, and superimpose the fourth quality assessment level and the primary data quality level according to a certain weight to obtain the generated data quality level; The model adjustment module includes a defect acquisition unit and a model adjustment unit; The defect acquisition unit is used to acquire standard defects according to the first quality assessment level, acquire training defects according to the second quality assessment level, and acquire utility defects according to the third quality assessment level and the fourth quality assessment level; The model adjustment unit is used to adjust the model parameters of the large model according to the standard defect, the training defect and the utility defect.

[0006] Beneficial effects of the present invention: The present invention obtains similar training data and similar historical generated data of large model generated data to obtain the primary data quality level of generated data, and obtains defective data in the generated data according to the primary data quality level, marks it as risk data, and intercepts low-quality generated data before it is put into application scenarios, thereby improving the overall data utility of generated data and ensuring the data quality of generated data put into application scenarios; Obtaining a fourth quality assessment level according to the data utility after the generated data is put into the application scenario, and obtaining a generated data quality level according to the fourth quality assessment and the primary data quality level, obtaining low-quality generated data, and obtaining standard defects, training defects, and utility defects of the low-quality generated data by performing defect analysis on the low-quality generated data, which is conducive to improving the generation process of the generated data and improving the data utility of the accuracy of the generated data; The model parameters of the large model are adjusted according to the standard defects, training defects and utility defects, which improves the model accuracy and ensures the data quality of the generated data. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 A basic flow chart of a method for determining the quality of data generated by a large model provided by one embodiment of the present invention; Figure 2 A basic flow chart of a large model generation data quality determination system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0008] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings. It is obvious that the described embodiments are only part of the embodiments of the present invention, but not all of the embodiments.

[0009] Example 1, reference Figure 1 , as an embodiment of the present invention, provides a method for determining the quality of data generated by a large model, comprising the following steps: Step S1, obtaining the generated data and source data of the large model; Step S2, inputting the generated data into a standard data model to obtain a first quality assessment level; Step S3, comparing the source data with the training data for building the large model, obtaining similar training data and corresponding similar generated data, and obtaining a second quality assessment level according to the similarity between the similar generated data and the generated data; Step S4, obtaining historical generation data of the large model, comparing the generation data with the historical generation data, obtaining similar historical generation data and data utility of the similar historical generation data, and obtaining a third quality assessment level according to the data utility; Step S5, determining the primary data quality level of the generated data according to different weights according to the first quality assessment level, the second quality assessment level and the third quality assessment level, and acquiring risk data in the generated data according to the primary data quality level; Step S6, after the generated data is put into the application scenario, the utility of the generated data is obtained, and the quality level of the generated data is obtained according to the utility of the generated data and the quality level of the primary data; Step S7, determining low-quality generated data according to the generated data quality level, and adjusting the model parameters of the large model according to the data defects of the low-quality generated data.

[0010] In this embodiment, similar training data and similar historical generated data of the large model generated data are obtained to obtain the primary data quality level of the generated data, and defect data in the generated data is obtained according to the primary data quality level, and it is marked as risk data. The low-quality generated data is intercepted before it is put into the application scenario, thereby improving the overall data utility of the generated data and ensuring the data quality of the generated data put into the application scenario; The fourth quality assessment level is obtained according to the data utility after the generated data is put into the application scenario, and the generated data quality level is obtained according to the fourth quality assessment and the primary data quality level, and low-quality generated data is obtained. By performing defect analysis on the low-quality generated data, standard defects, training defects and utility defects of the low-quality generated data are obtained, and the model parameters of the large model are adjusted according to the standard defects, training defects and utility defects, so as to improve the data utility of the accuracy of the generated data and ensure the data quality of the generated data.

[0011] The step S2 specifically includes the following steps: Step S201, inputting the generated data into a standard data model; Step S202, using a standard data model to obtain an integrity assessment value, a consistency assessment value, and a timeliness assessment value of the generated data; Step S203, determining the first quality assessment level of the generated data according to the integrity assessment value, the consistency assessment value and the timeliness assessment value of the generated data according to certain weights, and different generated data correspond to different weight ratios according to different degrees of importance of integrity, consistency and timeliness to the generated data; The process of establishing the standard data model includes: Obtain a generated data set, annotate the generated data set with integrity assessment values, consistency assessment values, and timeliness assessment values, and input the annotated generated data set into a machine learning model for training, using the generated data as input and the integrity assessment values, consistency assessment values, and timeliness assessment values ​​as output for model training.

[0012] The integrity assessment value is used to assess the integrity of generated data; The consistency evaluation value is used to evaluate the relevance of the generated data and the input data; The timeliness evaluation value is used to evaluate the timeliness of generated data.

[0013] In this embodiment, a standard data model is used to obtain the integrity assessment value, consistency assessment value and timeliness assessment value of the generated data, and the first quality weight of the generated data is obtained based on the integrity assessment value, consistency assessment value and integrity assessment value, thereby providing accurate data results for evaluating the integrity, consistency and timeliness of the generated data.

[0014] The step S3 specifically comprises the following steps: Step S301, comparing the source data with the training data for building a large model, obtaining the training data with the highest similarity to the source data, and defining it as similar training data; Step S302, obtaining the similarity between the source data and the similar training data, which is defined as a first similarity, and obtaining similar generated data corresponding to the similar training data; Step S303, comparing the similar generated data with the generated data to obtain a difference value between the similar generated data and the generated data, which is defined as a first difference value; Step S304: acquiring a generation accuracy of generated data according to the first difference value and the first similarity, and acquiring a corresponding second quality assessment level according to the generation accuracy, wherein the generation accuracy and the second quality assessment level are in a one-to-one correspondence.

[0015] The generation accuracy is used to represent the accuracy and precision of data generated by the large model based on the training data.

[0016] In this embodiment, the generation accuracy of the generated data is obtained based on similar training data and similar generated data, and the second quality assessment grade is obtained based on the generation accuracy, which provides a reliable data basis for evaluating the generation precision and generation accuracy of the generated data, and provides a basis for subsequently obtaining the primary data quality grade.

[0017] The step S4 specifically comprises the following steps: Step S401, comparing the generated data with the historical generated data, obtaining the historical generated data with the highest similarity, and defining it as similar historical generated data; Step S401, defining the similarity between the similar historical generated data and the generated data as a second similarity; Step S403, calling data feedback information of the similar history generation data, and obtaining data utility of the similar history generation data according to the data feedback information, wherein the data feedback information includes user satisfaction and data call times; Step S404, obtaining data utility of the generated data according to the second similarity and the data utility of the similar historical generated data, and determining a second quality assessment level of the generated data according to the data utility of the generated data; Compare the generated data with the historical generated data, and obtain the historical generated data with the highest similarity, including: Compare each data item of the generated data with each data item of each historical generated data, obtain the similarity of each data item according to the difference of each data item, calculate the similarity of each data item corresponding to each historical generated data according to a certain weight, obtain the similarity of the generated data and each historical generated data, and arrange the similarities in ascending order, obtain the historical generated data with the highest similarity, and define it as similar historical generated data; The step of obtaining the data utility of generated data according to the second similarity and the data utility of the similar historical generated data includes: The calculation expression for obtaining the data utility of generated data is: , Among them, P is the data utility of generating data, H is the second similarity, Data utility for generating data for similar histories, is the model error rate, is the change difference of the application scenario.

[0018] In this embodiment, the data utility of the historical generated data is obtained through the data feedback information of the historical generated data, and the data utility of the generated data is obtained according to the second similarity, and the second quality assessment level is obtained according to the data utility of the generated data, so as to provide accurate data support for evaluating the generated data before being put into the application scenario, thereby ensuring the data quality of the generated data put into the application scenario.

[0019] The step S5 specifically comprises the following steps: Step S501, calculating the first quality assessment level, the second quality assessment level and the third quality assessment level according to certain weights to obtain the primary data quality level; Step S502: if the primary data quality level of the generated data is less than or equal to 0.5, the generated data is defined as risk data, the risk data is marked and the activity of putting the risk data into the application scenario is suspended; If the primary data quality level of the generated data is greater than 0.5, the generated data is defined as qualified.

[0020] In this embodiment, risk data is acquired and marked through primary data quality levels, thereby ensuring the data quality of generated data put into application scenarios.

[0021] The step S6 specifically comprises the following steps: Step S601, after the generated data is put into application scenario, data feedback information of the generated data is retrieved after one application cycle, and the utility of the generated data is obtained according to the product of user satisfaction and the number of data calls, wherein one application cycle represents the unit time after the generated data is put into application scenario; Step S602, determining a fourth quality assessment level of the generated data according to the utility of the generated data; Step S603: Superimpose the fourth quality assessment level and the primary data quality level to obtain a generated data quality level.

[0022] In this embodiment, the generated data quality level is obtained based on the primary fourth quality assessment level and the data quality level, which is used to evaluate the data quality of the generated data after one application cycle. The real utility of the generated data is put into the generated data quality assessment process, which improves the accuracy and comprehensiveness of the generated data quality assessment and provides a reliable data basis for the subsequent acquisition of low-quality generated data.

[0023] The step S7 specifically includes the following steps: Step S701, if the quality level of the generated data is less than or equal to 2, define the generated data as low-quality generated data, and obtain the first quality assessment level, the second quality assessment level, the third quality assessment level and the fourth quality assessment level of the low-quality generated data respectively; Step S702, obtaining data defects of the low-quality generated data according to the first quality assessment level, the second quality assessment level, the third quality assessment level and the fourth quality assessment level, wherein the data defects include standard defects, training defects and utility defects, and the standard defects include integrity defects, consistency defects and timeliness defects; Step S703: adjusting the model parameters of the large model according to the data defects of the low-quality generated data.

[0024] In this embodiment, data defects of generated data are obtained according to the quality level of generated data, and model parameters of the large model are adjusted according to the data defects, thereby improving the data accuracy of the model generated data.

[0025] The step S702 specifically includes the following steps: Step S7021: if the first quality assessment level is less than or equal to 1, then obtaining the integrity value, consistency value, and timeliness value of the generated data, and obtaining the standard defect of the generated data according to the integrity value, consistency value, and timeliness value; Step S7022: If the second quality assessment level is less than or equal to 1, the FreaAI method is used to filter out data slices with insufficient generation accuracy in the generated data, define the data slices as low-precision data areas, and obtain training defects of the generated data by comparing the low-precision data areas with the standard data areas; Step S7023, the third quality assessment level and the fourth quality assessment level are superimposed and calculated according to a certain weight to obtain a data utility level. If the data utility level is less than or equal to 1, the utility defect is obtained based on the data feedback information of similar historical generated data and the data feedback information of generated data.

[0026] The method of using the FreaAI method to filter out data slices with insufficient generation accuracy in the generated data includes: Use the highest prior posterior density method to find univariate data slices with low precision that reduce the search space and reveal where our data is more likely to have problems; Use decision trees to find bivariate data slices with low precision. These bivariate data slices reduce the search space of categorical predictors and second-order interactions to reveal where our data is more likely to be problematic.

[0027] In this embodiment, standard defects, training defects and utility defects of generated data are obtained according to the first quality assessment level, the second quality assessment level, the third quality assessment level and the fourth quality assessment level, providing a complete data reference for subsequent adjustment of model parameters of the large model according to data defects.

[0028] Example 2, reference Figure 2 Another embodiment of the present invention is different from the first embodiment in that it provides a large model generation data quality determination system, including a data acquisition module, a quality assessment module and a model adjustment module; The data acquisition module is used to obtain the generation data of the large model in a timely manner; The quality assessment module is used to obtain the corresponding first quality assessment level, second quality assessment level, third quality assessment level and fourth quality assessment level according to the generated data, and obtain the primary data quality level according to the first quality assessment level, the second quality assessment level and the third quality assessment level, and obtain the generated data quality level according to the primary data quality level and the fourth quality assessment level; The model adjustment module is used to obtain data defects according to the first quality assessment level, the second quality assessment level, the third quality assessment level and the fourth quality assessment level, and adjust the model parameters of the large model according to the data defects.

[0029] In this embodiment, similar training data and similar historical generated data of the large model generated data are obtained to obtain the primary data quality level of the generated data, and defect data in the generated data is obtained according to the primary data quality level, and it is marked as risk data. The low-quality generated data is intercepted before it is put into the application scenario, thereby improving the overall data utility of the generated data and ensuring the data quality of the generated data put into the application scenario; The fourth quality assessment level is obtained according to the data utility after the generated data is put into the application scenario, and the generated data quality level is obtained according to the fourth quality assessment and the primary data quality level, and low-quality generated data is obtained. By performing defect analysis on the low-quality generated data, standard defects, training defects and utility defects of the low-quality generated data are obtained, and the model parameters of the large model are adjusted according to the standard defects, training defects and utility defects, so as to improve the data utility of the accuracy of the generated data and ensure the data quality of the generated data.

[0030] The quality assessment module includes a primary quality assessment unit and a generated quality assessment unit; The primary quality assessment unit is used to obtain a first quality assessment level, a second quality assessment level and a third quality assessment level according to the generated data of the large model, and obtain the primary data quality level according to the first quality assessment level, the second quality assessment level and the third quality assessment level according to a certain weight; The generated data quality assessment unit is used to obtain the generated data utility according to the data feedback information of the generated data, determine the fourth quality assessment level according to the generated data utility, and superimpose the fourth quality assessment level and the primary data quality level according to a certain weight to obtain the generated data quality level.

[0031] The model adjustment module includes a defect acquisition unit and a model adjustment unit; The defect acquisition unit is used to acquire standard defects according to the first quality assessment level, acquire training defects according to the second quality assessment level, and acquire utility defects according to the third quality assessment level and the fourth quality assessment level; The model adjustment unit is used to adjust the model parameters of the large model according to the standard defect, the training defect and the utility defect.

[0032] In this embodiment, data defects of the generated data are obtained by performing quality assessment on the generated data, and model parameters of the large model are adjusted according to the data defects, thereby improving the generation process of the generated data and ensuring the data quality and data accuracy of the generated data.

[0033] In this embodiment, similar training data and similar historical generated data of the large model generated data are obtained to obtain the primary data quality level of the generated data, and defect data in the generated data is obtained according to the primary data quality level, and it is marked as risk data. The low-quality generated data is intercepted before it is put into the application scenario, thereby improving the overall data utility of the generated data and ensuring the data quality of the generated data put into the application scenario; Obtaining a fourth quality assessment level according to the data utility after the generated data is put into the application scenario, and obtaining a generated data quality level according to the fourth quality assessment and the primary data quality level, obtaining low-quality generated data, and obtaining standard defects, training defects, and utility defects of the low-quality generated data by performing defect analysis on the low-quality generated data, which is conducive to improving the generation process of the generated data and improving the data utility of the accuracy of the generated data; The model parameters of the large model are adjusted according to the standard defects, training defects and utility defects, which improves the model accuracy and ensures the data quality of the generated data.

[0034] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program codes. Among them, the storage medium may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, referred to as EPROM), programmable read-only memory (Programmable Red-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic memory, flash memory, magnetic disk or optical disk. These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0035] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for determining the quality of data generated by a large model, characterized in that: include: Step S1, obtaining the generated data and source data of the large model; Step S2, inputting the generated data into a standard data model to obtain a first quality assessment level; Step S3, comparing the source data with the training data for building the large model, obtaining similar training data and corresponding similar generated data, and obtaining a second quality assessment level according to the similarity between the similar generated data and the generated data; Step S4, obtaining historical generation data of the large model, comparing the generation data with the historical generation data, obtaining similar historical generation data and data utility of the similar historical generation data, and obtaining a third quality assessment level according to the data utility; Step S5, determining the primary data quality level of the generated data according to different weights according to the first quality assessment level, the second quality assessment level and the third quality assessment level, and acquiring risk data in the generated data according to the primary data quality level; Step S6, after the generated data is put into the application scenario, the utility of the generated data is obtained, and the quality level of the generated data is obtained according to the utility of the generated data and the quality level of the primary data; Step S7, determining low-quality generated data according to the generated data quality level, and adjusting the model parameters of the large model according to the data defects of the low-quality generated data.

2. A method for determining the quality of large model generation data as claimed in claim 1, characterized in that: The step S2 specifically includes the following steps: Step S201, inputting the generated data into a standard data model; Step S202, using a standard data model to obtain an integrity assessment value, a consistency assessment value, and a timeliness assessment value of the generated data; Step S203, determining the first quality assessment level of the generated data according to the integrity assessment value, the consistency assessment value and the timeliness assessment value of the generated data according to certain weights, and different generated data correspond to different weight ratios according to different degrees of importance of integrity, consistency and timeliness to the generated data; The process of establishing the standard data model includes: Obtain a generated data set, annotate the generated data set with integrity assessment values, consistency assessment values, and timeliness assessment values, and input the annotated generated data set into a machine learning model for training, using the generated data as input and the integrity assessment values, consistency assessment values, and timeliness assessment values ​​as output for model training.

3. A method for determining the quality of large model generation data as claimed in claim 1, characterized in that: The step S3 specifically comprises the following steps: Step S301, comparing the source data with the training data for building a large model, obtaining the training data with the highest similarity to the source data, and defining it as similar training data; Step S302, obtaining the similarity between the source data and the similar training data, which is defined as a first similarity, and obtaining similar generated data corresponding to the similar training data; Step S303, comparing the similar generated data with the generated data to obtain a difference value between the similar generated data and the generated data, which is defined as a first difference value; Step S304: acquiring a generation accuracy of generated data according to the first difference value and the first similarity, and acquiring a corresponding second quality assessment level according to the generation accuracy, wherein the generation accuracy and the second quality assessment level are in a one-to-one correspondence.

4. A method for determining the quality of large model generation data as claimed in claim 1, characterized in that: The step S4 specifically comprises the following steps: Step S401, comparing the generated data with the historical generated data, obtaining the historical generated data with the highest similarity, and defining it as similar historical generated data; Step S401, defining the similarity between the similar historical generated data and the generated data as a second similarity; Step S403, calling data feedback information of the similar history generation data, and obtaining data utility of the similar history generation data according to the data feedback information, wherein the data feedback information includes user satisfaction and data call times; Step S404, obtaining data utility of the generated data according to the second similarity and the data utility of the similar historical generated data, and determining a second quality assessment level of the generated data according to the data utility of the generated data; Compare the generated data with the historical generated data, and obtain the historical generated data with the highest similarity, including: Compare each data item of the generated data with each data item of each historical generated data, obtain the similarity of each data item according to the difference of each data item, calculate the similarity of each data item corresponding to each historical generated data according to a certain weight, obtain the similarity of the generated data and each historical generated data, and arrange the similarities in ascending order, obtain the historical generated data with the highest similarity, and define it as similar historical generated data; The step of obtaining the data utility of generated data according to the second similarity and the data utility of the similar historical generated data includes: The calculation expression for obtaining the data utility of generated data is: ， Among them, P is the data utility of generating data, H is the second similarity, Data utility for generating data for similar histories, is the model error rate, is the change difference of the application scenario.

5. A method for determining the quality of large model generation data as claimed in claim 1, characterized in that: The step S5 specifically comprises the following steps: Step S501, calculating the first quality assessment level, the second quality assessment level and the third quality assessment level according to certain weights to obtain the primary data quality level; Step S502: if the primary data quality level of the generated data is less than or equal to 0.5, the generated data is defined as risk data, the risk data is marked and the activity of putting the risk data into the application scenario is suspended; If the primary data quality level of the generated data is greater than 0.5, the generated data is defined as qualified.

6. A method for determining the quality of large model generation data as claimed in claim 1, characterized in that: The step S6 specifically comprises the following steps: Step S601, after the generated data is put into application scenario, data feedback information of the generated data is retrieved after one application cycle, and the utility of the generated data is obtained according to the product of user satisfaction and the number of data calls, wherein one application cycle represents the unit time after the generated data is put into application scenario; Step S602, determining a fourth quality assessment level of the generated data according to the utility of the generated data; Step S603: Superimpose the fourth quality assessment level and the primary data quality level to obtain a generated data quality level.

7. A method for determining the quality of large model generation data as claimed in claim 1, characterized in that: The step S7 specifically includes the following steps: Step S701, if the quality level of the generated data is less than or equal to 2, define the generated data as low-quality generated data, and obtain the first quality assessment level, the second quality assessment level, the third quality assessment level and the fourth quality assessment level of the low-quality generated data respectively; Step S702, obtaining data defects of the low-quality generated data according to the first quality assessment level, the second quality assessment level, the third quality assessment level and the fourth quality assessment level, wherein the data defects include standard defects, training defects and utility defects, and the standard defects include integrity defects, consistency defects and timeliness defects; Step S703: adjusting the model parameters of the large model according to the data defects of the low-quality generated data.

8. A method for determining the quality of large model generation data as claimed in claim 7, characterized in that: The step S702 specifically includes the following steps: Step S7021: if the first quality assessment level is less than or equal to 1, then obtaining the integrity value, consistency value, and timeliness value of the generated data, and obtaining the standard defect of the generated data according to the integrity value, consistency value, and timeliness value; Step S7022: If the second quality assessment level is less than or equal to 1, the FreaAI method is used to filter out data slices with insufficient generation accuracy in the generated data, define the data slices as low-precision data areas, and obtain training defects of the generated data by comparing the low-precision data areas with the standard data areas; Step S7023, the third quality assessment level and the fourth quality assessment level are superimposed and calculated according to a certain weight to obtain a data utility level. If the data utility level is less than or equal to 1, the utility defect is obtained based on the data feedback information of similar historical generated data and the data feedback information of generated data.

9. A large model generation data quality determination system, characterized in that: It includes data acquisition module, quality assessment module and model adjustment module; The data acquisition module is used to obtain the generation data of the large model in a timely manner; The quality assessment module is used to obtain the corresponding first quality assessment level, second quality assessment level, third quality assessment level and fourth quality assessment level according to the generated data, and obtain the primary data quality level according to the first quality assessment level, the second quality assessment level and the third quality assessment level, and obtain the generated data quality level according to the primary data quality level and the fourth quality assessment level; The model adjustment module is used to obtain data defects according to the first quality assessment level, the second quality assessment level, the third quality assessment level and the fourth quality assessment level, and adjust the model parameters of the large model according to the data defects.

10. A large model generation data quality determination system as claimed in claim 9, characterized in that: The quality assessment module includes a primary quality assessment unit and a generated quality assessment unit; The primary quality assessment unit is used to obtain a first quality assessment level, a second quality assessment level and a third quality assessment level according to the generated data of the large model, and obtain the primary data quality level according to the first quality assessment level, the second quality assessment level and the third quality assessment level according to a certain weight; The generated data quality assessment unit is used to obtain the generated data utility according to the data feedback information of the generated data, determine the fourth quality assessment level according to the generated data utility, and superimpose the fourth quality assessment level and the primary data quality level according to a certain weight to obtain the generated data quality level; The model adjustment module includes a defect acquisition unit and a model adjustment unit; The defect acquisition unit is used to acquire standard defects according to the first quality assessment level, acquire training defects according to the second quality assessment level, and acquire utility defects according to the third quality assessment level and the fourth quality assessment level; The model adjustment unit is used to adjust the model parameters of the large model according to the standard defect, the training defect and the utility defect.

Images