Method, device, equipment, medium and program product for quality assessment of a data set

Abstract

The application provides a quality evaluation method, device, equipment, medium and program product of a data set, relates to the technical field of artificial intelligence, and the method comprises the following steps: obtaining a text data set to be evaluated in quality; performing hierarchical clustering analysis on the text data set to obtain a clustering result of the text data set; and performing topic extraction on sample data in the text data set based on a preset large model to obtain a semantic topic of the sample data; constructing a multi-layer label tree based on the clustering result and the semantic topic, and calculating the weight of each node of the multi-layer label tree based on the label distribution of the text data set to generate a weighted frequency label tree; and extracting quantization indexes of multiple dimensions of the text data set based on the weighted frequency label tree to evaluate the quality of the text data set. The application can improve the adaptability of the data set in a multi-task large model training scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a data set quality evaluation method and device, equipment, medium and program product. BACKGROUND

[0002] In related technologies, the quality evaluation of large-scale data sets usually only focuses on a single dimension (such as label frequency), ignoring factors such as semantic hierarchy, task instruction coverage, and language balance, thereby leading to poor adaptability of large-scale data sets in the training and reinforcement fine-tuning of multi-task large models. For example, even if two large-scale sample data sets have the same instruction frequency, they can be completely different in semantic hierarchy, and for example, candidate large-scale data sets lack data coverage in certain key fields that have a direct impact on multi-task large models, which ultimately affects the generalization ability of multi-task large models.

[0003] In summary, related technologies for quality evaluation of large-scale data sets only focus on a single dimension, making it difficult to comprehensively analyze data characteristics, resulting in poor adaptability of large-scale data sets in multi-task large model training scenarios. SUMMARY

[0004] The main purpose of the embodiments of the present application is to propose a data set quality evaluation method, device, equipment, medium and program product, which aims to comprehensively analyze large-scale data sets to fine-tune and optimize data quality, thereby improving the adaptability of data sets in multi-task large model training scenarios.

[0005] To achieve the above purpose, a first aspect of the embodiments of the present application proposes a data set quality evaluation method, which comprises:

[0006] Obtain a text data set whose quality is to be evaluated;

[0007] Perform hierarchical clustering analysis on the text data set to obtain a clustering result of the text data set, and perform topic extraction on sample data in the text data set based on a preset large model to obtain semantic topics of the sample data;

[0008] Construct a multi-layer label tree based on the clustering result and the semantic topics, and calculate the weight of each node of the multi-layer label tree based on the label distribution of the text data set to generate a weighted frequency label tree;

[0009] Extract quantization indicators of multiple dimensions of the text data set based on the weighted frequency label tree to evaluate the quality of the text data set.

[0010] In some embodiments, the topic extraction on the sample data in the text data set based on the preset large model comprises:

[0011] obtaining a preset text prompt; the text prompt at least includes a processing requirement of data and an output requirement of a data processing result;

[0012] Based on the text prompt, guiding the pre-trained large model to perform topic extraction on the sample data in the text data set according to the processing requirement, and outputting the first semantic topic extracted according to the output requirement; the semantic topic of the sample data includes the first semantic topic.

[0013] In some embodiments, after the hierarchical clustering analysis of the text data set is performed to obtain the clustering result of the text data set, the method further comprises:

[0014] Based on the preset text prompt, guiding the pre-trained large model to generate a second semantic topic for the cluster in the clustering result, and taking the second semantic topic as the semantic topic of the sample data in the text data set.

[0015] In some embodiments, the clustering result includes a tree-shaped clustering result; and the constructing a multi-layer label tree based on the clustering result and the semantic topic comprises:

[0016] Generating a semantic node name of a target node in the tree-shaped clustering result based on the semantic topic, so as to construct the tree-shaped clustering result into a multi-layer label tree.

[0017] In some embodiments, the calculating the weight of each node of the multi-layer label tree based on the label distribution of the text data set to generate a weighted frequency label tree comprises:

[0018] Determining the frequency of each node in the multi-layer label tree based on the label distribution of the text data set;

[0019] Based on the frequency of each node and the depth of each node in the multi-layer label tree, the weight of each node is calculated to obtain the weight of each node;

[0020] The weight of each node is normalized to generate a weighted frequency label tree;

[0021] The normalization processing includes horizontal normalization processing and vertical normalization processing, the horizontal normalization processing is used for normalizing the weight of the nodes belonging to the same level in the multi-layer label tree, and the vertical normalization processing is used for normalizing the weight of all nodes in the multi-layer label tree.

[0022] In some embodiments, the method further comprises:

[0023] adding labels to sample data in the text data set, and determining a label distribution of the text data set based on the labels of the sample data; the labels of the sample data correspond to nodes in the multi-layer label tree;

[0024] determining the frequency of each node in the multi-layer label tree based on the label distribution of the text data set, includes:

[0025] determining the frequency of sample data in the text data set based on the label distribution of the text data set;

[0026] counting the frequency of each node in the multi-layer label tree based on the frequency of the sample data.

[0027] In some embodiments, the extracting the quantification indicators of multiple dimensions of the text data set based on the weighted frequency label tree includes:

[0028] performing calculation of target dimension indicators based on the weighted frequency label tree and sample label frequency information of the text data set, to obtain the quantification indicators of multiple dimensions of the text data set;

[0029] wherein the sample label frequency information includes the frequency of sample data in the text data set; and the target dimension indicators include at least two of a domain diversity indicator, an instruction diversity indicator, a complexity indicator, and an English-Chinese proportion indicator.

[0030] In some embodiments, the text data set includes a benchmark text data set and a candidate text data set, and the weighted frequency label tree includes a first weighted frequency label tree of the benchmark text data set and a second weighted frequency label tree of the candidate text data set;

[0031] The method further includes:

[0032] performing cross-entropy comparison on the first weighted frequency label tree and the second weighted frequency label tree, to obtain a structural similarity between the first weighted frequency label tree and the second weighted frequency label tree;

[0033] evaluating a structural distribution difference between the candidate text data set and the benchmark text data set based on the structural similarity, to obtain a structural distribution difference evaluation result; the structural distribution difference evaluation result is one of results of quality evaluation on the candidate text data set.

[0034] To achieve the above object, a second aspect of the embodiment of the present application provides a data set quality evaluation device, which includes:

[0035] an acquisition module configured to acquire a text data set to be evaluated in quality;

[0036] The clustering and model topic extraction module is configured to perform hierarchical clustering analysis on the text dataset to obtain a clustering result of the text dataset, and perform topic extraction on sample data in the text dataset based on a preset large model to obtain a semantic topic of the sample data.

[0037] The label tree construction module is configured to construct a multi-layer label tree based on the clustering result and the semantic topic, and calculate a weight of each node of the multi-layer label tree based on a label distribution of the text dataset to generate a weighted frequency label tree.

[0038] The dataset quality evaluation module is configured to extract quantization indicators of multiple dimensions of the text dataset based on the weighted frequency label tree to evaluate the quality of the text dataset.

[0039] To achieve the above object, a third aspect of embodiments of the present application provides a dataset quality evaluation device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the dataset quality evaluation method of the first aspect when executing the computer program.

[0040] To achieve the above object, a fourth aspect of embodiments of the present application provides a vehicle, which is provided with a dataset quality evaluation device, the dataset quality evaluation device comprises a memory and a processor, the memory stores a computer program, and the processor implements the dataset quality evaluation method of the first aspect when executing the computer program.

[0041] To achieve the above object, a fifth aspect of embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the dataset quality evaluation method of the first aspect.

[0042] To achieve the above object, a sixth aspect of embodiments of the present application provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the dataset quality evaluation method of the first aspect.

[0043] The data set quality evaluation method, device, equipment, vehicle, computer readable storage medium and computer program product provided by the application obtain a text data set to be evaluated for quality. Hierarchical clustering analysis is performed on the text data set to obtain a clustering result of the text data set. A semantic topic of sample data in the text data set is extracted based on a preset large model. A multi-layer label tree is constructed based on the clustering result and the semantic topic. A weighted frequency label tree is generated based on a weight of each node of the multi-layer label tree calculated based on a label distribution of the text data set. Quantitative indicators of multiple dimensions of the text data set are extracted based on the weighted frequency label tree to evaluate the quality of the text data set.

[0044] Compared with the quality evaluation method of a large-scale data set from a single dimension, the embodiment of the application performs hierarchical clustering analysis on a large-scale text data set to be evaluated for quality and extracts a semantic topic of sample data in the text data set based on a preset large model. A multi-layer label tree reflecting a semantic hierarchy is constructed based on the clustering result obtained by hierarchical clustering analysis and the semantic topic extracted by topic extraction. The weight of each node of the multi-layer label tree is calculated based on the label distribution of the text data set to generate a weighted frequency label tree. Quantitative indicators of multiple dimensions of the text data set are extracted based on the weighted frequency label tree to evaluate the quality of the text data set. In this way, the embodiment of the application combines hierarchical clustering of a large-scale data set and semantic extraction of a large model to construct a label tree structure reflecting a semantic hierarchy. Quantitative indicators are extracted from multiple dimensions to evaluate the quality of the data set based on the weighted frequency label tree. The quality of the data set can be comprehensively analyzed to realize fine management and optimization of the data quality, thereby improving the adaptability of the data set in a multi-task large model training scenario. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 The step flowchart of the data set quality evaluation method provided by the embodiment of the application in some embodiments;

[0046] Figure 2 The step flowchart of the data set quality evaluation method provided by the embodiment of the application in some other embodiments;

[0047] Figure 3 The detailed step flowchart of step S102 in the embodiment of the application; Figure 1 The detailed step flowchart of step S103 in the embodiment of the application;

[0048] Figure 4 The detailed step flowchart of step S103 in the embodiment of the application; Figure 1 The detailed step flowchart of step S103 in the embodiment of the application;

[0049] Figure 5A hierarchical clustering and weighted label tree multi-dimensional data set distribution analysis process schematic diagram is involved in a complete embodiment of the data set quality evaluation method provided in the embodiment of the application.

[0050] Figure 6 A large model topic extraction and label marking cooperation process schematic diagram is involved in a complete embodiment of the data set quality evaluation method provided in the embodiment of the application.

[0051] Figure 7 A four-index-based data quality measurement process schematic diagram is involved in a complete embodiment of the data set quality evaluation method provided in the embodiment of the application.

[0052] Figure 8 A structure schematic diagram of the data set quality evaluation device provided in the embodiment of the application is shown in FIG. 6.

[0053] Figure 9 A hardware structure schematic diagram of the data set quality evaluation device provided in the embodiment of the application is shown in FIG. 7. DETAILED DESCRIPTION

[0054] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not intended to limit the present application.

[0055] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0057] First, the overall concept of the data set quality evaluation method provided in the embodiment of the application is described.

[0058] With the development of artificial intelligence technology, especially the application of large language models (such as GPT, GLM, BERT, etc.), the demand for high-quality large-scale data sets is increasing due to the high dependence of model training results on the diversity, coverage, and structural rationality of data sets. In addition, with the gradual popularization of multi-task models (MTM) and reinforcement fine-tuning (RFT), the requirements for the diversity of task types in sample data, the complexity of instruction design, and the balance of language proportion are also increasing.

[0059] However, most data analysis methods in the related art still remain in the stage of statistical analysis of single dimensions such as data label frequency, sample quantity, and text length, lacking multi-dimensional and structured panoramic evaluation methods, which may lead to problems such as uneven data distribution, domain redundancy, and single language structure that are difficult to discover in a timely manner. For example, during the quality evaluation stage of large-scale data sets in the related art, only a single dimension (such as label frequency) is usually focused on, and factors such as semantic hierarchy, task instruction coverage, and language balance are ignored, which may lead to poor adaptability of the data set in the training of multi-task large models. This is because even two large-scale sample data sets with the same instruction frequency may be completely different in semantic hierarchy, or the candidate large-scale data set lacks data coverage in certain key domains that directly affect the multi-task large model, which ultimately affects the generalization ability of the multi-task large model.

[0060] Therefore, there is an urgent need for a structured, multi-dimensional, and normalizable data analysis mechanism to perform multi-dimensional data set analysis with strong interpretability, clear structure, and comprehensive coverage for large-scale data sets, so as to achieve fine-grained management and optimization of data quality by comprehensively understanding the distribution characteristics of the data set, and to support data engineers and algorithm developers to scientifically evaluate the data set quality, analyze the task adaptability, and optimize the distribution bias.

[0061] Based on this, the embodiments of the present application provide a data set quality evaluation method, device, equipment, vehicle, computer readable storage medium, and computer program product, which comprehensively analyze large-scale data sets to achieve fine-grained management and optimization of data quality, thereby improving the adaptability of the data set in the training scenario of multi-task large models.

[0062] Considering that structured methods such as hierarchical clustering and label tree have good semantic analysis and hierarchical modeling capabilities, they can provide a new perspective for data analysis. Therefore, the embodiments of the present application construct an analysis system that combines the hierarchical structure of labels and the modeling capability of distribution indicators, and creatively propose a mechanism that combines hierarchical clustering (Hierarchical Clustering) and large model topic extraction, constructs a weighted label tree (Label Tree), and introduces a weight-normalized multi-dimensional data analysis mechanism. By fusing label frequency and hierarchical structure, the field distribution, task instruction distribution, language proportion, and complexity indicators can be quantified, and on this basis, a normalized distribution comparison mechanism (such as Cross-Entropy Distribution Matching) can be constructed to realize the structural alignment and evaluation between the candidate text dataset and the benchmark text dataset, thereby effectively overcoming the limitations of the data analysis methods in the related art.

[0063] It should be noted that hierarchical clustering is a clustering method that divides data into hierarchical clusters with tree structure by using recursive method. This method continuously merges the closest data points or clusters by calculating the similarity or distance between samples (such as Ward method), thereby constructing a hierarchical clustering structure. The label tree is a data label system expressed in tree structure, where each node represents a clustering topic or label category, and the child node is a subcategory. The label tree can reflect the hierarchical and dependent relationship between labels and is used for multi-dimensional distribution analysis. Cross-Entropy Distribution Matching is used to compare the similarity of field distribution between the candidate text dataset and the benchmark text dataset. By constructing a weighted frequency label tree and calculating its cross-entropy, the difference in distribution structure between the two datasets is quantified.

[0064] In the embodiments of the present application, multi-dimensional data distribution analysis based on hierarchical clustering and weighted frequency label tree aims to comprehensively quantify and evaluate the distribution characteristics of large-scale text datasets from a structured perspective. By combining hierarchical clustering with large model semantic extraction, a label tree structure reflecting semantic hierarchy is constructed, and the weight of the label tree is calculated based on sample frequency and node depth, thereby forming a weighted frequency label tree (Weighted Frequency Tree). On this basis, quantitative indicators are extracted from multiple dimensions (such as domain diversity, instruction diversity, complexity score, and English-Chinese proportion), and structured comparative analysis between the candidate text dataset and the benchmark text dataset is supported, thereby improving the adaptability, balance, and quality control capability of the dataset in the multi-task model training scenario.

[0065] It should be noted that the weighted frequency label tree is a label tree variant that fuses label frequency and node level information. The weight of each node in the weighted frequency label tree is determined by its sample frequency and level depth, which is used to measure the importance of the node in the overall tree structure. Domain diversity represents the distribution breadth and uniformity of different semantic domains contained in the dataset. Through label tree structure and information entropy calculation method, the distribution of each domain in the dataset is quantified. Instruction diversity represents the richness and balance of different task instruction types contained in the dataset. It is calculated by comparing the frequency distribution of each type of instruction in the dataset, and is commonly used to judge the coverage ability of the dataset. The complexity index measures the structural complexity of the sample text or task. By calculating the difference between each sample and the standard complexity, the complexity fluctuation level of the existing dataset is obtained.

[0066] In the embodiments of the present application, through a unique data modeling and evaluation mechanism, specifically including: 1. Constructing a multi-layer label tree through hierarchical clustering and large language model topic extraction; 2. Calculate the node weight combining sample frequency and level depth, and introduce horizontal and vertical normalization methods to generate weighted frequency label tree; 3. Define and implement the calculation method of multi-dimensional indicators such as domain diversity, instruction diversity, complexity and English-Chinese ratio; 4. Perform cross-entropy-based normalized distribution similarity comparison between candidate text dataset and benchmark text dataset, which constitutes an integrated multi-dimensional data analysis framework that is evaluable, interpretable and comparable.

[0067] Next, the dataset quality evaluation method, device, equipment, vehicle, computer readable storage medium and computer program product provided by the embodiments of the present application are specifically described as follows. First, the dataset quality evaluation method provided by the embodiments of the present application is described in detail.

[0068] It should be noted that in each specific embodiment of the present application, when it is necessary to process relevant data related to the identity or characteristics of the user according to user information, user behavior data, user history data and user location information, the user's permission or consent will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the separate permission or separate consent of the user will be obtained through a pop-up window or a jump to a confirmation page, and after obtaining the separate permission or separate consent of the user, the necessary user-related data for enabling the embodiments of the present application to function normally will be obtained.

[0069] It should be noted that the data set quality evaluation method provided in the embodiments of the present application can be applied to a terminal, can also be applied to a server end, and can also be software running in the terminal or the server end. In some embodiments, the terminal can be a vehicle-mounted terminal on a vehicle, can also be a smart phone, a tablet computer, a notebook computer, a desktop computer, and the like, and the terminal can also be associated with the vehicle, that is, the terminal can communicate with the vehicle based on a network. The server end can be a background server terminal device of the vehicle, can be configured as an independent physical server, can be configured as a server cluster or a distributed system formed by multiple physical servers, and can also be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, content delivery network (CDN), and big data and artificial intelligence platform. The software can be an application, a computer program, and a storage medium carrying the computer program, and the like, which implement the data set quality evaluation method. It should be understood that, based on different design needs of actual applications, in different feasible embodiments, the terminal, the server end, and the software, and the like, to which the data set quality evaluation method provided in the embodiments of the present application is applied can certainly be other forms not listed herein, and the data set quality evaluation method provided in the embodiments of the present application is not specifically limited in this regard.

[0070] Furthermore, the present application can be used in a variety of general purpose or special purpose computer systems environments or configurations. For example: vehicles, personal computers, server computers, hand-held or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, personal computers (PCs), minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The present application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0071] For the sake of understanding and elaboration, the quality evaluation method of the data set provided by the embodiments of the present application is taken as an example in the following to explain the specific embodiments of the present application in detail. The implementation of the quality evaluation method of the data set provided by any of the above-mentioned subject matters can refer to the process of the quality evaluation method of the data set of the terminal device elaborated below.

[0072] Please refer to Figure 1 , Figure 1 The step flowchart of the quality evaluation method of the data set provided by the embodiments of the present application is shown in the following. It should be understood that, although Figure 1 and subsequent other step flowcharts show the execution order of some method steps, based on different design needs of actual applications, the quality evaluation method of the data set provided by the embodiments of the present application can of course adopt an execution order of method steps different from that shown in the figure. That is, Figure 1 The order of the method steps shown does not constitute a limitation on the execution logic order of the quality evaluation method of the data set provided by the embodiments of the present application. Any reasonable change based on the order of the method steps shown should be included in the protection scope of the quality evaluation method of the data set provided by the embodiments of the present application. Figure 1

[0073] As Figure 1 shown, in some embodiments, the quality evaluation method of the data set provided by the embodiments of the present application can include the following steps S101 to S104.

[0074] Step S101: Obtain a text data set to be evaluated for quality.

[0075] The terminal device can continuously collect relevant sample data (such as image description data and audio text data, etc.) to automatically construct a large-scale text data set for large model training or fine-tuning, and then use the constructed large-scale data set as a text data set to be evaluated for quality before actual model training or fine-tuning application.

[0076] In some embodiments, the terminal device can also directly receive a text data set to be evaluated for quality through a pre-set data interface (such as a user interface, an application APP, and a voice assistant, etc. human-computer interaction interface).

[0077] Step S102: Perform hierarchical clustering analysis on the text data set to obtain a clustering result of the text data set, and perform topic extraction on sample data in the text data set based on a pre-set large model to obtain a semantic topic of the sample data.

[0078] ​After obtaining the text data set to be evaluated, the terminal device processes the text data set by combining hierarchical clustering and large model semantic extraction, that is, performs uneven clustering analysis on the text data set to obtain a clustering result of the text data set, and performs topic extraction on sample data in the text data set based on a preset large model to obtain semantic topics (such as “basic mathematics”, “open-ended question and answer”, and the like) of the sample data.

[0079] It should be noted that the preset large model can be any language model with rich semantic expression capability, such as GPT-4, ChatGPT, and the like.

[0080] Step S103: constructing a multi-layer label tree based on the clustering result and the semantic topic, and generating a weighted frequency label tree based on a label distribution of the text data set.

[0081] In the process of processing the text data set by combining hierarchical clustering and large model semantic extraction, after obtaining the clustering result of the text data set and the semantic topics of the sample data in the text data set, the terminal device can further construct a label tree structure capable of reflecting semantic levels, that is, a multi-layer label tree, based on the clustering result and the semantic topics. Moreover, the terminal device further calculates the weight of each node in the multi-layer label tree in combination with the label distribution of the text data set, thereby generating a weighted frequency label tree.

[0082] In some embodiments, the clustering result obtained by the terminal device by performing hierarchical clustering analysis on the text data set includes a tree clustering result.

[0083] The terminal device can use a clustering analysis module in an integrated multi-dimensional data analysis framework to perform hierarchical clustering analysis on the text data set. The clustering analysis module takes the original text data set as input, and outputs preliminary clustering labels and a topic representation vector of each cluster based on hierarchical clustering algorithm (such as Ward) and large model topic extraction processing on the text data set. Among them, the preliminary clustering labels can be the clustering result of the text data set, and the topic representation vector of each cluster can be the semantic topics of the sample data in the text data set.

[0084] Exemplarily, the specific steps of the clustering algorithm adopted by the clustering analysis module can include: 1, sample representation: converting the samples in the text data set into vector representation, commonly using TF-IDF, sentence vector (such as BERT, SBERT Embedding). 2, similarity calculation: calculate the Euclidean distance or cosine similarity based on the converted vector to form a distance matrix. 3, clustering method selection: adopt Ward's linkage to recursively merge samples or class clusters by minimizing intra-class variance. 4, construct tree clustering structure: generate a tree structure dendrogram, and each merging operation corresponds to a parent node in the label tree.

[0085] In this case, the step of "constructing a multi-layer label tree based on the clustering result and the semantic topic" in the above step S102 can include the following steps:

[0086] Generating a semantic node name of a target node in the tree clustering result based on the semantic topic, so as to construct the tree clustering result into a multi-layer label tree.

[0087] When the terminal device constructs a multi-layer label tree based on the clustering result and the semantic topic, the semantic topic can be used as a label tree node name of the tree clustering result, so that a multi-layer label tree reflecting the semantic hierarchy is constructed.

[0088] It should be noted that the label tree node name named by the semantic topic in the multi-layer label tree is the basic framework for further label tagging and label weight calculation. Therefore, the accuracy of the topic extraction based on the large model on the sample data in the text data set determines the quality of the semantic structure of the label tree.

[0089] In some embodiments, the terminal device can use a label tree construction module in an integrated multi-dimensional data analysis framework to input the clustering result of the text data set, so as to construct a multi-layer label tree based on the clustering result by the label tree construction module, calculate the frequency and depth of each node in the multi-layer label tree, and output a label tree structure with frequency information.

[0090] In some embodiments, when the terminal device calculates the weight of each node in the multi-layer label tree based on the label distribution of the text data set to generate a weighted frequency label tree, it can determine the label frequency corresponding to each node in the multi-layer label tree based on the label distribution, so as to generate a weighted label tree (weighted frequency label tree) in combination with the label frequency and the tree clustering result. In this way, the data distribution has hierarchical interpretability, which is better than the flat label statistical method adopted in the related art.

[0091] In some embodiments, the terminal device can further employ a weight calculation and normalization module in the integrated multi-dimensional data analysis framework to take the frequency and depth of the multi-layer label tree and its nodes as input, and thus calculate the weight w of each node i based on the weight calculation and normalization module using Formula 1 as shown below i , and generate a weighted frequency label tree and output by performing horizontal / vertical normalization processing on the label tree.

[0092] w i = alpha f i + beta(1 / L i ), Formula 1

[0093] where f i is the frequency of node i, and L i is the depth of node i.

[0094] Step S104: Extracting quantitative indicators of multiple dimensions of the text data set based on the weighted frequency label tree to evaluate the quality of the text data set.

[0095] After the terminal device constructs the weighted frequency label tree, it can further calculate indicators from multiple dimensions based on this, and thus extract quantitative indicators of multiple dimensions of the text data set. In this way, the terminal device can independently evaluate the text data set in terms of semantic coverage, task structure, language balance, and text complexity based on the extracted quantitative indicators, thus forming a panoramic portrayal of the quality of the text data set.

[0096] In some embodiments, the terminal device can extract quantitative indicators from four dimensions of domain diversity, instruction diversity, complexity, and Chinese-English ratio when extracting quantitative indicators of multiple dimensions of the text data set based on the weighted frequency label tree.

[0097] Based on this, the step of "extracting quantitative indicators of multiple dimensions of the text data set based on the weighted frequency label tree" in the above step S104 can include the following steps:

[0098] Based on the weighted frequency label tree and the sample label frequency information of the text data set, the target dimension indicator is calculated to obtain quantitative indicators of multiple dimensions of the text data set. The sample label frequency information includes the frequency of the sample data in the text data set, and the target dimension indicator includes at least two of the domain diversity indicator, the instruction diversity indicator, the complexity indicator, and the Chinese-English ratio indicator.

[0099] It should be noted that the sample label frequency information of the text data set is the label distribution frequency of the samples in the text data set. The terminal device can pre-label the sample data in the text data set, so that the label distribution frequency of the sample data in the text data set can be calculated.

[0100] In some embodiments, the terminal device can use a label tagging module in the integrated multi-dimensional data analysis framework to take the text data set as input, so as to perform multi-label tagging on the sample data in the text data set according to the domain through the label tagging module, and / or perform single-label tagging on the sample data according to the instruction through the label tagging module, and then output a labeled sample data structure. Based on the sample data structure, the label distribution frequency of the sample data in the text data set can be obtained by performing sample label statistics.

[0101] When the terminal device extracts the quantization indicators of multiple dimensions of the text data set based on the weighted frequency label tree, the terminal device combines the weighted frequency label tree and the sample label frequency information (the label distribution frequency of the sample data in the text data set) of the text data set to calculate the target dimension indicators from the domain diversity, the instruction diversity, the complexity, and the English-Chinese ratio, i.e., to calculate the domain diversity indicators from the domain diversity dimension, to calculate the instruction diversity indicators from the instruction diversity dimension, to calculate the complexity indicators from the complexity dimension, and to calculate the English-Chinese ratio indicators from the English-Chinese ratio dimension. Among them, the terminal device can calculate the corresponding indicators from any two or more of the above four dimensions. In this way, the terminal device can obtain the quantization indicators of multiple dimensions of the entire text data set, i.e., the quantization indicators of at least two dimensions.

[0102] In some embodiments, the terminal device can use an indicator calculation module in the integrated multi-dimensional data analysis framework to take the weighted frequency label tree and the sample label frequency information as input, so as to perform information entropy, instruction distribution comparison, complexity calculation, and English-Chinese ratio statistics based on the indicator calculation module, and obtain the four-dimensional indicators of domain diversity D_domain, instruction diversity D_instruction, complexity D_complexity, and English-Chinese ratio D_lang.

[0103] In some embodiments, after obtaining the quantification indicators of multiple dimensions of the text data set, the terminal device can use the indicators to comprehensively quantify the distribution characteristics of the data set from different dimensions when performing quality evaluation on the text data set based on the indicators. For example, based on the domain diversity indicator, it can be identified whether there is a problem of "unbalanced domain coverage (e.g., some topics are too concentrated)" in the data set, based on the instruction diversity indicator, it can be identified whether there is a problem of "single instruction type that affects multi-task generalization" in the data set, based on the complexity indicator, it can be identified whether there is a problem of "unreasonable text structure complexity that affects model fitting ability" in the data set, and / or based on the Chinese-English ratio indicator, it can be identified whether there is a problem of "unbalanced language distribution that affects Chinese-English multi-language ability" in the data set. In this way, the terminal device extracts quantification indicators from four dimensions of domain diversity, instruction diversity, complexity, and Chinese-English ratio, and can establish a systematic evaluation mechanism for the structure rationality, training adaptability, and generalization ability of the text data set through these indicators, thereby providing an important basis for supporting data governance and data optimization.

[0104] In the embodiments of the present application, the terminal device obtains a text data set to be evaluated for quality, and then processes the text data set by combining hierarchical clustering and large model semantic extraction, i.e., by performing uneven clustering analysis on the text data set to obtain a clustering result of the text data set, and performing topic extraction on sample data in the text data set based on a pre-set large model to obtain semantic topics of the sample data, and further constructing a label tree structure that can reflect semantic levels, i.e., a multi-layer label tree, based on the clustering result and the semantic topics, and for each node in the multi-layer label tree, calculating the weight of each node in combination with the label distribution of the text data set, thereby generating a weighted frequency label tree. Finally, based on the weighted frequency label tree, indicators are calculated from multiple dimensions, thereby extracting quantification indicators of multiple dimensions of the text data set, to independently evaluate the text data set in terms of semantic coverage, task structure, language balance, and text complexity based on the extracted quantification indicators, thereby constituting a panoramic characterization of the quality of the text data set.

[0105] Compared with the quality evaluation of a large-scale data set from a single dimension, the embodiment of the application performs hierarchical clustering analysis on a large-scale text data set to be evaluated and extracts semantic topics from sample data based on a preset large model, constructs a multi-layer label tree capable of reflecting semantic levels based on the clustering results obtained by the hierarchical clustering analysis and the semantic topics obtained by the topic extraction, calculates the weights of nodes of the multi-layer label tree through label distribution of the text data set to generate a weighted frequency label tree, and finally extracts quantitative indicators of multiple dimensions based on the weighted frequency label tree to evaluate the quality of the text data set. In this way, the embodiment of the application combines hierarchical clustering of a large-scale data set with semantic extraction of a large model to construct a label tree structure capable of reflecting semantic levels, and further constructs a weighted frequency label tree, and on this basis, quantitative indicators of multiple dimensions are extracted to evaluate the quality of the data set, for example, the field diversity, instruction coverage, complexity and language proportion are analyzed at the same time, a complete data set portrait is constructed, so that comprehensive analysis of a large-scale data set can be realized to finely manage and optimize the data quality, thereby improving the adaptability, balance and quality control capability of the data set in the multi-task large model training scenario.

[0106] In some embodiments, the terminal device can obtain a weighted frequency label tree for the candidate text data set and the reference text data set through the above operations, that is, the text data set includes the reference text data set and the candidate text data set, and the weighted frequency label tree includes a first weighted frequency label tree of the reference text data set and a second weighted frequency label tree of the candidate text data set. Then, the terminal device compares and analyzes the structure between the candidate text data set and the reference text data set based on the two weighted frequency label trees, so as to improve the adaptability, balance and quality control capability of the data set in the multi-task model training scenario by comparing and analyzing the structure between the candidate text data set and the reference text data set.

[0107] It should be noted that the reference text data set is usually an existing mature data set, for example, a training corpus of a certain large model, and it can be considered that the reference text data set has an ideal data distribution. The structure of the first weighted frequency label tree represents a control sample with an ideal data distribution. The candidate text data set can be a data set to be evaluated in quality. The second weighted frequency label tree represents the structural characteristics of the data set to be evaluated in quality.

[0108] For reference Figure 2 , Figure 2 The step flowchart of the data set quality evaluation method provided by the embodiment of the application in some other embodiments.

[0109] As Figure 1As shown, in some embodiments, the method for evaluating the quality of a data set provided by the embodiments of the present application can further include steps S201 and S202 as shown below.

[0110] Step S201: Cross-entropy comparison is performed on the first weighted frequency label tree and the second weighted frequency label tree to obtain the structural similarity between the first weighted frequency label tree and the second weighted frequency label tree.

[0111] The terminal device can generate the first weighted frequency label tree of the benchmark text data set and the second weighted frequency label tree of the candidate text data set by performing the operations described in the above embodiments on the candidate text data set and the benchmark text data set, respectively. After that, the terminal device obtains the structural similarity between the first weighted frequency label tree and the second weighted frequency label tree by performing cross-entropy comparison on the first weighted frequency label tree and the second weighted frequency label tree.

[0112] In some embodiments, the terminal device can use a cross-set comparison module in an integrated multi-dimensional data analysis framework to input the benchmark text data set label tree (the first weighted frequency label tree) and the candidate text data set label tree (the second weighted frequency label tree), and then perform cross-entropy calculation and normalized output similarity processing on the two weighted frequency label trees based on the cross-set comparison module to obtain a normalized cross-entropy index and output. The normalized cross-entropy index output by the cross-set comparison module is the structural similarity between the first weighted frequency label tree and the second weighted frequency label tree.

[0113] Step S202: Based on the structural similarity, the structural distribution difference between the candidate text data set and the benchmark text data set is evaluated to obtain a structural distribution difference evaluation result; the structural distribution difference evaluation result is one of the results of the quality evaluation of the candidate text data set.

[0114] After obtaining the structural similarity between the first weighted frequency label tree and the second weighted frequency label tree, the terminal device performs structured comparison analysis between the candidate text data set and the benchmark text data set based on the structural similarity to evaluate the structural distribution difference between the two data sets, thereby obtaining a structural distribution difference evaluation result. In this way, the structural distribution difference evaluation result can also be one of the results of the quality evaluation of the candidate text data set.

[0115] In this embodiment, by the terminal device, on the basis of constructing a weighted frequency label tree for the text data set, further through the weighted frequency label tree and cross entropy calculation, the distribution similarity analysis of the candidate text data set and the benchmark text data set at multiple levels can be realized, thereby supporting cross-data set structured comparison and providing quantitative basis for data selection and optimization. That is, by evaluating the structural distribution difference between different data sets (candidate text data set and benchmark text data set), the selection and optimization of cross-corpus data are supported.

[0116] In some embodiments, when the terminal device performs topic extraction based on the preset large model, the text prompt can be designed in advance, and then the large model is guided to accurately extract semantic topics based on the text prompt.

[0117] Please refer to Figure 3 , Figure 3 for Figure 1 the detailed step flowchart of step S102.

[0118] As Figure 3 shown, in some embodiments, the step of "performing topic extraction on the sample data in the text data set based on the preset large model" in step S102 can include steps S301 and S302 as shown below.

[0119] Step S301: obtaining a preset text prompt; the text prompt at least includes data processing requirements and output requirements of data processing results.

[0120] It should be noted that the data processing requirement in the text prompt can be what the large model needs to solve for the input data, such as summarizing topics, etc. In addition, the output requirement of the data processing result in the text prompt can be the output form of the result obtained by the large model for solving the problem, such as outputting clear and concise topic words or phrases, etc. For example, the text prompt can be "You are a language expert, please summarize the common theme of the following text collection, and require to output clear and concise topic words or phrases".

[0121] The terminal device can combine the real-time design of data processing requirements and data processing result output requirements of the text data set in the process of quality evaluation of the text data set, thereby generating the text prompt.

[0122] In some embodiments, the terminal device can also receive the text prompt designed by the staff manually through a preset data interface, such as a human-computer interaction interface.

[0123] Step S302: based on the text prompt, instruct the pre-trained large model to perform topic extraction on the sample data in the text data set according to the processing requirement, and output the first semantic topic extracted according to the output requirement; the semantic topic of the sample data includes the first semantic topic.

[0124] After the terminal device obtains the text prompt, the terminal device inputs the text prompt and the text data set to be subjected to topic extraction into the previously selected large model, so as to guide the large model to perform topic extraction on the sample data in the text data set based on the text prompt. Among them, after receiving the input text data set and the text prompt, the large model performs topic extraction on the sample data in the text data set based on the data processing requirement in the text prompt, and further outputs the first semantic topic of the sample data according to the data processing result output requirement in the text prompt. In this way, the terminal device can output the first semantic topic of the large model as the semantic topic of the sample data in the text data set.

[0125] In some embodiments, the terminal device can also perform hierarchical clustering on the text data set, and perform topic extraction on each clustering cluster in the clustering result based on the preset large model, so as to generate the topic of the large model as the semantic topic of the sample data in the text data set.

[0126] Based on this, after the step of "performing hierarchical clustering analysis on the text data set to obtain the clustering result of the text data set" in the above step S102, the quality evaluation method of the data set provided by the embodiments of the present application can also include the following steps:

[0127] Based on the preset text prompt, instruct the pre-trained large model to generate a second semantic topic for the clustering cluster in the clustering result, and take the second semantic topic as the semantic topic of the sample data in the text data set.

[0128] It should be noted that the text prompt here can be the same as the text prompt described above. Alternatively, the terminal device can also design the processing requirement of the data and the output requirement of the data processing result in real time based on the clustering result of the text data set, so as to generate the text prompt.

[0129] After the terminal device performs hierarchical clustering on the text data set to obtain the clustering result of the text data set, the terminal device further inputs the clustering result and a preset text prompt into the previously selected large model, so that the large model directly performs topic extraction on each cluster in the input clustering result based on the text prompt to generate a second semantic topic of each cluster. In this way, the terminal device can take the second semantic topic generated and output by the large model as the semantic topic of the sample data in the text data set.

[0130] It should be noted that the step of obtaining the second semantic topic by performing topic extraction on the cluster in the clustering result based on the large model is performed by the terminal device in the step S302 described above. Alternatively, in order to improve accuracy, the terminal device can also perform both steps, and then select the semantic topic with higher accuracy from the first semantic topic and the second semantic topic as the final semantic topic of the sample data in the text data set. For example, the terminal device can display the first semantic topic and the second semantic topic through the human-computer interaction interface, so that the semantic topic selected by the staff through manual inspection is taken as the final semantic topic of the sample data.

[0131] In some embodiments, when the terminal device performs topic extraction on the cluster in the clustering result based on the large model, the terminal device can take the sample content in each cluster as input of the large model after being combined and summarized, so that the large model directly processes the summary under the guidance of the text prompt to generate the semantic topic of the cluster.

[0132] In some embodiments, when the terminal device performs topic extraction on the cluster in the clustering result based on the large model, the terminal device can directly select a representative sample corresponding to the cluster from the text data set as input of the large model, so that the large model processes the representative sample under the guidance of the text prompt to generate the semantic topic of the cluster.

[0133] In this embodiment, the terminal device obtains a text prompt, and then guides the large model to perform topic extraction to generate a semantic topic based on the text prompt, so as to combine hierarchical clustering processing on the text data set and large model semantic extraction processing on the text data set to construct a label tree structure that can reflect semantic levels for subsequent generation of a weighted frequency label tree and calculation of a plurality of dimensions of quantitative indexes of the text data set, so that the data distribution has hierarchical interpretability through this structure modeling manner, which has obvious advantages compared with the flat label statistics manner in the related art.

[0134] In some embodiments, when the terminal device calculates the weight of each node in the multi-layer label tree to generate the weighted frequency label tree, the terminal device can calculate the weight of each node in the multi-layer label tree based on the frequency and depth of each node.

[0135] Please refer to Figure 4 , Figure 4 As Figure 1 the refinement step flowchart of step S103.

[0136] As Figure 4 shown, in some embodiments, the step of "calculating the weight of each node in the multi-layer label tree based on the label distribution of the text data set" in step S103 described above can include steps S401 to S403 as shown below.

[0137] Step S401: Determine the frequency of each node in the multi-layer label tree based on the label distribution of the text data set.

[0138] When the terminal device calculates the weight of each node in the multi-layer label tree, it first determines the frequency of the node in the multi-layer label tree for which the weight is to be calculated based on the label distribution of the text data set.

[0139] In some embodiments, the terminal device can "match" and "map" the labels of the sample data in the text data set to each node in the multi-layer label tree, i.e., determine which topic cluster (node) or multiple topic branches (node set) in the multi-layer label tree the sample belongs to, i.e., the attribution binding from the sample data to the multi-layer label tree structure. In this way, the terminal device can obtain the frequency of each node in the multi-layer label tree based on the frequency of the sample data.

[0140] In some embodiments, the terminal device can label the sample data in the text data set before calculating the weight of each node in the multi-layer label tree, thereby obtaining the label distribution of the text data set. Then, when calculating the weight of each node in the multi-layer label tree, the terminal device can obtain the frequency of the sample data in the text data set based on the label distribution, and further obtain the frequency of each node in the multi-layer label tree.

[0141] Based on this, the quality evaluation method for the data set provided by the embodiments of the present application can further include the following steps:

[0142] Add labels to the sample data in the text data set, and determine the label distribution of the text data set based on the labels of the sample data; the labels of the sample data correspond to the nodes in the multi-layer label tree.

[0143] The terminal device can perform multi-label labeling on each piece of sample data in the text data set according to the field, or perform single-label labeling on each piece of sample data according to the instruction (for example, the instruction type). Among them, the label added by the terminal device for the sample data in the text data set is "matched" and "mapped" with the node (topic node) in the multi-level label tree extracted and named based on the large model, that is, the label of the sample data corresponds to the node in the multi-level label tree. In this way, the terminal device can obtain the label distribution of the entire text data set by counting the label of each piece of sample data in the text data set.

[0144] In this case, the step S401 of determining the frequency of each node in the multi-level label tree based on the label distribution of the text data set can include the following steps:

[0145] determining the frequency of the sample data in the text data set based on the label distribution of the text data set;

[0146] counting the frequency of each node in the multi-level label tree based on the frequency of the sample data.

[0147] After the terminal device determines the label distribution of the text data set, when determining the frequency of the node in the multi-level label tree based on the label distribution, the terminal device can first determine the frequency of the sample data in the text data set based on the label distribution, and then count the frequency of the node in the multi-level label tree corresponding to the label of the sample data based on the frequency of the sample data.

[0148] The step S402 of performing weight calculation based on the frequency of each node and the depth of each node in the multi-level label tree to obtain the weight of each node.

[0149] When the terminal device calculates the weight of each node in the multi-level label tree, after determining the frequency of the node in the multi-level label tree for which the weight is currently calculated, the terminal device can further use the following formula 2 to perform weight calculation in combination with the frequency fi of the node i and the depth (level) Li of the node i in the multi-level label tree, so as to obtain the weight of the node i .

[0150] , formula 2

[0151] wherein α and β are both adjustment coefficients.

[0152] Step S403: normalizing the weight of each node to generate a weighted frequency label tree; wherein the normalization includes horizontal normalization and vertical normalization, the horizontal normalization is used for normalizing the weight of the nodes belonging to the same level in the multi-layer label tree, and the vertical normalization is used for normalizing the weight of all nodes in the multi-layer label tree.

[0153] When the terminal device calculates the weight of each node in the multi-layer label tree, after the weight of each node in the multi-layer label tree is calculated, the weighted frequency label tree is further generated by combining the horizontal and vertical normalization methods. That is, the weighted frequency label tree is generated by performing horizontal normalization and vertical normalization on the weight of each node. Among them, the horizontal normalization of the weight of each node refers to normalizing the weight of the nodes belonging to the same level in the multi-layer label tree, and the vertical normalization of the weight of each node refers to normalizing the weight of each node in the multi-layer label tree.

[0154] In some embodiments, when the terminal device performs horizontal normalization on the weight of each node in the multi-layer label tree, the weight of the node set in each layer of the multi-layer label tree can be normalized by using formula 3 as shown below, so that the sum of the weights of the node set in each layer of the multi-layer label tree is 1.

[0155] Formula 3.

[0156] Among them, wi,j represents the weight value of node i after horizontal normalization, j represents the index variable of the other nodes in the same layer as node i, N represents the number of nodes in the same layer as node i, and l wi,j represents the weight value of node i after horizontal normalization, j represents the index variable of the other nodes in the same layer as node i, N represents the number of nodes in the same layer as node i, and j wi,j represents the weight value of node i after horizontal normalization, j represents the index variable of the other nodes in the same layer as node i, N represents the number of nodes in the same layer as node i, and

[0157] In some embodiments, when the terminal device performs vertical normalization on the weight of all nodes in the multi-layer label tree, the weight of the node set in the entire multi-layer label tree can be normalized by using formula 4 as shown below, so that the overall weight is normalized.

[0158] Formula 4.

[0159] Among them, wi,j represents the weight value of node i after horizontal normalization, j represents the index variable of the other nodes in the same layer as node i, N represents the number of nodes in the same layer as node i, and wi,j represents the weight value of node i after horizontal normalization, j represents the index variable of the other nodes in the same layer as node i, N represents the number of nodes in the same layer as node i, and

[0160] In this embodiment, the terminal device calculates the weight based on the frequency of each node in the multi-layer label data and the depth of each node in the multi-layer label tree, introduces horizontal and vertical normalization methods to generate a weighted frequency label tree, so that the distribution similarity of different data sets at multiple levels can be analyzed by subsequent weighted frequency label tree and cross entropy calculation, thereby providing quantitative basis for data selection and optimization. Moreover, by constructing the weighted frequency label tree, the multi-dimensional data analysis framework of being evaluable, interpretable and comparable can be constituted by combining the previously constructed multi-layer label tree, the subsequent calculation of multi-dimensional indexes such as field diversity, instruction diversity, complexity and English proportion, and the comparison of normalized distribution similarity between different data sets based on cross entropy, so as to realize the comprehensive analysis of large-scale data sets for fine management and optimization of data quality, thereby improving the adaptability, balance and quality control ability of the data set in the multi-task large model training scene.

[0161] Next, a complete embodiment of the data set quality evaluation method provided by the embodiments of the present application is proposed.

[0162] Please refer to Figure 5 , Figure 5 is a hierarchical clustering and multi-dimensional data set distribution analysis process schematic diagram related to a complete embodiment of the data set quality evaluation method provided by the embodiments of the present application.

[0163] As Figure 5As shown, the data set quality evaluation method provided by the embodiments of the present application can be used to input the original text data into the clustering analysis module after the terminal device obtains the original text data (text data set) to be evaluated, so as to perform hierarchical clustering and large model topic extraction on the original text data by the clustering analysis module, and output the preliminary clustering label and the per-cluster topic representation vector. Then, the terminal device inputs the clustering result obtained by the clustering analysis module and the semantic topic extracted by the large model into the label tree construction module, constructs a multi-layer label tree by the label tree construction module, calculates the frequency and depth of each node in the multi-layer label tree, and outputs the label tree structure with frequency information. Moreover, the terminal device inputs the original text data into the label tagging module, so as to perform multi-label tagging according to the field by the label tagging module, and / or perform single-label tagging according to the instruction by the label tagging module, and then outputs the sample data structure with labels. Then, the terminal device inputs the label tree structure with frequency information and the sample data structure with labels into the weight calculation and normalization module, calculates the weight of each node by the weight calculation and normalization module, and generates the weighted frequency label tree by performing horizontal / vertical normalization on the label tree. Finally, the terminal device inputs the weighted frequency label tree into the index calculation module to perform information entropy, instruction distribution comparison, complexity calculation, and English proportion statistics, obtains the four-dimensional indexes of field diversity D_domain, instruction diversity D_instruction, complexity D_complexity, and English proportion D_lang, and uses the cross-set comparison module to perform cross-entropy calculation and normalized output similarity comparison on the benchmark text data set label tree (the weighted frequency label tree of the benchmark text data set with ideal data distribution) and the candidate text data set label tree (the weighted frequency label tree corresponding to the original text data), obtains the normalized cross-entropy index, and then outputs the evaluation report and the similarity index for quality evaluation of the original text data.

[0164] In some embodiments, the data set quality evaluation method provided by the embodiments of the present application can be used to match and map the label tagged by the label tagging module to each sample (sample data in the original text data) with the topic node extracted by the large model, so as to complete the attribution binding of the sample to the label tree structure.

[0165] As shown, Figure 6As shown, the terminal device extracts the label tree topic node by performing large model processing on the cluster sample (the sample content in the cluster in the clustering result obtained by performing hierarchical clustering on the original text), and inputs the cluster sample into the label tagging module to obtain sample label information, and then combines the label tree topic node and the sample label information to construct a label tree to realize the binding between the sample and the node in the label tree. Moreover, the terminal device can perform semantic consistency verification on the label tree structure (multi-layer label tree) after the binding of the sample and the node is completed. For example, the terminal device can display the label tree structure through a preset human-computer interaction interface, so that a staff member performs semantic consistency verification.

[0166] In some embodiments, the data set quality evaluation method provided by the embodiments of the present application can also be performed by the terminal device according to the process of measuring data quality based on the four indicators as shown. Figure 7 As shown, the terminal device extracts the label tree topic node by performing large model processing on the cluster sample (the sample content in the cluster in the clustering result obtained by performing hierarchical clustering on the original text), and inputs the cluster sample into the label tagging module to obtain sample label information, and then combines the label tree topic node and the sample label information to construct a label tree to realize the binding between the sample and the node in the label tree. Moreover, the terminal device can perform semantic consistency verification on the label tree structure (multi-layer label tree) after the binding of the sample and the node is completed. For example, the terminal device can display the label tree structure through a preset human-computer interaction interface, so that a staff member performs semantic consistency verification.

[0167] Please refer to Figure 8 The embodiments of the present application also provide a data set quality evaluation device, which can implement the data set quality evaluation method.

[0168] As shown Figure 8 The data set quality evaluation device provided by the embodiments of the present application includes an acquisition module 801, a clustering and model topic extraction module 802, a label tree construction module 803, and a data set quality evaluation module 804. Wherein,

[0169] The acquisition module 801 is configured to acquire a text data set to be evaluated for quality;

[0170] The clustering and model topic extraction module 802 is configured to perform hierarchical clustering analysis on the text dataset to obtain a clustering result of the text dataset, and perform topic extraction on sample data in the text dataset based on a preset large model to obtain semantic topics of the sample data.

[0171] The label tree construction module 803 is configured to construct a multi-layer label tree based on the clustering result and the semantic topics, and calculate weights of nodes in the multi-layer label tree based on a label distribution of the text dataset to generate a weighted frequency label tree.

[0172] The dataset quality evaluation module 804 is configured to extract quantization indicators of multiple dimensions of the text dataset based on the weighted frequency label tree to evaluate the quality of the text dataset.

[0173] In some embodiments, the clustering and model topic extraction module 802 is further configured to obtain a preset text prompt, wherein the text prompt at least includes a data processing requirement and an output requirement of a data processing result; and guide the pre-trained large model to perform topic extraction on the sample data in the text dataset according to the processing requirement and output the first semantic topic extracted according to the output requirement; and the semantic topics of the sample data include the first semantic topic.

[0174] In some embodiments, the clustering and model topic extraction module 802 is further configured to guide the pre-trained large model to generate a second semantic topic for a cluster in the clustering result based on the preset text prompt, and take the second semantic topic as a semantic topic of the sample data in the text dataset.

[0175] In some embodiments, the clustering result includes a tree-shaped clustering result; and the label tree construction module 803 is further configured to generate a semantic node name of a target node in the tree-shaped clustering result based on the semantic topics, so as to construct the tree-shaped clustering result into a multi-layer label tree.

[0176] In some embodiments, the label tree construction module 803 is further configured to determine a frequency of each node in the multi-layer label tree based on a label distribution of the text dataset, calculate a weight of each node based on the frequency of each node and a depth of each node in the multi-layer label tree, and perform normalization processing on the weight of each node to generate a weighted frequency label tree; wherein the normalization processing includes horizontal normalization processing and vertical normalization processing, the horizontal normalization processing is configured to perform normalization calculation on the weights of nodes belonging to the same level in the multi-layer label tree, and the vertical normalization processing is configured to perform normalization calculation on the weights of all nodes in the multi-layer label tree.

[0177] In some embodiments, the label tree construction module 803 is further configured to add labels to sample data in the text data set, and determine a label distribution of the text data set based on the labels of the sample data; the labels of the sample data correspond to nodes in the multi-layer label tree; and determine the frequency of the sample data in the text data set based on the label distribution of the text data set; and count the frequency of each node in the multi-layer label tree based on the frequency of the sample data.

[0178] In some embodiments, the data set quality evaluation module 804 is further configured to calculate target dimension indicators based on the weighted frequency label tree and sample label frequency information of the text data set, to obtain quantization indicators of multiple dimensions of the text data set; wherein the sample label frequency information includes the frequency of the sample data in the text data set; and the target dimension indicators include at least two of the field diversity indicator, the instruction diversity indicator, the complexity indicator, and the English-Chinese ratio indicator.

[0179] In some embodiments, the text data set includes a benchmark text data set and a candidate text data set, and the weighted frequency label tree includes a first weighted frequency label tree of the benchmark text data set and a second weighted frequency label tree of the candidate text data set; the data set quality evaluation module 804 is further configured to perform cross-entropy comparison on the first weighted frequency label tree and the second weighted frequency label tree, to obtain a structural similarity between the first weighted frequency label tree and the second weighted frequency label tree; and evaluate the structural distribution difference between the candidate text data set and the benchmark text data set based on the structural similarity, to obtain a structural distribution difference evaluation result; the structural distribution difference evaluation result is one of the results of quality evaluation on the candidate text data set.

[0180] Please refer to Figure 9 , Figure 9 Fig. 1 shows a hardware structure of a data set quality evaluation device according to an embodiment of the present application. The data set quality evaluation device includes:

[0181] The processor 901 can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., and is configured to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0182] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 902 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the related program codes are stored in the memory 902 and are called and executed by the processor 901 to implement the data set quality evaluation method of the embodiments of the present application;

[0183] The input / output interface 903 is configured to realize information input and output.

[0184] The communication interface 904 is configured to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0185] The bus 905 is configured to transmit information between various components (for example, the processor 901, the memory 902, the input / output interface 903, and the communication interface 904) of the device.

[0186] The processor 901, the memory 902, the input / output interface 903, and the communication interface 904 are connected to each other through the bus 905 to realize the communication connection between the device.

[0187] The embodiments of the present application also provide a vehicle. The vehicle is provided with a data set quality evaluation device. The data set quality evaluation device comprises a memory and a processor. The memory stores a computer program. The processor implements the data set quality evaluation method when executing the computer program.

[0188] The embodiments of the present application also provide a computer readable storage medium. The computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the data set quality evaluation method.

[0189] The memory is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor. These remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0190] The embodiment of the present application further provides a computer program product comprising a computer program which, when executed by a processor, implements steps substantially the same as those of the specific embodiment of the data set quality evaluation method, and thus is not described herein again.

[0191] The embodiments described in the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0192] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than shown in the figures, or combine certain steps or different steps.

[0193] The device embodiments described above are merely illustrative, and the units described as separate components can or can not be physically separated, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0194] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the functions of the modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0195] The terms "first", "second", "third", "fourth" and the like (if any) in the specification of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0196] It should be understood that, in the application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B, and A and B existing at the same time, wherein A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b, and c can be single or multiple.

[0197] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0198] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. According to actual needs, some or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0199] In addition, the functional units in each embodiment of the application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0200] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0201] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A method of quality assessment of a data set, characterized in that, The method comprises: Obtain a text dataset whose quality is to be evaluated; Performing a hierarchical clustering analysis on the text dataset to obtain a clustering result of the text dataset, and performing topic extraction on sample data in the text dataset based on a preset large model to obtain semantic topics of the sample data; Constructing a multi-layer tag tree based on the clustering results and the semantic topics, and calculating the weight of each node of the multi-layer tag tree based on the tag distribution of the text data set to generate a weighted frequency tag tree; Extracting quantitative indicators of multiple dimensions of the text dataset based on the weighted frequency label tree to perform quality assessment on the text dataset; The quality assessment of the text dataset includes: Comprehensively quantifying the distribution characteristics of the text dataset from multiple dimensions based on the quantitative indicators of the multiple dimensions; The step of calculating the weight of each node of the multi-layer tag tree based on the tag distribution of the text data set to generate a weighted frequency tag tree includes: The frequency fi of each node in the multi-layer label tree is determined based on the label distribution of the text data set, and the weight of each node is calculated based on the frequency fi of each node and the depth Li of each node in the multi-layer label tree. , α and β are adjustment coefficients, which are used to adjust the weight of each node. Normalization is performed to generate a weighted frequency label tree.

2. The method of claim 1, wherein, The subject extraction of the sample data in the text dataset based on the preset large model includes: Obtaining a preset text prompt; the text prompt at least includes data processing requirements and output requirements of data processing results; Based on the text prompt, guide the pre-trained large model to extract topics from the sample data in the text dataset according to the processing requirements, and output the extracted first semantic topics according to the output requirements; the semantic topics of the sample data include the first semantic topics.

3. The method of claim 1, wherein, After performing hierarchical cluster analysis on the text data set to obtain a clustering result of the text data set, the method further includes: Based on preset text prompts, the pre-trained large model is guided to generate a second semantic theme for the cluster cluster in the clustering result, and the second semantic theme is used as the semantic theme of the sample data in the text dataset.

4. The method of claim 1, wherein, The clustering result includes a tree-shaped clustering result; and constructing a multi-layer tag tree based on the clustering result and the semantic topic includes: A semantic node name of a target node in the tree clustering result is generated based on the semantic topic, so as to construct the tree clustering result into a multi-layer label tree.

5. The method of claim 1, wherein, The normalization processing includes horizontal normalization processing and vertical normalization processing. The horizontal normalization processing is used to normalize the weights of nodes belonging to the same level in the multi-layer label tree, and the vertical normalization processing is used to normalize the weights of all nodes in the multi-layer label tree.

6. The method of claim 5, wherein, The method further comprises: Adding labels to sample data in the text dataset, and determining a label distribution of the text dataset based on the labels of the sample data; the labels of the sample data correspond to nodes in the multi-layer label tree; Determining the frequency of each node in the multi-layer tag tree based on the tag distribution of the text data set includes: Determining the frequency of sample data in the text dataset based on the label distribution of the text dataset; The frequency of each node in the multi-layer tag tree is counted based on the frequency of the sample data.

7. The method of claim 1, wherein, The step of extracting quantitative indicators of multiple dimensions of the text dataset based on the weighted frequency label tree includes: performing calculation of target dimension indicators based on the weighted frequency label tree and sample label frequency information of the text data set, to obtain quantization indicators of multiple dimensions of the text data set; wherein the sample label frequency information comprises frequency of sample data in the text data set; and the target dimension indicators comprise at least two of domain diversity indicators, instruction diversity indicators, complexity indicators, and English-Chinese proportion indicators.

8. The method according to any one of claims 1 to 7, characterized in that, The text data set comprises a benchmark text data set and a candidate text data set, and the weighted frequency label tree comprises a first weighted frequency label tree of the benchmark text data set and a second weighted frequency label tree of the candidate text data set. The method further comprises: performing cross-entropy comparison on the first weighted frequency label tree and the second weighted frequency label tree, to obtain structural similarity between the first weighted frequency label tree and the second weighted frequency label tree; based on the structural similarity, evaluating structural distribution difference between the candidate text data set and the benchmark text data set, to obtain a structural distribution difference evaluation result; the structural distribution difference evaluation result is one of results of quality evaluation on the candidate text data set.

9. A data set quality assessment apparatus, characterized by comprising: The device comprises: an acquisition module configured to acquire a text data set to be evaluated for quality; a clustering and model topic extraction module configured to perform hierarchical clustering analysis on the text data set, to obtain a clustering result of the text data set, and to perform topic extraction on sample data in the text data set based on a preset large model, to obtain semantic topics of the sample data; a label tree construction module configured to construct a multi-layer label tree based on the clustering result and the semantic topics, and to generate a weighted frequency label tree based on weights of nodes of the multi-layer label tree calculated based on label distribution of the text data set; a data set quality evaluation module configured to extract quantization indicators of multiple dimensions of the text data set based on the weighted frequency label tree, to evaluate quality of the text data set; the data set quality evaluation module is further configured to comprehensively quantify distribution characteristics of the text data set from multiple dimensions based on the quantization indicators of the multiple dimensions; The label tree construction module is further configured to determine a frequency fi of each node in the multi-layer label tree based on a label distribution of the text data set, and perform weight calculation based on the frequency fi of each node and a depth Li of each node in the multi-layer label tree, to obtain a weight of each node , and α and β are both adjustment coefficients, and the weight of each node is normalized to generate a weighted frequency label tree. ​ 10. A data set quality evaluation device, characterized by, The data set quality evaluation device comprises a memory and a processor, the memory stores a computer program, and the processor implements the data set quality evaluation method of any one of claims 1 to 8 when executing the computer program.

11. A computer readable storage medium characterized by, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the data set quality evaluation method of any one of claims 1 to 8.

12. A computer program product, characterised in that, The computer program product comprises a computer program, and the computer program is executed by the processor to implement the data set quality evaluation method of any one of claims 1 to 8.

Images