Data processing method and device, equipment, storage medium and program product

Abstract

The invention provides a data processing method and device, equipment, a storage medium and a program product, and relates to the technical field of natural language processing. The method comprises the following steps: preprocessing original thinking chain data, and determining ordered thinking chain information, question information and question and answer pair information; evaluating the data quality of the thinking chain information, the question information and the question and answer pair information based on a preset multi-dimensional evaluation index, and outputting a corresponding quality score and a label; and screening the quality scores and labels corresponding to the thinking chain information, the question information and the question and answer pair information according to a preset screening condition, and determining high-quality data which is used for model training. According to the method and the device, the problem of low model training accuracy caused by lack of a quantitative quality grading system for marking the thinking chain data is solved by constructing the multi-dimensional data quality evaluation for the thinking chain data.

Description

Technical Field

[0001] This disclosure relates to the field of natural language processing technology, and in particular to a data processing method, apparatus, device, storage medium, and program product. Background Technology

[0002] In the field of large-scale language model training and inference, thought chain technology has become a key means to improve the model's reasoning ability. By integrating thought chain data during pre-training, the model can simultaneously output the answer and the corresponding thought process during reasoning. In this reasoning model, the role of the thought chain is not only to improve the correctness of the answer, but its own reasoning process also plays a significant role and is crucial for user comprehension. Therefore, the quality of the thought chain data used for training becomes critical to model training.

[0003] In related technologies, manually labeled reasoning processes are pre-trained together with question-answer pairs to enable the model to automatically generate thought chains. However, the manual selection and labeling of thought chain data is subjective and lacks a quantitative quality grading system, which makes the model training susceptible to low-quality, highly subjective thought chain data, thus affecting the accuracy of model training.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] This disclosure provides a data processing method, apparatus, device, storage medium, and program product, which at least to some extent overcomes the problem of low model training accuracy caused by the lack of a quantitative quality grading system for labeled thought chain data in related technologies.

[0006] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0007] According to one aspect of this disclosure, a data processing method is provided, comprising: preprocessing raw thought chain data to determine ordered thought chain information, question information, and question-answer pair information; evaluating the data quality of the thought chain information, the question information, and the question-answer pair information based on preset multi-dimensional evaluation indicators, and outputting corresponding quality scores and labels; and filtering the quality scores and labels corresponding to the thought chain information, the question information, and the question-answer pair information according to preset filtering conditions to determine high-quality data, wherein the high-quality data is used for model training.

[0008] In some embodiments, the preprocessing of the original thought chain data to determine the thought chain information, question information, and question-answer pair information includes: unifying the format, converting the encoding, removing invalid characters, and standardizing the punctuation of the original thought chain data, and outputting preprocessed data; parsing the preprocessed data into sentences according to the rule base and the connection dictionary to determine the thought chain information and question information; and extracting the question field and answer field from the preprocessed data to determine the question-answer pair information.

[0009] In some embodiments, the evaluation of the data quality of the thought chain information, the question information, and the question-and-answer pair information based on preset multi-dimensional evaluation indicators, and the output of corresponding quality scores and tags, includes: evaluating the data quality of the thought chain information, the question information, and the question-and-answer pair information using preset ten-dimensional evaluation indicators, generating a tag for the content diversity dimension and quality scores for the other nine dimensions, wherein the ten-dimensional evaluation indicators include: reasoning atomicity, reasoning logic, reasoning informativeness, reasoning completeness, reasoning complexity, answer correctness, language purity, text compliance, text repetition, and content diversity.

[0010] In some embodiments, the quality score for the atomicity of reasoning is determined based on the smallest logical unit; the quality score for the logicality of reasoning is determined based on external and internal logic; the quality score for the informativeness of reasoning is determined based on new and invalid information; the quality score for the completeness of reasoning is determined based on completeness; the quality score for the complexity of reasoning is determined based on difficulty; the quality score for the correctness of the answer is determined based on correctness and linguistic purity; the quality score for text compliance is determined based on sensitive information; and the quality score for text redundancy includes determination based on redundant information.

[0011] In some embodiments, the step of filtering the quality scores and tags corresponding to the thought chain information, the question information, and the question-and-answer pair information according to preset filtering conditions to determine high-quality data includes: filtering the quality scores of nine dimensions according to preset thresholds to determine the datasets that meet the preset thresholds; and achieving category balance of the datasets that meet the preset thresholds under the tags of the content diversity dimension according to an imbalance processing strategy to determine the high-quality data.

[0012] In some embodiments, determining the high-quality data by achieving category balance for the datasets meeting a preset threshold under the labels of the content diversity dimension according to the imbalance processing strategy includes: determining the distribution of the datasets meeting the preset threshold under the labels of the content diversity dimension based on the datasets meeting the preset threshold and the labels of the content diversity dimension, wherein the distribution under the labels of the content diversity dimension includes high-frequency labels and low-frequency labels; and determining the high-quality data by undersampling the data of the high-frequency labels and resampling the data of the low-frequency labels according to the imbalance processing strategy.

[0013] According to another aspect of this disclosure, a data processing apparatus is also provided, comprising: a preprocessing module for preprocessing raw thought chain data to determine ordered thought chain information, question information, and question-answer pair information; an evaluation module for evaluating the data quality of the thought chain information, the question information, and the question-answer pair information based on preset multi-dimensional evaluation indicators, and outputting corresponding scores and labels; and a high-quality data output module for filtering the scores and labels corresponding to the thought chain information, the question information, and the question-answer pair information according to preset filtering conditions to determine high-quality data, wherein the high-quality data is used for model training.

[0014] According to another aspect of this disclosure, an electronic device is also provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the data processing method described in any of the preceding claims by executing the executable instructions.

[0015] According to another aspect of this disclosure, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the data processing method described in any of the preceding claims.

[0016] According to another aspect of this disclosure, a computer program product is also provided, including a computer program that, when executed by a processor, implements the data processing method described in any of the above.

[0017] The data processing method provided in the embodiments of this disclosure preprocesses the original thought chain data to determine ordered thought chain information, question information, and question-answer pair information; evaluates the data quality of the thought chain information, question information, and question-answer pair information based on preset multi-dimensional evaluation indicators, and outputs corresponding quality scores and labels; and filters the quality scores and labels corresponding to the thought chain information, question information, and question-answer pair information according to preset filtering conditions to determine high-quality data, wherein the high-quality data is used for model training. This disclosure solves the problem of low model training accuracy caused by the lack of a quantitative quality grading system for labeled thought chain data by constructing a multi-dimensional data quality evaluation for thought chain data.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0020] Figure 1 This diagram illustrates a data processing system structure according to an embodiment of the present disclosure. Figure 2 This diagram illustrates a data processing method according to an embodiment of the present disclosure. Figure 3 A flowchart illustrating a specific example of a data processing method according to an embodiment of this disclosure is shown. Figure 4 A flowchart illustrating another specific example of a data processing method according to an embodiment of the present disclosure; Figure 5 A flowchart illustrating yet another specific example of a data processing method according to an embodiment of the present disclosure; Figure 6 This diagram illustrates a data processing apparatus according to an embodiment of the present disclosure. Figure 7 A structural block diagram of a computer device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0021] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0022] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0023] The specific implementation methods of the embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0024] Figure 1 A schematic diagram of an exemplary application system architecture to which the data processing methods of the embodiments of this disclosure can be applied is shown. For example... Figure 1 As shown, the system architecture may include terminal device 101, network 102 and server 103.

[0025] Network 102 is a medium used to provide a communication link between terminal device 101 and server 103, and can be a wired network or a wireless network.

[0026] Optionally, the aforementioned wireless or wired networks use standard communication technologies and / or protocols. The network is typically the Internet, but can also be any network, including but not limited to Local Area Networks (LANs), Metropolitan Area Networks (MANs), Wide Area Networks (WANs), mobile, wired or wireless networks, private networks, or any combination of virtual private networks. In some embodiments, technologies and / or formats, including Hyper Text Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network. Furthermore, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Networks (VPNs), and Internet Protocol Security (IPSec) can be used to encrypt all or some links. In other embodiments, custom and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0027] Terminal device 101 can be various electronic devices, including but not limited to smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, wearable devices, augmented reality devices, virtual reality devices, etc.

[0028] Optionally, the client of the application installed on different terminal devices 101 may be the same, or the client of the same type of application based on different operating systems. Depending on the terminal platform, the specific form of the application client may also be different; for example, the application client may be a mobile client, a PC client, etc.

[0029] In a specific example, the terminal preprocesses the raw thought chain data collected by the server to determine ordered thought chain information, question information, and question-answer pair information; it evaluates the data quality of thought chain information, question information, and question-answer pair information based on preset multi-dimensional evaluation indicators, and outputs the corresponding quality scores and labels; it filters the quality scores and labels corresponding to thought chain information, question information, and question-answer pair information according to preset filtering conditions to determine high-quality data, which is used for model training.

[0030] Server 103 can be a server that provides various services, such as a backend management server that supports the device operated by the user using terminal device 101. The backend management server can analyze and process received requests and other data, and feed the processing results back to the terminal device.

[0031] Optionally, the server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0032] In a specific example, the server preprocesses the raw thought chain data collected by the terminal to determine ordered thought chain information, question information, and question-answer pair information; it evaluates the data quality of thought chain information, question information, and question-answer pair information based on preset multi-dimensional evaluation indicators, and outputs the corresponding quality scores and labels; it filters the quality scores and labels corresponding to thought chain information, question information, and question-answer pair information according to preset filtering conditions to determine high-quality data, which is used for model training.

[0033] Those skilled in the art will know that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative; any number of terminal devices, networks, and servers can be included depending on actual needs. This disclosure does not limit the scope of the embodiments.

[0034] Figure 2 A flowchart of a data processing method according to an embodiment of this disclosure is shown, such as Figure 2 As shown, the data processing method provided in this embodiment includes the following steps: S202, preprocess the original thought chain data to determine the ordered thought chain information, question information and question-answer pair information.

[0035] It should be noted that the aforementioned raw thought chain data can be multi-sample data containing three main features: question, thought chain, and answer. For example, raw thought chain data samples containing questions, reasoning processes, and answers can be collected through publicly available online datasets, literature corpora, and existing model outputs to provide a raw corpus foundation for subsequent processing. The aforementioned preprocessing can involve performing preprocessing operations such as format standardization, encoding conversion, invalid character removal, and punctuation normalization on the collected raw thought chain data to ensure a standardized data structure and clean content. The aforementioned ordered thought chain information can be obtained by using a connective dictionary and syntactic rules to parse the reasoning process after preprocessing the raw thought chain data, breaking down the continuous reasoning text into an ordered sequence of reasoning steps, facilitating quality assessment and data filtering in subsequent reasoning sections. The aforementioned question information can be the original input describing the task to be solved, such as question text. The aforementioned question-answer pair information can be a combination of a question and a final answer, such as question text and a final answer.

[0036] S204 evaluates the data quality of thought chain information, question information, and question-and-answer pair information based on preset multi-dimensional evaluation indicators, and outputs the corresponding quality scores and tags.

[0037] It should be noted that the aforementioned evaluation indicators can be detection rules, which can avoid subjective evaluation and provide executable detection rules for automated evaluation. The aforementioned data quality can be quantifiable values, used to quantify abstract concepts and generate indicators that can support the setting of screening thresholds. The aforementioned tags can be used to evaluate the breadth of data distribution, such as content diversity tags.

[0038] S206. Based on preset filtering conditions, the quality scores and labels corresponding to the thought chain information, question information, and question-and-answer pair information are filtered to determine high-quality data, which is used for model training.

[0039] It should be noted that the high-quality data mentioned above can be thought chain data that meets the requirements of both data quality and distribution balance after multi-dimensional quantitative evaluation and dynamic screening.

[0040] This disclosure addresses the problem of low model training accuracy caused by the lack of a quantitative quality grading system for labeled thought chain data by constructing a multi-dimensional data quality assessment for thought chain data.

[0041] In one embodiment of this disclosure, such as Figure 3 As shown, the target detection method provided in this embodiment can preprocess the original thought chain data through the following steps to extract and structure core information: S302 performs format unification, encoding conversion, invalid character removal, and punctuation standardization on the original thought chain data, and outputs preprocessed data. S304, based on the rule base and the connection dictionary, the preprocessed data is parsed into sentences to determine the thought chain information and the question information; S306, extract the question and answer fields from the preprocessed data to determine the question-answer pair information.

[0042] It should be noted that the above-mentioned thought chain information is ordered thought chain information, including the sequence of reasoning steps, and retains the logical order; the above-mentioned question information clarifies the target problem to be solved by separating the original question description; the above-mentioned question-answer pair information includes related questions and final answers, constructing a complete input-output mapping.

[0043] In one example of this disclosure, the above-mentioned evaluation of the data quality of thought chain information, question information, and question-answer pair information based on preset multi-dimensional evaluation indicators, and the output of corresponding quality scores and tags include: evaluating the data quality of thought chain information, question information, and question-answer pair information through preset ten-dimensional evaluation indicators, generating a tag for the content diversity dimension and quality scores for the other nine dimensions, wherein the ten-dimensional evaluation indicators include: reasoning atomicity, reasoning logic, reasoning informativeness, reasoning completeness, reasoning complexity, answer correctness, language purity, text compliance, text repetition, and content diversity.

[0044] This disclosure utilizes ten dimensions of quality assessment indicators, including reasoning atomicity, logic, informativeness, completeness, complexity, answer correctness, language purity, text compliance, repeatability, and content diversity, covering the entire process of reasoning chains and question-answer pairs from structure and semantics to compliance. It overcomes the shortcomings of traditional assessment methods that only focus on answer accuracy or language quality; and effectively supports the hierarchical management and optimal use of training data for large language models.

[0045] In one example of this disclosure, the quality score for the atomicity of reasoning is determined based on the smallest logical unit; the quality score for the logicality of reasoning is determined based on external and internal logic; the quality score for the informativeness of reasoning is determined based on new and invalid information; the quality score for the completeness of reasoning is determined based on completeness; the quality score for the complexity of reasoning is determined based on the degree of difficulty; the quality score for the correctness of the answer is determined based on correctness and linguistic purity; the quality score for text compliance is determined based on sensitive information; and the quality score for text redundancy includes determination based on redundant information.

[0046] For example, reasoning atomicity assesses whether reasoning is broken down into the smallest logical units, avoiding excessive merging of steps and improving the detail of reasoning. Reasoning logicality includes external logic (derivative relationships between steps) and internal logic (internal reasoning relationships within a single step), evaluating the logicality of each step in the reasoning. Reasoning informativeness assesses whether the reasoning steps introduce new information and whether there is redundancy or illusory content. Reasoning completeness assesses whether the reasoning process is complete and whether any steps are missing. Reasoning complexity assesses whether the reasoning is challenging and whether it poses a difficulty to the underlying model. Answer correctness assesses whether the reasoning answer is consistent with the true answer (closed-ended or open-ended judgment). Language purity assesses whether the language is a single language, excluding cases of multilingual mixing. Text compliance assesses whether it contains sensitive or other non-compliant information. Text redundancy assesses whether there is semantic or textual structural repetition. Content diversity assesses whether the types of tasks or instructions in the data are rich and diverse.

[0047] This disclosure combines rule matching with a large-scale model evaluation mechanism to propose quantitative evaluation methods for various dimensions of indicators. For example, atomicity is determined step-by-step by connecting words and using a large-scale model to identify the smallest reasoning unit, while logicality is evaluated by assessing the logical consistency of the reasoning through the derivation chain of steps. This enables quality assessment without human intervention; improves the consistency, efficiency, and scalability of data evaluation; and supports automated screening and quality optimization of large-scale data.

[0048] In a specific example, the quality score for the atomicity of the aforementioned reasoning includes: determining whether each clause of the thought chain information, question information, and question-answer pair information constitutes a minimum logical reasoning unit based on a large language model; calculating the proportion of clauses judged as minimum logical reasoning units; and using this as the quality score for the atomicity of reasoning. For example, for thought chain information, the thought chain text (information) is initially segmented into clauses based on predefined conjunctions (such as "firstly," "next," "therefore," etc.); the large language model is used to determine whether each clause constitutes a minimum logical reasoning unit; and the proportion of clauses judged as "atomic-level reasoning" is calculated as the atomicity score of reasoning.

[0049] In a specific example, the quality score for the aforementioned reasoning logic includes: based on the logical judgment of the thought chain information, question information, and question-answer pair information by the large language model, outputting the average of the internal and external logical scores as the quality score for reasoning logic. For example, for thought chain information, after breaking down the thought chain text (information) into steps, an external logical judgment is made for each step: whether it is derived from the previous step and the question; an internal logical judgment is made for each step: whether there is a clear reasoning connection between the preceding and following clauses; the large model is called to score step by step, and finally the average of the internal and external logical scores is output.

[0050] In a specific example, the quality score for the inference informationality mentioned above includes: judging whether each inference statement of the thought chain information, question information, and question-answer pair information provides new useful information, whether there is illusory information, and whether there is redundancy or repetition based on the large language model; and combining the correct information ratio of each step as the quality score for inference informationality. For example, the large model is called step by step to judge whether each inference statement provides new useful information; at the same time, it is judged whether there is illusory information and redundancy or repetition; and the overall correct information ratio of each step is used as the informationality score.

[0051] In a specific instance, the quality score for the completeness of the above reasoning includes: using a large model to combine the question and the final answer to determine whether there are any missing steps in the reasoning, and outputting a completeness index using a scoring method.

[0052] In a specific example, the quality score for the aforementioned reasoning complexity includes: performing reasoning on the original problem using multiple large models with relatively few parameters, and verifying the correctness of the reasoned answers. If most of the large models (a preset proportion) can provide the correct answer, the problem is considered to have low complexity; otherwise, it is considered to have high complexity. Reasoning complexity can be used to filter out data that poses a greater challenge to the models.

[0053] In a specific instance, the quality score for the correctness of the above answer includes: the algorithm categorizes the evaluation methods based on the type of question. For closed-ended questions (such as multiple choice or arithmetic questions), it determines whether the answer is consistent with the standard answer through rule templates or keyword matching; for open-ended questions, it calls the answer scoring model to score semantic consistency.

[0054] In a specific instance, the quality score for language purity mentioned above includes: using multilingual recognition tools (e.g., lightweight language detection libraries, text processing engines, etc.) to determine the language type of each sentence in the text, statistically analyzing the language type data in the text, and determining whether there is a situation of mixed use of multiple languages.

[0055] In a specific instance, the quality score for the aforementioned text compliance includes: performing multi-dimensional compliance detection on the text by combining a detection model (e.g., a multimodal model that can process both text and images), a bias identification model, and a sentiment classification model; determining whether the text contains sensitive content; and finally determining the compliance score by weighting the results of multiple sub-models.

[0056] In a specific instance, the quality score for text repetition mentioned above includes: generating sentence vectors using a pre-trained text embedding model (e.g., a sentence embedding model); analyzing semantic similarity within sentences or between samples; and marking excessively high repetition as low quality.

[0057] In a specific example, the above content diversity assessment includes: determining whether the types of tasks or instructions in the data are rich and diverse. For example, using a task classification model, the instruction text tasks or questions of each data in the dataset are divided into several preset types (such as reasoning, question answering, summarizing, etc.) to obtain the corresponding data; calculating the category coverage of the samples, and calculating the entropy value of the category distribution of the overall samples, and weighting the sum as the diversity score; it can be used to analyze the task coverage in the dataset. For example, the above content diversity score can be calculated by the following formula (1): - (1) Where M represents the content diversity score, The number of preset types, For the number of types covered by the dataset, The proportion of the i-th type of data in the dataset. This indicates the coverage of the dataset. This indicates the balance of the distribution of different classes in the dataset. and The weights are for the two parts, where, .

[0058] In a specific instance, when M is greater than the target threshold, it indicates that the type of task or instruction in the data (e.g., a certain dimension mentioned above) is a high-frequency task (marked as a high-frequency label), and an undersampling strategy can be used to reduce the high-frequency task data; when M is less than or equal to the target threshold, it indicates that the type of task or instruction in the data is a low-frequency task (marked as a low-frequency label), and a resampling strategy is used to increase the low-frequency task data.

[0059] This disclosure uses an initial screening based on a threshold set by the evaluation score, and then combines entropy evaluation and category resampling to balance the distribution of various tasks (data in various dimensions). This can avoid training bias caused by high repetition or imbalance in task categories; and supports the construction of high-performance training corpora that balance diversity and quality.

[0060] In one embodiment of this disclosure, such as Figure 4 As shown, the target detection method provided in this embodiment of the present disclosure filters the quality scores and tags corresponding to thought chain information, question information, and question-answer pair information according to preset filtering conditions to determine high-quality data, including the following steps: S402, Based on preset thresholds, the quality scores of nine dimensions are filtered to determine the datasets that meet the preset thresholds; S404, based on the imbalance processing strategy, achieves category balance for datasets that meet the preset threshold under the label of content diversity dimension, and determines high-quality data.

[0061] In one embodiment of this disclosure, such as Figure 5 As shown, the target detection method provided in this embodiment of the disclosure, according to the imbalance processing strategy, achieves category balance for datasets that meet a preset threshold under the label of content diversity dimension, and determines high-quality data, including the following steps: S502, based on the dataset that meets the preset threshold and the labels of the content diversity dimension, determine the distribution of the dataset that meets the preset threshold under the labels of the content diversity dimension, wherein the distribution of the dataset under the labels of the content diversity dimension includes high-frequency labels and low-frequency labels. S504, according to the imbalance processing strategy, the data of the high-frequency tags is undersampled and the data of the low-frequency tags is resampled to determine the high-quality data.

[0062] For example, in the dataset quality assessment stage, the quality of the thought chain, questions, and answers in the data is assessed by preset indicators. The corresponding scoring results or labels are output for subsequent quality calculation. A content diversity label and the scores of the other nine quality items are generated for each data point, with a score range of 0 to 1. Finally, a 9-dimensional feature vector and a content diversity classification label are formed. The nine standards and content diversity are aggregated and calculated to obtain the average score of the dataset on each indicator. In particular, the content diversity score is calculated by formula (1). Dataset screening stage: First, the scoring results of each data point on the ten quality assessment dimensions are received, including the content diversity label. Then, according to the set threshold, the data that does not meet the standards of the other nine indicators besides content diversity are removed to ensure the quality baseline. The task type analysis is performed on the data after the initial screening to identify the distribution of each task under the "content diversity" dimension. In response to the imbalance of task categories, an undersampling strategy is used to reduce high-frequency task data, or a resampling strategy is used to increase low-frequency task data to achieve category balance. Finally, a dataset with a balanced distribution of task types in content diversity and overall quality is output. This result set can be directly used for model training, improving the model's generalization ability and stability in various task processing.

[0063] This disclosure overcomes the limitations of existing large-scale model training data evaluation methods, providing a data quality assessment and screening mechanism that is clearly structured, comprehensive in indicators, highly efficient in algorithms, and can be automated. It has significant value in improving the reasoning capabilities of large language models and ensuring the reliability and compliance of model generation. It possesses broad engineering adaptability and industrial application prospects, and can be applied to scenarios such as large-scale model training, evaluation, and data engineering. It addresses the lack of a unified data quality assessment standard in current thinking chain data quality assessment systems, and solves the problems of reliance on manual assessment, low efficiency, and strong subjectivity through an automated data quality assessment system. It establishes a systematic assessment framework, achieves automated scoring and ranking, significantly improves data screening efficiency and consistency, reduces manual intervention, supports customized screening and combination as needed, ensures applicability across multiple domains, and continuously optimizes data quality.

[0064] Based on the same inventive concept, this disclosure also provides a data processing apparatus, as described in the following embodiments. Since the principle by which this apparatus solves the problem is similar to that of the method embodiments described above, the implementation of this apparatus embodiment can refer to the implementation of the method embodiments described above, and repeated details will not be repeated.

[0065] Figure 6 This diagram illustrates a data processing apparatus according to an embodiment of the present disclosure, such as... Figure 6 As shown, the device includes: a preprocessing module 61, an evaluation module 62, and a high-quality data output module 63.

[0066] The preprocessing module 61 is used to preprocess the original thought chain data to determine the ordered thought chain information, question information, and question-answer pair information.

[0067] The evaluation module 62 is used to evaluate the data quality of thought chain information, question information and question-answer pair information based on preset multi-dimensional evaluation indicators, and output the corresponding scores and tags.

[0068] The high-quality data output module 63 is used to filter the scores and tags corresponding to the thought chain information, question information and question-and-answer information according to preset filtering conditions to determine high-quality data, which is used for model training.

[0069] In one example of this disclosure, the preprocessing module is further used to: standardize the format of the original thought chain data, convert the encoding, remove invalid characters, and standardize punctuation, and output preprocessed data; parse the preprocessed data into sentences according to the rule base and the connection dictionary to determine the thought chain information and question information; and extract the question field and answer field from the preprocessed data to determine the question-answer pair information.

[0070] In one example of this disclosure, the aforementioned evaluation module is also used to: evaluate the data quality of thought chain information, question information, and question-and-answer pair information using ten preset evaluation indicators, and generate a label for the content diversity dimension and quality scores for the remaining nine dimensions. The ten evaluation indicators include: reasoning atomicity, reasoning logic, reasoning informativeness, reasoning completeness, reasoning complexity, answer correctness, language purity, text compliance, text repetition, and content diversity.

[0071] In one example of this disclosure, the quality score for the atomicity of reasoning in the above evaluation module is determined based on the smallest logical unit; the quality score for the logicality of reasoning is determined based on external and internal logic; the quality score for the informativeness of reasoning is determined based on new and invalid information; the quality score for the completeness of reasoning is determined based on completeness; the quality score for the complexity of reasoning is determined based on the degree of difficulty; the quality score for the correctness of the answer is determined based on correctness and linguistic purity; the quality score for text compliance is determined based on sensitive information; and the quality score for text redundancy includes determination based on redundant information.

[0072] In one example of this disclosure, the high-quality data output module is also used to: filter the quality scores of nine dimensions according to a preset threshold to determine the dataset that meets the preset threshold; and, according to an imbalanced processing strategy, balance the categories of the dataset that meets the preset threshold under the label of the content diversity dimension to determine high-quality data.

[0073] In one example of this disclosure, the high-quality data output module is further configured to: determine the distribution of the dataset meeting the preset threshold under the labels of the content diversity dimension based on the dataset meeting the preset threshold and the labels of the content diversity dimension, wherein the distribution under the labels of the content diversity dimension includes high-frequency labels and low-frequency labels; and determine high-quality data by undersampling the data of high-frequency labels and resampling the data of low-frequency labels according to the imbalance processing strategy.

[0074] It should be noted that the preprocessing module 61, evaluation module 62, and high-quality data output module 63 mentioned above correspond to S202 to S206 in the method embodiments. The examples and application scenarios implemented by these modules and their corresponding steps are the same, but they are not limited to the content disclosed in the above method embodiments. It should be noted that the above modules, as part of the apparatus, can be executed in a computer system such as a set of computer-executable instructions.

[0075] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0076] The following reference Figure 7 To describe an electronic device 700 according to such an embodiment of the present disclosure. Figure 7 The electronic device 700 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0077] like Figure 7 As shown, the electronic device 700 is manifested in the form of a general-purpose computing device. The components of the electronic device 700 may include, but are not limited to: at least one processing unit 710, at least one storage unit 720, and a bus 730 connecting different system components (including storage unit 720 and processing unit 710).

[0078] The storage unit stores program code that can be executed by the processing unit 710, causing the processing unit 710 to perform the steps described in the "Exemplary Methods" section above according to various exemplary embodiments of this disclosure.

[0079] For example, the processing unit 710 can execute the following steps of the above method embodiment: a data processing method, which preprocesses the original thought chain data to determine ordered thought chain information, question information, and question-answer pair information; evaluates the data quality of the thought chain information, question information, and question-answer pair information based on preset multi-dimensional evaluation indicators, and outputs the corresponding quality scores and labels; filters the quality scores and labels corresponding to the thought chain information, question information, and question-answer pair information according to preset filtering conditions to determine high-quality data, wherein the high-quality data is used for model training.

[0080] For example, the processing unit 710 can perform the following steps in the above method embodiment: unify the format, convert the encoding, remove invalid characters, and standardize the punctuation of the original thought chain data, and output preprocessed data; parse the preprocessed data into sentences according to the rule base and the connection dictionary to determine the thought chain information and question information; extract the question field and answer field from the preprocessed data to determine the question-answer pair information.

[0081] For example, the processing unit 710 can perform the following steps in the above method embodiment: evaluate the data quality of the thought chain information, question information, and question-answer pair information through ten preset evaluation indicators, and generate a label for the content diversity dimension and quality scores for the other nine dimensions. The ten evaluation indicators include: reasoning atomicity, reasoning logic, reasoning informativeness, reasoning completeness, reasoning complexity, answer correctness, language purity, text compliance, text repetition, and content diversity.

[0082] For example, the processing unit 710 can execute the following steps of the above method embodiment: the quality score of reasoning atomicity is determined based on the smallest logical unit; the quality score of reasoning logicality is determined based on external logic and internal logic; the quality score of reasoning informativeness is determined based on new information and invalid information; the quality score of reasoning completeness is determined based on completeness; the quality score of reasoning complexity is determined based on the degree of difficulty; the quality score of answer correctness is determined based on correctness and linguistic purity; the quality score of text compliance is determined based on sensitive information; and the quality score of text repetition includes determination based on redundant information.

[0083] For example, the processing unit 710 can perform the following steps in the above method embodiment: filter the quality scores of nine dimensions according to a preset threshold, and determine the dataset that meets the preset threshold; Based on the imbalance processing strategy, datasets that meet the preset threshold are classified into categories under the label of content diversity dimension to determine high-quality data.

[0084] For example, the processing unit 710 can perform the following steps in the above method embodiment: based on the dataset that meets the preset threshold and the labels of the content diversity dimension, determine the distribution of the dataset that meets the preset threshold under the labels of the content diversity dimension, wherein the distribution of the dataset under the labels of the content diversity dimension includes high-frequency labels and low-frequency labels; according to the imbalanced processing strategy, undersample the data of high-frequency labels and resample the data of low-frequency labels to determine high-quality data.

[0085] Storage unit 720 may include readable media in the form of volatile storage units, such as random access memory (RAM) 7201 and / or cache 7202, and may further include read-only memory (ROM) 7203.

[0086] The storage unit 720 may also include a program / utility 7204 having a set (at least one) program module 7205, such program module 7205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0087] Bus 730 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0088] Electronic device 700 can also communicate with one or more external devices 740 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 700, and / or with any device that enables electronic device 700 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 750. Furthermore, electronic device 700 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 760. As shown, network adapter 760 communicates with other modules of electronic device 700 via bus 730. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0089] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0090] In particular, according to embodiments of this disclosure, the process described above with reference to the flowchart can be implemented as a computer program product, which includes a computer program that, when executed by a processor, implements the above-described data processing method.

[0091] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. A program product capable of implementing the methods described above is stored thereon. In some possible implementations, various aspects of this disclosure may also be implemented as a program product including program code, which, when run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0092] More specific examples of computer-readable storage media in this disclosure may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0093] In this disclosure, a computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device.

[0094] Optionally, the program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0095] In practical implementation, program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0096] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0097] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0098] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0099] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A data processing method, characterized in that, include: Preprocess the raw thought chain data to determine the ordered thought chain information, question information, and question-answer pair information; The data quality of the thought chain information, the question information, and the question-and-answer pair information is evaluated based on preset multi-dimensional evaluation indicators, and the corresponding quality scores and tags are output. The quality scores and tags corresponding to the thought chain information, the question information, and the question-and-answer pair information are filtered according to preset filtering conditions to determine high-quality data, wherein the high-quality data is used for model training.

2. The data processing method according to claim 1, characterized in that, The preprocessing of the original thought chain data to determine the thought chain information, question information, and question-answer pair information includes: The original thought chain data is formatted, encoded, invalid characters are removed, and punctuation is standardized, and preprocessed data is output. The preprocessed data is parsed into sentences based on the rule base and the connection lexicon to determine the thought chain information and question information; The question and answer fields are extracted from the preprocessed data to determine the question-answer pair information.

3. The data processing method according to claim 1, characterized in that, The data quality of the thought chain information, the question information, and the question-and-answer pair information is evaluated based on preset multi-dimensional evaluation indicators, and the corresponding quality scores and tags are output, including: The data quality of the thought chain information, the question information, and the question-and-answer pair information is evaluated using ten preset evaluation indicators, generating a label for the content diversity dimension and quality scores for the other nine dimensions. The ten evaluation indicators include: reasoning atomicity, reasoning logic, reasoning informativeness, reasoning completeness, reasoning complexity, answer correctness, language purity, text compliance, text repetition, and content diversity.

4. The data processing method according to claim 3, characterized in that, The quality score of the atomicity of the inference is determined based on the smallest logical unit; The quality score for the logical reasoning is determined based on external and internal logic. The quality score of the reasoning information is determined based on new and invalid information; The quality score for the completeness of the reasoning is determined based on the completeness. The quality score for the reasoning complexity is determined based on the degree of difficulty. The quality score for the correctness of the answer is determined based on both correctness and linguistic purity. The quality score for text compliance is determined based on sensitive information; The quality score for text redundancy is determined based on redundant information.

5. The data processing method according to any one of claims 3 to 4, characterized in that, The step of filtering the thought chain information, the question information, and the question-and-answer pair information according to preset filtering conditions to determine high-quality data includes: Based on preset thresholds, the quality scores of nine dimensions are filtered to determine the datasets that meet the preset thresholds; According to the imbalance processing strategy, the datasets that meet the preset threshold are classified and balanced under the label of content diversity dimension to determine the high-quality data.

6. The data processing method according to claim 5, characterized in that, The step of implementing an imbalanced processing strategy involves balancing the categories of datasets that meet a preset threshold under the label of content diversity dimension, thereby determining the high-quality data, which includes: Based on the dataset that meets the preset threshold and the labels of the content diversity dimension, determine the distribution of the dataset that meets the preset threshold under the labels of the content diversity dimension, wherein the distribution of the content diversity dimension under the labels includes high-frequency labels and low-frequency labels. According to the imbalance processing strategy, the data of the high-frequency labels are undersampled and the data of the low-frequency labels are resampled to determine the high-quality data.

7. A data processing apparatus, characterized in that, include: The preprocessing module is used to preprocess the raw thought chain data to determine the ordered thought chain information, question information, and question-answer pair information; The evaluation module is used to evaluate the data quality of the thought chain information, the question information, and the question-and-answer pair information based on preset multi-dimensional evaluation indicators, and output the corresponding scores and tags. A high-quality data output module is used to filter the scores and tags corresponding to the thought chain information, the question information, and the question-and-answer pair information according to preset filtering conditions to determine high-quality data, wherein the high-quality data is used for model training.

8. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the data processing method of any one of claims 1 to 6 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the data processing method according to any one of claims 1 to 6.

10. A computer program product, comprising: A computer program or instruction, characterized in that, when executed by a processor, the computer program or instruction implements the data processing method according to any one of claims 1 to 6.

Images