Data processing method, model training method, electronic equipment and computer medium
By cleaning, labeling, filtering, and augmenting the dataset, and evaluating it using a pre-trained model, the problems of insufficient data quality and diversity were solved, and the training effect and applicability of large models were improved.
Patent Information
- Application Number
- CN202411503636.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-25
- Publication Date
- 2024-11-26
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing model training data is of inconsistent quality, has high costs for manual intervention, and lacks diversity, resulting in poor training performance and application effects of large models.
By acquiring the dataset to be processed, performing data cleaning, labeling, and filtering, data augmentation, and using pre-trained models to evaluate the results, the output dataset that meets the preset indicators is reduced, thus reducing manual intervention and costs.
It improves the quality and diversity of datasets, enhances the training effect and application applicability of large models, and reduces data processing costs.
Smart Images

Figure CN119025515A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, a model training method, an electronic device and a computer medium. Background Art
[0002] With the development of neural network technology, the new generation of artificial intelligence has gradually developed into learning and data analysis through machine learning models with billions or even trillions of parameters. Since large models usually require large amounts of data for pre-training, they have the characteristics of large scale, strong versatility, and good emergence.
[0003] As technology continues to update, model training requires more data. Existing training data generally has problems such as uneven quality, high cost of manual intervention, and insufficient diversity, which reduces the training performance and application effect of large models. Summary of the invention
[0004] To solve the above technical problems, the present application provides a data processing method, a model training method, an electronic device and a computer medium.
[0005] To solve the above problems, the present application provides a first technical solution: providing a data processing method, comprising: obtaining at least one first data set to be processed, and performing data cleaning on the above first data set; annotating and filtering the cleaned first data set; performing data enhancement on the filtered first data set; inputting the enhanced first data set into a pre-trained model to obtain a first evaluation result of the pre-trained model for the above first data set; and outputting the above first data set when the above first evaluation result meets the preset indicators.
[0006] Optionally, the above-mentioned labeling and filtering of the cleaned first data set includes: obtaining the first data set to be processed, performing data cleaning on the first data set, and labeling the cleaned first data set; performing quality scoring on the labeled first data set to obtain scoring information of the first data set; and filtering the labeled first data set based on the scoring information of the first data set.
[0007] Optionally, the above-mentioned first data set after annotation is filtered based on the multiple scoring information of the above-mentioned first data set, including: when the above-mentioned scoring information indicates that the data of the above-mentioned first data set does not meet the first preset condition, the corresponding data is filtered out from the above-mentioned first data set and added to the second data set; data enhancement is performed on the data in the above-mentioned second data set; quality scoring is performed on the enhanced above-mentioned second data set; when the scoring information of the above-mentioned second data set meets the above-mentioned first preset condition, the corresponding data is added to the above-mentioned first data set to obtain the filtered above-mentioned first data set.
[0008] Optionally, the data enhancement of the filtered first data set includes: inputting the filtered first data set into the generator of the adversarial network to obtain enhanced samples output by the generator; inputting the enhanced samples into the discriminator of the adversarial network to obtain a second evaluation result after the discriminator evaluates the enhanced samples; when the second evaluation result meets the second preset condition, outputting the enhanced samples as the enhanced first data set.
[0009] Optionally, the data enhancement of the filtered first data set includes: performing data enhancement on the filtered first data set through a variational autoencoder to obtain an enhanced first data set, or applying one or more data enhancement operations to each data sample of the filtered first data set, and mixing the enhanced data samples according to a preset ratio to obtain an enhanced first data set.
[0010] Optionally, the data cleaning of the first data set includes: removing preset characters in the first data set, and / or performing grammatical correction on data of the first data set.
[0011] Optionally, the above-mentioned acquisition of at least one first data set to be processed includes: collecting the above-mentioned first data sets of preset types from multiple data sources; after the above-mentioned step of inputting the enhanced above-mentioned first data set into the pre-trained model to obtain the first evaluation result of the above-mentioned pre-trained model, the above-mentioned data processing method also includes: based on the above-mentioned first evaluation result, obtaining update information of the above-mentioned data source; when the above-mentioned update information indicates that the above-mentioned data source includes multiple versions of data, obtaining a third data set after the above-mentioned data source is updated; and inputting the above-mentioned third data set into the above-mentioned pre-trained model to obtain a third evaluation result of the above-mentioned pre-trained model for the above-mentioned third data set.
[0012] To solve the above problems, the present application provides a second technical solution: providing a model training method, comprising: obtaining a first data set through the above data processing method; inputting the above first data set into the application model to be trained to obtain the prediction result output by the above application model based on the above first data set; and iteratively updating the above application model based on the above prediction result.
[0013] To solve the above problems, the present application provides a third technical solution: an electronic device is provided, comprising a processor and a memory connected to the processor, wherein the memory stores program instructions; the processor is used to execute the program instructions stored in the memory to implement the above method.
[0014] To solve the above problem, the present application provides a fourth technical solution: providing a computer-readable storage medium, the computer-readable storage medium storing program instructions, and the program instructions can be executed by a processor to implement the above method.
[0015] Different from the prior art, the data processing method of the present application obtains at least one first data set to be processed, and performs data cleaning on the first data set; annotates and filters the cleaned first data set; performs data enhancement on the filtered first data set; inputs the enhanced first data set into the pre-training model to obtain a first evaluation result of the pre-training model for the first data set; and outputs the first data set when the first evaluation result meets the preset indicators. In the above manner, the data processing method can perform multi-level data processing and optimization on the first data set to improve the quality and diversity of the first data set, reduce manual intervention in the data processing process, reduce data processing costs, and the output first data set can improve the training effect of the large model when applied to the training of the large model, thereby improving the application effect and applicability of the large model. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them: Figure 1 It is a flowchart of the first embodiment of the data processing method provided by the present application; Figure 2 is a flow chart of a second embodiment of the data processing method provided by the present application; Figure 3 is a flowchart of a third embodiment of the data processing method provided by the present application; Figure 4is a flowchart of a fourth embodiment of the data processing method provided by the present application; Figure 5 It is a flowchart of an embodiment of the model training method provided by the present application; Figure 6 is a structural schematic diagram of an embodiment of an electronic device provided by the present application; Figure 7 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION
[0017] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0018] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back...), such directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.
[0019] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.
[0020] See also Figure 1 , Figure 1 1 is a flow chart of the first embodiment of the data processing method provided by the present application. Figure 1 As shown, the data processing method of this embodiment includes: Step S11: Acquire at least one first data set to be processed, and perform data cleaning on the first data set.
[0021] Specifically, at least one first data set to be processed can be collected through technologies such as web crawlers and application programming interfaces (APIs), or can be obtained by calling existing database data, and the data source of the first data source is not limited here. After obtaining the first data set to be processed, the first data set is cleaned to reduce noise in the first data set and improve the semantic accuracy and consistency of the data in the first data set.
[0022] Among them, the above-mentioned first data set can be relevant application data applied to a certain field or a certain scenario. Exemplarily, in some embodiments, the data of the first data set can be applied to the training of a large language model. For example, the large language model can be trained and used to process the following tasks: machine translation, intelligent question and answer, text generation, information extraction, sentiment analysis, personalized recommendation, etc.; the large language model can also be used for text processing in professional fields, such as legal document analysis, medical text processing, industry standards and specifications, etc., to meet scenarios with extremely high data quality requirements. In other possible implementations, the data of the first data set can be applied to the training of speech recognition and synthesis models, and the data types of the first data set include but are not limited to speech data, etc., or the data of the first data set can be applied to computer vision models, and the data types of the first data set include but are not limited to image data, video data, etc.
[0023] Step S12: labeling and filtering the cleaned first data set.
[0024] Specifically, after the first data set is cleaned, the cleaned first data set is labeled, and the data labeling is used to provide key information for understanding and parsing the first data set. Among them, the method of labeling the first data set includes but is not limited to using machine learning, deep learning and other technologies to perform preliminary labeling on the data to obtain preliminary labeling results and implement labeling in combination with manual review to improve the accuracy of data labeling; or, a data labeling platform can be constructed through machine learning, deep learning, multi-task learning models and other technologies to achieve automatic labeling and improve the efficiency of data labeling. By labeling the cleaned first data set, the first data set is provided with a label indicating which type of object the data belongs to, which facilitates the understanding and parsing of data attributes in the subsequent data processing process. After the first data set is labeled, the labeled first data set is filtered to filter out low-quality and irrelevant data samples in the first data set based on the labeling, thereby improving the data relevance of the first data set.
[0025] Step S13: performing data enhancement on the filtered first data set.
[0026] After filtering the first data set, data enhancement is performed on the filtered first data set to improve the diversity and quality of the first data set. When the first data set includes text data, the first data set can be enhanced by random insertion, random replacement, synonym replacement, data expansion, and sentence reconstruction; when the first data set includes image data, the first data set can be enhanced by cropping, flipping, and rotating. In other possible implementations, data enhancement can also be performed by machine learning, deep learning, and other technologies, which are not specifically limited here.
[0027] Step S14: input the enhanced first data set into the pre-training model to obtain a first evaluation result of the pre-training model for the first data set.
[0028] After data enhancement is performed on the first data set, the enhanced first data set is input into the pre-training model to obtain a first evaluation result output by the pre-training model for the first data set. The training task of the pre-training model corresponds to the application field and scenario of the first data set, so that the first evaluation result output by the pre-training model for the first data set can be used to indicate the relevance and quality of the first data set to the application field.
[0029] Step S15: When the first evaluation result meets the preset indicator, output the first data set.
[0030] When the first evaluation result meets the preset indicators, it means that the first data set is highly relevant to the application field and the quality of the first data set meets the preset requirements, and the first data set is output so that the first data set can be applied to the training of the application model, improve the training effect and performance of the application model, and improve the versatility of the application model. Among them, the preset indicators may be related to factors such as the architecture of the pre-trained model, the task type, and the requirements of data quality, and the preset indicators include but are not limited to variance, mean square error, error rate, precision rate, recall rate, etc.
[0031] In an embodiment of the present application, the data processing method obtains at least one first data set to be processed, cleans the first data set, and annotates and filters the cleaned first data set; performs data enhancement on the filtered first data set; inputs the enhanced first data set into a pre-training model to obtain a first evaluation result of the pre-training model for the first data set; and outputs the first data set when the first evaluation result meets a preset indicator. In the above manner, the data processing method can perform multi-level data processing and optimization on the first data set to improve the quality and diversity of the first data set, reduce manual intervention in the data processing process, and reduce data processing costs; and, because the first evaluation result of the pre-training model is used to evaluate whether the first data set meets the quality requirements, the first data set output after the data processing method of this embodiment can significantly improve the training effect of the large model when applied to the training of the large model, thereby improving the application effect and versatility of the large model.
[0032] In one embodiment, see Figure 2 , Figure 2 1 is a flow chart of the second embodiment of the data processing method provided by the present application. Figure 2 As shown, in the data processing method of this embodiment, step S12 includes: Step S21: performing a quality score on the annotated first data set to obtain score information of the first data set.
[0033] Specifically, the quality of the annotated first data set is scored according to the preset scoring rules to obtain the scoring information of the first data set. Among them, the preset scoring rules can score the integrity, repeatability, timeliness, effectiveness, accuracy and consistency of the first data set to obtain the scoring information. In a possible implementation, integrity is used to indicate whether there are missing null values in the first data set; repeatability is used to indicate whether there are duplicate data sets in the first data set; timeliness is used to indicate the timeliness of the data of the first data set, for example, whether the time of obtaining data in the first data set is quite different; effectiveness is used to indicate whether the data value, format, etc. of the first data set meet the requirements of the data definition or business definition, such as whether the format of some proprietary data such as telephone and mailbox meets the requirements; accuracy is used to indicate whether the data of the first data set can reflect the real business content, such as whether the data source is derived from the database in the real business scenario; consistency is used to indicate whether the type and meaning of the data in the first data set are consistent and clear, such as the consistency comparison of the data table before and after the data is migrated or processed. It can be understood that by scoring the first data set through the above multiple dimensions, weighted data of the scores of the first data set in different dimensions can be obtained to obtain the scoring information.
[0034] Optionally, before step S21, the data processing method may also include: constructing a filtering model using deep learning, reinforcement learning, machine learning and other technologies, and performing preliminary filtering on the labeled first data set through the filtering model to filter out data samples with obviously low quality and poor relevance in the first data set.
[0035] Step S22: filtering the annotated first data set based on the scoring information of the first data set.
[0036] After obtaining the score information of the first data set, the annotated first data set is filtered based on the score information of the first data set. The filtering method includes but is not limited to: sorting the data of the first data set based on the score information, and filtering out the data with higher scores as the filtered first data set by conditional filtering, such as retaining the data above the median or average after sorting, or retaining the data with a score greater than a certain threshold after sorting, or retaining some of the data at the top after sorting, etc.
[0037] In an embodiment of the present application, a quality score is performed on the annotated first data set to obtain scoring information of the first data set. Based on the scoring information of the first data set, the annotated first data set is filtered to remove low-quality and irrelevant data samples in the first data set, thereby improving the quality and diversity of the first data set, thereby improving the application effect and versatility of the large model.
[0038] Optionally, see Figure 3 , Figure 3 FIG. 1 is a flow chart of the third embodiment of the data processing method provided by the present application. Figure 3 As shown, in the data processing method of this embodiment, step S21 includes: Step S31: when the scoring information indicates that the data in the first data set does not satisfy the first preset condition, the corresponding data is filtered out from the first data set and added to the second data set.
[0039] Specifically, the scoring information of the first data set may indicate the scores of multiple data in the first data set. When the scoring information of the first data set indicates that the score of a certain data satisfies the first preset condition, the corresponding data is added to the second data set, so that the second data set is used as the filtered first data set. For example, the first data set includes the first data, the second data, and the third data, and the scoring information includes the first score corresponding to the first data, the second score corresponding to the second data, and the third score corresponding to the third data. When the first score and the second score satisfy the first preset condition and the third score does not satisfy the first preset condition, the third data is transferred from the first data set to the second data set, so that the third data is filtered out from the first data set and stored in the second data set.
[0040] The specific setting of the first preset condition is related to the quality assessment rules, the application scenario of the first data set, the accuracy requirements of the data, etc., and is not specifically limited here.
[0041] Step S32: performing data enhancement on the data in the second data set.
[0042] Specifically, since the data in the second data set do not meet the first preset condition, that is, the data in the second data set does not meet the data quality standard. Therefore, data enhancement is performed on the data in the second data set to improve the quality and diversity of the data in the second data set. The data enhancement method may be the same as or different from the above method.
[0043] Step S33: Performing a quality score on the enhanced second data set.
[0044] The enhanced second data set is scored for quality to obtain scoring information of the second data set. The quality scoring method of the second data set is similar to the above, and will not be described in detail here.
[0045] Step S34: when the scoring information of the second data set meets the first preset condition, the corresponding data is added to the first data set to obtain a filtered first data set.
[0046] When the scoring information of the second data set indicates that a certain data meets the first preset condition, the corresponding data is added to the first data set to obtain the filtered second data set. When the scoring information of the second data set indicates that a certain data does not meet the first preset condition, the corresponding data is discarded to ensure the quality of the filtered first data set.
[0047] In an embodiment of the present application, the data processing method filters out the corresponding data from the first data set and adds it to the second data set when the scoring information indicates that the data of the first data set does not meet the first preset condition, performs data enhancement on the data in the second data set, and performs quality scoring on the enhanced second data set. When the scoring information of the second data set meets the first preset condition, the corresponding data is added to the first data set to obtain a filtered first data set, remove low-quality and irrelevant data samples in the first data set, improve the quality and diversity of the first data set, and thereby improve the application effect and versatility of the large model.
[0048] Further, the scoring information of the first data set can be used to indicate the quality of the data of the first data set. Specifically, the scoring information includes a first scoring interval, a second scoring interval, and a third scoring interval, and the data quality indicated by the first scoring interval is higher than the data quality of the second scoring interval, and the data quality indicated by the second scoring interval is higher than the data quality of the third scoring interval. When the data of the first data set is in the first scoring interval, that is, the data meets the first preset condition, the data is retained in the first data set. When the data of the first data set is in the second scoring interval and the third scoring interval, that is, the data does not meet the first preset condition, the data is stored in the second data set; and when the data in the second data set is in the second scoring interval, conventional data enhancement operations are performed on the data, such as using synonym replacement, data expansion, and sentence reconstruction methods to reduce data computing power while achieving data enhancement; when the data in the second data set is in the third scoring interval, the data is enhanced by using technologies such as adversarial networks, variational autoencoders, or hybrid enhancements to improve the data enhancement effect, thereby improving the data quality of the second data set.
[0049] Through the above method, the data processing method of this embodiment can apply corresponding data enhancement methods when the quality scores are different, thereby improving the data enhancement effect while reducing the computing power of the data processing process and reducing the cost of data enhancement.
[0050] Optionally, in one embodiment, see Figure 4 , Figure 4 FIG. 4 is a flow chart of the fourth embodiment of the data processing method provided by the present application. Figure 4 As shown, in the data processing method of this embodiment, step S13 includes: Step S41: Input the filtered first data set into the generator of the adversarial network to obtain enhanced samples output by the generator.
[0051] Specifically, after filtering the first data set, the filtered first data set is input into the generator of the adversarial network to obtain enhanced samples output by the generator. The adversarial network is a pre-trained neural network for data enhancement, and the adversarial network is constructed by a generator and a discriminator. The generator is used to perform enhancement operations such as modification, padding, and adding noise to the data of the first data set to obtain enhanced samples; the discriminator is used to predict whether the enhanced samples belong to the original first data set to determine whether the enhanced samples belong to real data.
[0052] Step S42: input the enhanced sample to the discriminator of the adversarial network to obtain a second evaluation result after the discriminator evaluates the enhanced sample.
[0053] After obtaining the enhanced sample output by the generator, the enhanced sample is input to the discriminator, so that the discriminator outputs a second evaluation result for the enhanced sample. The second evaluation result can be an evaluation of the authenticity of the enhanced sample by scoring, or an evaluation by providing a judgment of whether it belongs to real data or not.
[0054] Step S43: When the second evaluation result meets the second preset condition, the enhanced sample is output as the enhanced first data set.
[0055] Specifically, after obtaining the second evaluation result, determine whether the second evaluation result meets the second preset condition. If the second evaluation result meets the second preset condition, the enhanced sample is output as the enhanced first data set; if the second evaluation result does not meet the second evaluation result, the enhanced sample continues to be input into the generator, so that the generator adjusts the enhancement method and degree of the data sample based on the second evaluation result, and performs multiple cycles of iteration until the corresponding evaluation result output by the discriminator meets the second preset condition. Among them, the specific setting of the second preset condition is related to the model parameters of the adversarial network, the application scenario of the first data set, the accuracy requirements of the data, etc., and is not specifically limited here.
[0056] In an embodiment of the present application, the data processing method inputs the filtered first data set into the generator of the adversarial network to obtain an enhanced sample output by the generator; inputs the enhanced sample into the discriminator of the adversarial network to obtain a second evaluation result after the discriminator evaluates the enhanced sample; when the second evaluation result meets the second preset condition, outputs the enhanced sample as the enhanced first data set, so that the data sample after data enhancement by the adversarial network can meet the data quality requirements in the large model training process, thereby improving the training effect of the large model.
[0057] Optionally, in another embodiment, step S13 of the data processing method of this embodiment also includes: performing data enhancement on the filtered first data set through a variational autoencoder to obtain an enhanced first data set, or, applying one or more data enhancement operations to each data sample of the filtered first data set, and mixing the enhanced data samples according to a preset ratio to obtain an enhanced first data set.
[0058] Specifically, the data processing method of this embodiment can perform representation learning on the filtered first data set through a variational autoencoder to learn the data distribution and characteristics of the filtered first data set; and decode random samples based on the learned data distribution and characteristics to generate enhanced data samples having similar characteristics to the first data set through decoding, thereby achieving data enhancement.
[0059] Alternatively, the data processing method of this embodiment can also apply one or more data enhancement operations to each data sample of the filtered first data set; wherein, when the first data set includes text data, the data enhancement operation may include random insertion, random replacement, synonym replacement, data expansion, and sentence reconstruction, etc.; when the first data set includes image data, the data enhancement operation may include cropping, flipping, rotation, and color disturbance, etc. After the data enhancement operation, the enhanced data samples are mixed according to a preset ratio to obtain an enhanced first data set; wherein the preset ratio can be one or more proportion data arbitrarily extracted from a proportion data set including multiple proportions. After data enhancement by the above-mentioned mixing operation of the data samples, the data diversity of the enhanced first data set can be improved, and the generalization ability of the large model can be improved.
[0060] In one embodiment, the step of performing data cleaning on the first data set in step S11 includes performing preliminary cleaning and / or deep cleaning on the first data set.
[0061] In a possible implementation, the preliminary cleaning step may include, but is not limited to: removing preset characters in the first data set. Specifically, the above-mentioned method of removing the preset characters in the first data set can be implemented by using regular expressions and related text processing tools. The preset characters may include HTML link tags, line breaks, irrelevant symbols such as ¥, extra spaces, stop words or banned words, and other non-essential characters in text data. By removing the preset characters in the first data set, obvious noise and errors in the first data set can be removed, thereby improving the accuracy of the first data set.
[0062] In another possible implementation, the deep cleaning method includes but is not limited to grammatical correction of the data of the first data set. Specifically, word meaning segmentation and part-of-speech analysis are performed on the data of the first data set, and spelling check or correction is performed on the data of the first data set after analysis, so as to further reduce the noise data of the first data set and improve the semantic accuracy and consistency of the first data set.
[0063] The data of the first data set can be grammatically corrected by text processing tools, natural language processing models, etc. Furthermore, the accuracy of data cleaning can be further improved by combining technologies such as named entity recognition (NER) and sentiment analysis.
[0064] In one embodiment, obtaining at least one first data set to be processed in step S11 further includes: collecting first data sets of a preset type from multiple data sources. After step S14, the data processing method of this embodiment further includes: obtaining update information of the data source based on the first evaluation result; when the update information indicates that the data source includes multiple versions of data, obtaining a third data set after the data source is updated; and inputting the third data set into the pre-trained model to obtain a third evaluation result of the pre-trained model for the third data set.
[0065] Specifically, the data processing method can obtain a first data set by collecting corresponding data on a large scale from multiple credible data sources. According to the application scenario and field of the first data set, the credible data source can be set as a relevant database set up by authoritative institutions such as research institutions, media, and libraries, such as news websites, standard documents, and academic paper databases, to ensure the diversity and coverage of the data.
[0066] When the first evaluation result indicates that the first data set has a low degree of relevance to the application field or the quality of the first data set is low, update information of the data source is obtained; and when the update information indicates that the corresponding data source of the first data set includes multiple versions of data, a third data set after the data source is updated is further obtained. It can be understood that the third data set is the data after the data source is updated after the above-mentioned cleaning, labeling, filtering, enhancement and other processing steps; the third data set can be defined as the new version of data, and the first data set can be defined as the old version of data. The third data set is input into the pre-trained model to obtain a third evaluation result of the pre-trained model for the third data set, and the third data set is output when the third evaluation result meets the preset indicators.
[0067] Furthermore, after obtaining the first data set and the third data set processed by the above steps, the first data set and the third data set can be output to the training process of the application model to be applied, and the data versions of the first data set and the third data set can be marked so that the application model can subsequently feed back its version to the user when calling related data, thereby improving the user experience.
[0068] Furthermore, when the first evaluation result indicates that the data acquisition time of the first data set is long, the data source of the first data set can also be detected and updated in the above manner to improve the data quality and timeliness of the data set, and further improve the application effect and versatility of the large model.
[0069] In one embodiment, step S14 includes: dividing the first data set into multiple data subsets; selecting one of the multiple data subsets as a verification set and the remaining data subsets as training sets to perform multiple rounds of training on the pre-trained model to obtain verification results of multiple rounds of training; based on the verification results of multiple rounds of training, obtaining a first evaluation result of the pre-trained model.
[0070] Specifically, in each first data set, by dividing the first data set into multiple data subsets, and selecting any one of the multiple data subsets as a validation set and the remaining data subsets as training sets to perform multiple rounds of training on the pre-trained model, the performance of the training model can be more accurately evaluated, so that the first evaluation result obtained can more accurately characterize the quality of the first data set, improve the effect of data processing, and then improve the quality and diversity of the first data set.
[0071] See also Figure 5 , Figure 5 Schematic diagram of a flow chart of an embodiment of the model training method provided by the present application. Figure 5 As shown, the model training method of this embodiment includes the following steps: Step S51: Acquire a first data set by using the data processing method of any of the above embodiments.
[0072] Specifically, through the data processing method of any of the above embodiments, a first data set whose data quality meets the preset requirements can be obtained.
[0073] Step S52: input the first data set into the application model to be trained to obtain a prediction result output by the application model based on the first data set.
[0074] The first data set is input into the application model to be trained to obtain a prediction result output by the application model based on the first data set. The application model includes but is not limited to neural network models used in the fields of machine translation, intelligent question answering, text generation, information extraction, sentiment analysis, audio and video analysis, image processing, personalized recommendation, etc.
[0075] Step S53: iteratively update the application model based on the prediction results.
[0076] After obtaining the prediction results output by the application model, the loss function of the application model is calculated based on the prediction results, and the application model is iteratively updated based on the loss function until a converged application model is obtained.
[0077] Among them, after improving the quality and diversity of the first data set by the data processing method of any of the above embodiments, when the first data set is applied to the model training of machine translation, the accuracy and fluency of the machine translation model can be improved through the high-quality multilingual data set; when the first data set is applied to the model training of intelligent question and answer, the accuracy and intelligence level of the question and answer model can be improved through high-quality text data, thereby improving the user experience; when the first data set is applied to the model training of text generation, the naturalness and diversity of the text generation model can be improved through high-quality text data; when the first data set is applied to the model training of information extraction, the data quality of the information extraction task can be improved. Enhance the performance of the model in tasks such as entity recognition and relationship extraction; when the first data set is applied to the model training of sentiment analysis, the high-quality data set can be used to improve the model's sentiment recognition accuracy in different scenarios; when the first data set is applied to the model training of sentiment analysis, the high-quality data set can be used to improve the model's sentiment recognition accuracy in different scenarios; when the first data set is applied to the model training of personalized recommendation, the high-quality data set can be used to optimize the recommendation algorithm, and the accuracy of recommendation and user satisfaction can be improved; when the first data set is applied to the model training of personalized recommendation, the high-quality data set can be used to optimize the recommendation algorithm, and the accuracy of recommendation and user satisfaction can be improved.
[0078] In an embodiment of the present application, by performing multi-level data processing and optimization on the first data set, the quality and diversity of the first data set can be effectively improved, manual intervention in the data processing process can be reduced, and the data processing cost can be reduced; when the data of the first data set is applied to model training in various fields, the training effect of the application model can be effectively improved, and the application effect and versatility of the model can be improved.
[0079] In one embodiment, for application models of different scenarios, the types and sources of the first data set are different, and the above-mentioned specific processing methods can be adjusted according to different scenarios to obtain first data sets with different accuracies.
[0080] For example, when the application model is used in scenarios with extremely high data quality requirements, such as legal document analysis, medical text processing, etc., the accuracy of data cleaning can be further improved by introducing technologies such as natural language processing (NLP), named entity recognition (NER), and sentiment analysis. When the application model is used in scenarios that require a large amount of labeled data, such as image annotation in autonomous driving, text classification in the financial field, etc., the efficiency of data annotation can be improved by introducing large-scale data annotation, multi-task learning models, etc. to automatically annotate the data. When the application model is used in scenarios that require a large amount of diverse data, such as speech recognition, image generation, etc., the data diversity can be further improved by combining one or more methods such as adversarial networks (GAN), variational autoencoders (VAE), and hybrid enhancement. When the application model is used in scenarios that require processing data in specific fields or specific sources, such as market research, in addition to automatically collecting data through web crawlers and API interfaces, more data sources can be integrated, such as social media data, user-generated content (UGC), and internal corporate documents to further expand the diversity and coverage of data.
[0081] Through the use of data in different application scenarios, the data processing method of this embodiment can also adjust the specific processing process to improve the applicability of the processed first data set, improve the quality and diversity of the first data set, and then improve the application effect and versatility of the model.
[0082] In one embodiment, the above data processing flow can also be adjusted by acquiring real-time data of the application model during training and application to ensure the effectiveness of data quality improvement and improve the quality of the output data set. For example, the feedback data during the training and application of the application model indicates that the model's prediction data is inaccurate, and the data processing flow can be adjusted by expanding the data source, updating the data of the original data source, adjusting the data enhancement strategy, strengthening the data cleaning method, etc., to improve the quality of the output data set.
[0083] Optionally, algorithms such as reinforcement learning can be used to design a feedback mechanism for the above-mentioned application model so that the application model can automatically adjust and optimize the processing method of the data set according to the feedback information, making the application model more suitable for data processing tasks that require real-time feedback and adjustment, such as real-time recommendation systems, online advertising, etc.
[0084] See also Figure 6 , Figure 6 Schematic diagram of the structure of an embodiment of the electronic device provided by the present application. Figure 6As shown, the electronic device 50 of this embodiment includes a memory 52 and a processor 51, and the processor 51 is connected to the memory 52. The memory 52 is used to store program instructions for implementing the method described in any of the above embodiments. The processor 51 is used to execute the program instructions stored in the memory 52.
[0085] The processor 51 may also be referred to as a CPU (Central Processing Unit). The processor 51 may be an integrated circuit chip having the ability to process signals. The processor 51 may also be a general-purpose processor, a digital signaling processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0086] The memory 52 can be a memory stick, a TF card, etc., which can store all the information in the electronic device 50, including the input raw data, computer programs, intermediate operation results and final operation results are all stored in the memory. It stores and retrieves information according to the location specified by the controller. With the memory, the string matching prediction device has a memory function and can ensure normal operation. The memory of the string matching prediction device can be divided into main memory (internal memory) and auxiliary memory (external memory) according to the purpose. There is also a classification method of dividing it into external memory and internal memory. External memory is usually a magnetic medium or an optical disk, etc., which can store information for a long time. Memory refers to the storage component on the motherboard, which is used to store the data and programs currently being executed, but is only used to temporarily store programs and data. If the power is turned off or the power is cut off, the data will be lost.
[0087] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0088] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0089] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0090] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a system server, or a network device, etc.) or a processor to execute all or part of the steps of the various implementation methods of the present application.
[0091] See also Figure 7 , Figure 7 Schematic diagram of the structure of an embodiment of the computer-readable storage medium provided by the present application. Figure 7 As shown, the computer-readable storage medium of the present application stores program instructions 61 that can implement all the above methods, wherein the program instructions 61 can be stored in the above storage medium in the form of a software product, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the various implementation methods of the present application. The aforementioned storage device includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or electronic devices such as a computer, a server, a mobile phone, and a tablet.
[0092] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A data processing method, characterized in that: include: Acquire at least one first data set to be processed, and perform data cleaning on the first data set; Annotating and filtering the cleaned first data set; Performing data enhancement on the first filtered data set; Inputting the enhanced first data set into a pre-trained model to obtain a first evaluation result of the pre-trained model for the first data set; When the first evaluation result meets the preset indicator, the first data set is output.
2. The data processing method according to claim 1, characterized in that: The labeling and filtering of the cleaned first data set includes: Acquire a first data set to be processed, perform data cleaning on the first data set, and annotate the cleaned first data set; Performing a quality score on the annotated first data set to obtain score information of the first data set; The labeled first data set is filtered based on the scoring information of the first data set.
3. The data processing method according to claim 2, characterized in that: The filtering the labeled first data set based on the plurality of scoring information of the first data set includes: When the scoring information indicates that the data in the first data set does not meet the first preset condition, filtering the corresponding data from the first data set and adding it to the second data set; Performing data enhancement on the data in the second data set; Performing a quality score on the enhanced second data set; When the scoring information of the second data set meets the first preset condition, the corresponding data is added to the first data set to obtain the filtered first data set.
4. The data processing method according to claim 1, characterized in that: The performing data enhancement on the filtered first data set includes: Inputting the filtered first data set into a generator of an adversarial network to obtain enhanced samples output by the generator; Inputting the enhanced sample into the discriminator of the adversarial network to obtain a second evaluation result after the discriminator evaluates the enhanced sample; When the second evaluation result meets a second preset condition, the enhanced sample is output as the enhanced first data set.
5. The data processing method according to claim 1, characterized in that: The performing data enhancement on the filtered first data set includes: Performing data enhancement on the filtered first data set by a variational autoencoder to obtain an enhanced first data set, or, One or more data enhancement operations are applied to each data sample of the filtered first data set, and the enhanced data samples are mixed according to a preset ratio to obtain an enhanced first data set.
6. The data processing method according to claim 1, characterized in that: The performing data cleaning on the first data set includes: removing the preset characters in the first data set, and / or, Performing grammatical correction on the data of the first data set.
7. The data processing method according to claim 1, characterized in that: The obtaining of at least one first data set to be processed includes: Collecting the first data set of a preset type from a plurality of data sources; After the step of inputting the enhanced first data set into the pre-training model to obtain a first evaluation result of the pre-training model, the data processing method further includes: Based on the first evaluation result, obtaining update information of the data source; When the update information indicates that the data source includes multiple versions of data, acquiring a third data set after the data source is updated; The third data set is input into the pre-trained model to obtain a third evaluation result of the pre-trained model for the third data set.
8. A model training method, characterized in that: include: Acquire a first data set by the data processing method according to any one of claims 1 to 7; Inputting the first data set into an application model to be trained to obtain a prediction result output by the application model based on the first data set; The application model is iteratively updated based on the prediction result.
9. An electronic device, characterized in that: The invention comprises a processor and a memory connected to the processor, wherein: The memory stores program instructions; The processor is used to execute the program instructions stored in the memory to implement the method according to any one of claims 1 to 8.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions, and the program instructions can be executed by a processor to implement the method according to any one of claims 1 to 8.
Citation Information
Patent Citations
Information processing and model training method and device, equipment, medium and program product
CN117494761A
Fine adjustment data screening method and device, computer equipment and readable storage medium
CN118504663A
Method and apparatus for smart analysis of question and answer linguistic material, electronic device, and readable storage medium
WO2021151306A1
Cited By
Sample data synthesis method and device, equipment and medium
CN119761534A
