Data cleaning method

Abstract

The invention discloses a data cleaning method, and relates to the technical field of natural language processing, and the method comprises the steps: calculating a value evaluation score based on a loss function, and enabling the score to measure the prediction accuracy of model parameters on lexical elements, so as to recognize low-value lexical elements such as noise and redundancy through the score. Then, target lexical elements in a preset interval are arranged and screened according to the scores, so that the lexical elements which are low in value and contain noise and redundancy are cleaned away; therefore, parts having negative effects on fine tuning of the model are removed, the data are purer, the data quality is improved, and the technical problem that the data quality is affected by lexical element level noise and redundant information in a sample in the process of supervising fine tuning of the pre-training language model is solved.

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and in particular to a data cleaning method. Background Art

[0002] In the field of natural language processing, the birth of Large Language Models (LLMs) has brought about major changes and demonstrated powerful multi-task processing capabilities. Supervised Fine-Tuning (SFT), as a key link, can make LLMs more suitable for specific tasks and human expectations, and is widely used in intelligent customer service, intelligent writing, knowledge question and answer, and other fields. With the development of LLMs, the importance of data in model training has become increasingly prominent. Studies have shown that the impact of data quality on model performance far exceeds the amount of data.

[0003] At present, a variety of data cleaning and screening methods have been proposed in the SFT stage of LLM, such as screening based on the perplexity, completion length, and quality score of model generation at the sample level. However, these methods have obvious limitations. They only focus on the overall quality assessment of samples and ignore the quality differences within the samples, resulting in reduced model training efficiency, learning wrong patterns or information, affecting the performance of downstream tasks, and making it difficult to fully utilize the value of high-quality data, which limits the performance improvement of LLMs in SFT tasks. Summary of the invention

[0004] The present application provides a data cleaning method to at least solve the problem that data quality is affected by word-level noise and redundant information within the sample during the supervised fine-tuning process of a pre-trained language model.

[0005] This application provides a data cleaning method, including: Get the word to be processed; Based on the loss function of the pre-trained language model, the value evaluation score of the word to be processed is calculated. The value evaluation score is used to measure the prediction accuracy of different pre-trained language model parameters on the word to be processed; The words to be processed are arranged according to the value evaluation scores, and the target words in the preset range are screened out.

[0006] The present application also provides a data cleaning device, comprising: An acquisition module is used to acquire the word to be processed; A calculation module, used for calculating the value evaluation score of the word to be processed based on the loss function of the pre-trained language model, and the value evaluation score is used to measure the prediction accuracy of different pre-trained language model parameters on the word to be processed; The screening module is used to arrange the words to be processed according to the value evaluation scores, and screen out the target words in a preset range. The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned data cleaning methods when executing the computer program.

[0007] The present application also provides a computer-readable storage medium, in which a computer program is stored, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned data cleaning methods are implemented.

[0008] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned data cleaning methods when executed by a processor.

[0009] Through this application, the value assessment score of the word to be processed is calculated based on the loss function of the pre-trained language model. This score measures the prediction accuracy of the word by different pre-trained language model parameters. During the supervised fine-tuning process of the pre-trained language model, the noise and redundant information at the word level in the sample often cause the model to deviate from the prediction of these worthless or low-value words. By calculating the value assessment score, it is possible to identify which words have less impact on the model's prediction accuracy, and these words are likely to be the source of noise or redundant information.

[0010] This application arranges the tokens to be processed according to their value assessment scores and screens out target tokens in a preset range. This operation can exclude tokens with low value assessment scores (i.e., tokens that may contain noise and redundant information). Through this screening method, these tokens that have a negative impact on model fine-tuning can be effectively removed, thereby improving data quality and reducing their interference with the supervised fine-tuning of the pre-trained language model.

[0011] Therefore, the present application can solve the technical problem that data quality is affected by word-level noise and redundant information within the sample during the supervised fine-tuning of the pre-trained language model, thereby improving data quality, accelerating model convergence, enhancing the generalization ability of the model, and optimizing resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0013] Figure 1 A schematic diagram of an application environment architecture for running a data cleaning method provided in an embodiment of the present application; Figure 2A schematic diagram of a hardware architecture for running a data cleaning method provided in an embodiment of the present application; Figure 3 A flowchart of a data cleaning method provided in an embodiment of the present application; Figure 4 A schematic diagram of a process of iteratively training a pre-trained language model provided in an embodiment of the present application; Figure 5 A schematic diagram of the structure of a data cleaning device provided in an embodiment of the present application; Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0014] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0015] It should be noted that, in the description of this application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0016] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0017] In conjunction with the specific application environment architecture or the specific hardware architecture on which the execution of the data cleaning method depends, the specific application environment architecture or the specific hardware architecture is described herein.

[0018] like Figure 1 As shown, Figure 1 A schematic diagram of an application environment architecture for running a data cleaning method provided in an embodiment of the present application, wherein the application environment architecture includes a data storage layer, a computing framework layer, and a task scheduling and management system.

[0019] Among them, the data storage layer needs to have high-capacity and high-reliability storage devices, such as distributed file systems, to store massive amounts of corpus data to be cleaned and meet the cleaning method's requirements for large-scale data processing.

[0020] The computing framework layer is based on the deep learning framework and provides computing support for operations such as calculating word value evaluation scores based on loss functions. These frameworks can efficiently calculate the loss function of the pre-trained language model to obtain the word value evaluation score.

[0021] The task scheduling and management system is responsible for scheduling cleaning tasks, rationally allocating computing resources, managing the task execution of each computing node, and ensuring that the cleaning work is carried out efficiently and stably.

[0022] like Figure 2 As shown, Figure 2 A schematic diagram of a hardware architecture for running a data cleaning method provided in an embodiment of the present application, wherein the hardware architecture includes a high-performance central processing unit (CPU) cluster, a graphics processing unit (GPU) accelerator card, and high-speed network equipment.

[0023] Among them, the high-performance CPU cluster undertakes basic computing tasks such as model initialization, data preprocessing, and task coordination. Its multi-core and high-frequency characteristics help to quickly process data reading, simple calculations, and logic control.

[0024] GPU accelerator cards are core computing hardware, especially for complex matrix operations of pre-trained language models. They accelerate model training and word value evaluation score calculation processes, greatly improving cleaning efficiency.

[0025] High-speed network equipment is required to achieve fast data transmission between CPU and GPU and computing nodes, and 10 Gigabit Ethernet or even InfiniBand network is required. When cleaning large-scale corpus, data frequently interacts between different devices and nodes. High-speed network can reduce data transmission delay, ensure full utilization of computing resources, and avoid data transmission becoming a performance bottleneck.

[0026] Supervised Fine-Tuning (SFT) is the process of further training based on a pre-trained model using labeled data. By fine-tuning the pre-trained model on a labeled dataset for a specific task, the model can better adapt to the needs of the specific task, thereby improving its performance on the task. During the supervised fine-tuning stage of the pre-trained language model, the noise and redundant information at the word level within the sample seriously interfere with the data quality. Noise includes incorrect labels, information irrelevant to the task, etc., which cause the model to receive wrong signals during learning and confuse the acquisition of key knowledge. Redundant information, such as common words and phrases that appear repeatedly but are not very helpful for the task, increases the model training load and reduces training efficiency. These negative factors are intertwined, making it difficult to ensure data quality, hindering the pre-trained language model from fully leveraging the advantages of supervised fine-tuning, and urgently need to be solved by effective means.

[0027] In order to fully or partially solve the problems existing in the related technologies, an embodiment of the present application provides a data cleaning method, and the method is described in detail in conjunction with the execution process of the data cleaning method.

[0028] like Figure 3 As shown, the data cleaning method mainly includes the following steps S300~S302: S300: Obtain a word to be processed.

[0029] In some embodiments, the present application is applicable to multimodal data cleaning. When executing step S300, multimodal data is first obtained, and then the multimodal data is segmented and / or encoded to obtain the word to be processed. Thereby, the corpus of different modalities is uniformly represented as a continuous word sequence. Among them, multimodal data includes but is not limited to text, images, and videos. For text data, the present application can use sub-word segmentation methods such as word pieces (WordPiece) or byte pair encoding (Byte-Pair Encoding, BPE) to encode the word to be processed, so as to control the vocabulary while retaining the semantics. For image data, it can be divided into image blocks (patches) and encoded to obtain the word to be processed, which can capture local features. For video data, it can be decomposed into a frame sequence and spatiotemporally encoded to obtain the word to be processed, which comprehensively considers the relationship between frames and spatial information.

[0030] The above embodiment converts multimodal data into a unified word sequence, so that the pre-trained language model can process the corpus in a unified way, which is convenient for the model to learn different modal information and improve the compatibility of the pre-trained language model. Different segmentation and encoding methods are used for data of different modalities to form effective representations, which helps the pre-trained language model understand and associate different modal semantics, thereby improving model performance.

[0031] S301. Calculate the value assessment score of the word to be processed based on the loss function of the pre-trained language model.

[0032] The value assessment score is used to measure the prediction accuracy of different pre-trained language model parameters for the token to be processed. Different pre-trained language model parameters can be model parameters of different pre-trained language models, or different model parameters of the same pre-trained language model, which can be understood as model parameters at different stages of the pre-trained language model training process. The higher the prediction accuracy of the token to be processed, the smaller the value assessment score.

[0033] S302: Arrange the words to be processed according to the value evaluation scores, and select target words in a preset range.

[0034] The above steps S300~S302 can effectively remove noise and redundant words in the corpus by calculating the value evaluation score of the word unit, screening and cleaning the word units, and improving the quality of the corpus data, thereby improving the training efficiency and performance of the pre-trained language model.

[0035] In some embodiments, the pre-trained language model may be a different pre-trained language model, including a first model and a second model, and the performance of the second model is better than the first model. Correspondingly, the different pre-trained language model parameters include first model parameters and second model parameters. It should be noted that the present application does not limit the number of different pre-trained language models, and two are used as an example for illustration.

[0036] When executing step S301, firstly, a first loss function of the word to be processed under a preset context window and a first model parameter and a second loss function of the word to be processed under a preset context window and a second model parameter are calculated. Then, based on the difference between the second loss function and the first loss function, a value assessment score is calculated.

[0037] Optionally, in the process of calculating the value evaluation score based on the difference between the second loss function and the first loss function, the value evaluation score Score of the word to be processed can be calculated according to the following formula (1): (1) In formula (1), x i，j Indicates the token to be processed, x i，0：j-1 Indicates the preset context window where the token to be processed is located. θ and θ' represent different pre-trained language model parameters, such as the first model parameter θ and the second model parameter θ'. l ( x i，j | xi，0：j-1 ; θ ) is the word to be processed in the preset context window x i，0：j-1 And the loss function under the model parameters θ, such as the first loss function, ; l ( x i，j | x i，0：j-1 ; θ ') is the word to be processed in the preset context window x i，0：j-1 The loss function under the model parameters θ' is the second loss function. The change of the loss function can measure the information gain or loss of the pre-trained language model for the processed word under different model parameters θ and θ', reflecting the data quality related to the word to be processed. The pre-trained language model has a given context window. x i，0：j-1 In the case of x i，j The higher the prediction accuracy, the smaller the Score value.

[0038] The above-mentioned embodiment calculates the value assessment score of the word to be processed through the model parameters of different pre-trained language models. Due to the differences in architecture, training data and training objectives of different pre-trained language models, the understanding and focus of the same word are different. The value of the word can be considered from multiple angles to avoid the limitations of a single model, thereby more accurately judging the importance of the word to model training. By combining the judgments of different pre-trained language models, various noise words in the corpus can be more comprehensively identified. Different pre-trained language models can judge whether a word is redundant from different angles, thereby more accurately screening out redundant words and improving the quality and information density of the corpus. In addition, different pre-trained language models learn different language knowledge and patterns during the training process. By using the parameters of the two models for word cleaning, these diverse knowledge can be integrated into the corpus, so that the model is exposed to richer language information during the training process, thereby enhancing the generalization ability of the model.

[0039] Based on the above embodiment, in the process of calculating the value assessment score based on the difference between the second loss function and the first loss function, the frequency of occurrence of the word to be processed in the corpus is first obtained, and then the value assessment score is calculated based on the frequency of occurrence, the difference between the first loss function and the second loss function.

[0040] Specifically, the value assessment score Score is calculated as shown in formula (2).

[0041] (2) In formula (2), in addition to the difference between the first loss function and the second loss function, a penalty term regarding the expected frequency of occurrence of the word to be processed is included. The hyperparameter α is used to control the weight of the penalty term. N x，i，j is the number of times the token to be processed appears in the corpus, N total is the total number of tokens in the corpus. This penalty term indicates that the higher the frequency of the token to be processed in the corpus, the higher the possibility of being cleaned. This allows the pre-trained language model to focus on relatively uncommon but more critical tokens when processing the corpus.

[0042] The above embodiment uses the frequency penalty term to suppress the scores of high-frequency words, increase the possibility of being discarded, avoid the model's over-reliance on high-frequency common words, and encourage it to learn more diverse and discriminative word information. When faced with new data, the model can better adapt to different contexts and semantics, reduce the problem of insufficient generalization caused by the dominance of high-frequency words, and improve performance and generalization capabilities in different scenarios.

[0043] The value evaluation parameter can also be calculated based on accuracy, log-likelihood, and entropy, etc. The above is only an exemplary description, and the present application does not specifically limit the calculation method of the value evaluation parameter.

[0044] In some embodiments, when executing step S302, the word units to be processed are first clustered according to their parts of speech to obtain at least one part-of-speech clustering result; then, the word units to be processed in at least one part-of-speech clustering result are arranged according to the value evaluation scores, and the target word units in the preset interval are screened according to the preset deletable threshold.

[0045] The preset deletable threshold represents the maximum number of words that can be deleted under each part of speech, which is used to prevent too many words of a certain part of speech from being deleted at one time, so as to avoid semantic discontinuity and semantic loss.

[0046] Specifically, the parts of speech of the word to be processed include but are not limited to: nouns, pronouns, adjectives, adverbs, verbs, numerals, articles, prepositions, conjunctions, interjections, participles, and infinitives. The word to be processed is clustered according to these part-of-speech types to obtain clustering results of different parts of speech. Then, for each part-of-speech clustering result, the value evaluation scores of the word to be processed are arranged from high to low, and then the word is cleaned according to the preset deletable threshold and the preset interval to obtain the target word. Compare whether the number of other word units in the preset interval in each part-of-speech clustering result is greater than the preset deletable threshold. If so, it means that if these other word units are deleted at one time, semantic problems will occur. Therefore, in this application, the number of word units retained after cleaning and arranging the part-of-speech clustering results according to the preset deletable threshold is greater than the preset interval size. If the number of other word units is less than or equal to the preset deletable threshold, it means that if these other word units are deleted at one time, there will be no semantic problems. Then, these other word units are first deleted from the arranged part-of-speech clustering results to obtain the target word units in the preset interval.

[0047] The above embodiment uses part-of-speech clustering to allow the large model to clearly distinguish words with different grammatical functions, thereby improving the accuracy of understanding the semantics of the text, limiting the maximum number of words that can be deleted, and retaining high-value words, which is conducive to the model focusing on semantic information, avoiding learning too much redundant or low-value information, and reducing the risk of overfitting. This screening mechanism enables the model to learn more representative features, generalize better when facing new data, and improve the performance of the model in various scenarios.

[0048] In some embodiments, when executing step S302, the words to be processed are first arranged according to the value evaluation scores to obtain a first ascending result. Then, according to the total number of words to be processed and the preset proportional coefficient, a preset interval is calculated, and a target label is further set for the words to be processed in the preset interval in the first sorting result to screen and obtain the target word. The target label indicates that the target word is "valuable", indicating that the target label is retained.

[0049] Specifically, the value evaluation scores of all the words to be processed are arranged in order from high to low to obtain the sorting result. Then, the total number of words to be processed is taken as the base number, combined with the preset proportional coefficient (such as k , which can be a fixed constant), by multiplying the total number of words to be processed by the preset proportional coefficient, determining the range boundary of the preset interval, and further providing the words that are ranked at the top in the first sorting result. k % of the to-be-processed tokens are set with target labels to set the "valuable" label, and these to-be-processed tokens are retained as target tokens. Otherwise, the "valueless" label is set for the to-be-processed tokens, and these to-be-processed tokens are removed.

[0050] The above embodiment sets a proportional coefficient to filter out words with higher scores and greater value, removes a large number of low-value words, and improves the efficiency of model training. At the same time, it prevents the model from being disturbed by a large number of meaningless or low-value words, and improves the pertinence and effectiveness of data processing.

[0051] In the process of training the same pre-trained language model, the model parameters are adjusted to enable the pre-trained language model to better fit the training data, thereby improving the prediction ability of new data. In some embodiments, for the training of the same pre-trained language model, when executing step S300, the data sample set of the pre-trained language model is first divided into multiple data sample subsets, and then any data sample subset is used to fine-tune the basic model parameters of the pre-trained language model to obtain the initial model parameters. Any data sample subset is randomly selected from multiple data sample subsets. Then, the word units to be processed are obtained from other data sample subsets. The other data sample subsets are data sample subsets in multiple data sample subsets other than any of the aforementioned data sample subsets. Then, for these word units to be processed obtained from other data sample subsets, the aforementioned steps S301~S302 are executed.

[0052] In the process of executing steps S301 to S302 for the word-grams to be processed obtained from other data sample subsets, the loss function of the word-grams to be processed of other data sample subsets under their context windows and basic model parameters is first calculated, and the loss function of the word-grams to be processed of other data sample subsets under their context windows and initial model parameters is calculated. Then, based on the difference between the two loss functions, the value evaluation score of the word-grams to be processed of other data sample subsets is obtained, referring to formula (1) or (2).

[0053] It should be noted that when the model is updated from the basic model parameter θ to the initial model parameter θ', this parameter change will cause the model's prediction ability for each word to change. At this time, l ( x i，j | x i，0：j-1 ; θ )and l ( x i，j | x i，0：j-1 ; θ ') respectively represent the basic model parameters θ and the initial model parameters θ', x i，j This word is in the given context window x i，0：j-1The opposite number of the predicted probability under . Updating the model from parameter θ to θ' will change the predicted probability distribution of the pre-trained language model for the word to be processed, thereby affecting the prediction accuracy. A negative value of the value assessment score indicates that the pre-trained language model is more accurate in predicting the word to be processed after the model parameters are updated; a positive value of the value assessment score indicates that the pre-trained language model's prediction accuracy for the word to be processed is reduced after the model parameters are updated. For example, if the pre-trained language model predicts a certain word to be processed more accurately after the model parameters are updated, it means that the information carried by the word to be processed is more valuable to the training of the pre-trained language model, and the data quality related to the word to be processed is higher; conversely, if the pre-trained language model's prediction accuracy for a certain word to be processed is reduced after the model parameters are updated, it means that there may be problems with the data related to the word to be processed.

[0054] Further, the words to be processed in other data sample subsets are sorted according to these value evaluation scores, and the top ranked words are selected. k % of the to-be-processed words are used as target words for other data sample subsets. Here, the to-be-processed words of other data sample subsets can be clustered by part of speech to obtain at least one part of speech clustering result, and then the to-be-processed words in at least one part of speech clustering result are sorted in ascending order according to the value evaluation score, and then the words ranked at the top are filtered according to the preset deletable threshold. k % of the to-be-processed tokens are used as target tokens for other data sample subsets.

[0055] Afterwards, the target word units of other data sample subsets are used to fine-tune the initial model parameters to obtain the reference model parameters. Then, the third loss function of the target word unit under the preset context window and basic model parameters, and the fourth loss function of the target word unit under the preset context window and reference model parameters are calculated, and then the value evaluation score is obtained based on the difference between the fourth loss function and the third loss function, such as formula (1) or (2). The target word units are arranged according to these value evaluation scores, and the word units in the preset range are screened for iterative training of the pre-trained language model until the model converges.

[0056] It should be noted that in the above embodiment, the preset proportionality factor may be a fixed constant. k If the word element does not change, the same rules can be used to filter the word element, ensuring the consistency of data processing. This helps to improve the stability of the model training process, avoid training fluctuations caused by changes in word element filtering rules, and enable the model to learn data features more robustly. Optionally, the preset proportional coefficient may change with model iteration. Thus, in the above embodiment, when the target word units are arranged according to these value assessment scores, the word units in the preset interval are screened for iterative training of the pre-trained language model until the model converges, the following steps may be included: first, the target word units are arranged according to the value assessment scores to obtain a second sorting result, and then the preset interval is calculated according to the total number of target word units and the preset proportional coefficient, where the preset proportional coefficient is reduced as the number of iteration steps increases during the iterative training of the pre-trained language model. Further, the word units in the preset interval are screened out from the second sorting result for subsequent iterative training of the pre-trained language model until a converged pre-trained language model is obtained.

[0057] Specifically, in the initial stage of iterative training, in the process of executing steps S301 to S302 on other data sample subsets, the loss function of the to-be-processed word of other data sample subsets under its context window and basic model parameters is first calculated, and the loss function of the to-be-processed word of other data sample subsets under its context window and initial model parameters is calculated. Then the difference between the two loss functions is calculated to obtain the value evaluation score of the to-be-processed word of other data sample subsets, and the to-be-processed word of other data sample subsets is further sorted according to these value evaluation scores, and the preset interval is calculated according to the total number of to-be-processed word and a larger preset proportional coefficient, so as to select the to-be-processed word in the preset interval from the first sorting result as the target word. In the second iteration, the target word of other data sample subsets is used to fine-tune the initial model parameters to obtain the reference model parameters. Then the third loss function of the target word under the preset context window and basic model parameters, and the fourth loss function of the target word under the preset context window and reference model parameters are calculated, and then the value evaluation score is obtained based on the difference between the fourth loss function and the third loss function. The target words are arranged according to these value evaluation scores to obtain a second sorting result, and then another preset interval is calculated based on the total number of target words and a new preset proportional coefficient that is smaller than the preset proportional coefficient in the previous iteration, so as to filter out the target words in this new preset leading interval from the second sorting result as samples for the next iterative training. The subsequent iterative training process is similar to the above, wherein the preset proportional coefficient is gradually reduced to ensure the degree of word cleaning and the speed of model training, until a converged pre-trained language model is obtained.

[0058] For example, Figure 4 As shown, the data sample set D of the pre-trained language model is divided into multiple data sample subsets { D 0 , D 1 , …, D N}, and then use the data sample subset D 0 Fine-tune the base model parameters θ of the pre-trained language model to obtain the initial model parameters θ 0 , and then calculate other data sample subsets { D 1 , …, D N}, and use the value evaluation score to clean the worthless words to get the target word of the word to be processed. Then use the target word of the word to be processed to fine-tune the initial model parameter θ 0 Get the reference model parameter θ 1 , and then perform the second iteration. In the second iteration, based on the reference model parameter θ 1 Calculate the value evaluation score of the target word to further clean the low-value words and use the cleaned fine-tuned reference model parameters θ 1 Get the new model coefficient θ 2……n Perform subsequent iterations. In the subsequent iterations, based on the new model coefficients θ 2……n Calculate the value assessment parameters and clean the words until n=N. At the end of the iteration, obtain the cleaned data sample set and the pre-trained language model with converged parameter model.

[0059] In the above embodiment, in the early stage of training, the larger k This allows more words to be retained, avoiding the loss of potentially useful information due to premature screening, providing the model with rich and diverse information, allowing the model to quickly learn the basic patterns and broad features in the corpus and build a preliminary knowledge system. k Gradually narrowing down, the model begins to focus on a small number of more valuable words, deeply learning complex, key semantic information and detailed features, allowing the model to focus on processing core information, improving data utilization efficiency, while reducing computing costs and training time. This progressive learning from broad to fine improves the depth and accuracy of the model's understanding of the corpus.

[0060] On the basis of the above-mentioned embodiments, the present application provides an optional implementation manner, in which the preset proportional coefficient exhibits a cosine decay as the number of iteration steps increases during the iterative training process of the pre-trained language model. Accordingly, after the target words are arranged according to the value assessment scores to obtain the second sorting result, and before the preset interval is calculated according to the total number of the target words and the preset proportional coefficient, it also includes: first determining the current iteration step number and the total iteration step number of the pre-trained language model, and then calculating the preset proportional coefficient according to the first proportional coefficient boundary value, the second proportional coefficient boundary value, and the current iteration step number and the total iteration step number. The first proportional coefficient boundary value and the second proportional coefficient boundary value are pre-set, the first proportional coefficient is greater than the second proportional coefficient boundary value, and the first proportional coefficient boundary value can be expressed as kmax , the second proportional coefficient boundary value can be expressed as k min . So that the preset proportional coefficient in the iteration process of the pre-trained language model is increased from the first proportional coefficient boundary value k max Gradually reduce to the second proportional coefficient boundary value k min .

[0061] Specifically, according to the first proportional coefficient boundary value, the second proportional coefficient boundary value, the current iteration step number and the total iteration step number, the preset proportional coefficient is calculated as shown in formula (3): (3) In formula (3), k max is the first proportional coefficient boundary value, k min is the second proportional coefficient boundary value, n is the current iteration number, N is the total number of iteration steps. k It decays cosinely with the number of iterations.

[0062] The above embodiment utilizes the smooth change characteristic of cosine attenuation to avoid k Value mutations have an impact on the model, which helps stabilize the model's training and enables it to better transfer and apply learned knowledge when faced with new data.

[0063] In some embodiments, the present application establishes a feedback mechanism, specifically including using a converged pre-trained language model to perform downstream tasks, determining the accuracy of the converged pre-trained language model in the downstream tasks; then, adjusting the aforementioned first proportional coefficient boundary value and the second proportional coefficient boundary value according to the accuracy to achieve feedback on the iterative training of the large language model. The direction and magnitude of the adjustment of the first proportional coefficient boundary value and the second proportional coefficient boundary value are related to the type of downstream tasks.

[0064] Among them, the downstream tasks can be text classification tasks, image classification tasks, mathematical logic analysis tasks, etc., which are not specifically limited in this application. Different downstream tasks have different characteristics and requirements. Dynamically adjusting the first proportional coefficient boundary value and the second proportional coefficient boundary value according to the task performance can allow the model to find the most suitable word unit screening strategy in different tasks.

[0065] For example, for text classification tasks and image classification tasks, considering that the word to be processed contains more noise and the influence of the word to be processed on the task result is more uniform, if the accuracy of the pre-trained language model in the task decreases after the training convergence based on the cleaned word, the first proportional coefficient boundary value is increased. k maxand the second proportionality coefficient boundary value k min For tasks of mathematical logic analysis, due to the characteristic that the inference result changes suddenly due to the lack of a small number of keyword tokens, if the accuracy of the pre-trained language model decreases in the task after converging based on the cleaned tokens, then reduce the first proportionality coefficient boundary value k max and the second proportionality coefficient boundary value k min .

[0066] Through the above embodiments, by evaluating the performance of the model on downstream tasks, it is possible to directly understand the advantages and disadvantages of the model under the first proportionality coefficient boundary value and the second proportionality coefficient boundary value. If the task performance is not good, it can be adjusted针对性地 k , so as to change the way the model filters tokens, and then make the model focus on the information that is more conducive to completing downstream tasks, improve key indicators such as the accuracy and recall rate of the model on this task, and optimize the overall performance of the model. The model can flexibly adapt to the needs of different tasks by adjusting the first proportionality coefficient boundary value and the second proportionality coefficient boundary value, maintain good performance, and is conducive to improving the practicability and scalability of the model.

[0067] In summary, the present application provides a data cleaning method, which calculates the value evaluation score of the token to be processed based on the loss function of the pre-trained language model. This score measures the prediction accuracy of different pre-trained language model parameters for this token. During the supervised fine-tuning process of the pre-trained language model, noise and redundant information at the token level within the sample often cause the model to have deviations in predicting these worthless or low-value tokens, increasing the value of the loss function. By calculating the value evaluation score, it is possible to identify which tokens have less impact on the prediction accuracy of the model, and these tokens are likely to be the sources of noise or redundant information. For example, some function words that are frequent but have little semantic contribution (such as "de", "le", etc.) may get a lower score when calculating the value evaluation score because their role in improving the prediction accuracy of the model is limited.

[0068] Moreover, the present application arranges the tokens to be processed according to the value evaluation score and filters out the target tokens within the preset interval. This operation can exclude tokens with lower value evaluation scores (i.e., tokens that may contain noise and redundant information). Because the tokens corresponding to noise and redundant information often have lower value evaluation scores, through this filtering method, these tokens that have a negative impact on the fine-tuning of the model can be effectively removed, thereby improving the data quality and reducing their interference with the supervised fine-tuning of the pre-trained language model.

[0069] Compared with the related technologies, the present application has the following technical effects: (1) By removing noise and redundant information, the data used for supervised fine-tuning of pre-trained language models becomes purer and more valuable. High-quality data enables the model to learn more accurate and useful knowledge, thereby improving the performance and representation of the model. For example, in text classification tasks, after removing noise and redundant information, the model can focus more on the key semantic information in the text, improving the classification accuracy.

[0070] (2) As the quality of the cleaned data improves, the model can converge to the optimal solution faster during fine-tuning. By reducing the interference of noise and redundant information, the model can more effectively learn the patterns and regularities in the data and avoid making too many adjustments in the wrong direction, thereby shortening the training time and improving the training efficiency.

[0071] (3) By fine-tuning the model using cleaned data, it can learn more representative and universal features instead of overfitting the noise and redundant information in the training data. This helps the model to better predict and infer new and unseen data, improves the model's generalization ability, and enables it to perform more stably in different scenarios and tasks.

[0072] (4) After removing noise and redundant information, the amount of data is relatively reduced, which means that the computing resources (such as memory, CPU, GPU, etc.) required during model training will also be reduced accordingly. More tasks can be processed under the same hardware conditions, or smaller computing devices can be used to complete training, thereby reducing computing costs and improving resource utilization efficiency.

[0073] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0074] The embodiment of the present application also provides a data cleaning device, such as Figure 5 As shown, the device comprises: An acquisition module 500 is used to acquire a word to be processed; A calculation module 501 is used to calculate the value evaluation score of the word to be processed based on the loss function of the pre-trained language model, and the value evaluation score is used to measure the prediction accuracy of different pre-trained language model parameters on the word to be processed; The screening module 502 is used to arrange the to-be-processed words according to the value evaluation scores, and screen out target words in a preset range.

[0075] As an optional implementation provided in the embodiments of the present application, the acquisition module 500 is specifically used to: acquire multimodal data; segment and / or encode the multimodal data to obtain word units to be processed.

[0076] As an optional implementation provided in an embodiment of the present application, the pre-trained language model includes a first model and a second model, and the performance of the second model is better than that of the first model; The calculation module 501 is specifically used to: calculate the first loss function of the word to be processed under the preset context window and the first model parameters, and the second loss function of the word to be processed under the preset context window and the second model parameters; based on the difference between the second loss function and the first loss function, obtain a value assessment score.

[0077] As an optional implementation provided in an embodiment of the present application, the calculation module 501 is specifically used to: obtain the frequency of occurrence of the word to be processed in the corpus; calculate the value assessment score based on the difference between the second loss function and the first loss function, and the frequency of occurrence.

[0078] As an optional implementation provided in the embodiments of the present application, the screening module 502 is specifically used to: cluster the words to be processed according to their parts of speech to obtain at least one part-of-speech clustering result; arrange the words to be processed in at least one part-of-speech clustering result according to the value evaluation score, and screen according to a preset deletable threshold to obtain target words in a preset interval.

[0079] As an optional implementation provided in the embodiment of the present application, the screening module 502 is specifically used to: arrange the to-be-processed words according to the value assessment scores to obtain a first sorting result; calculate a preset interval according to the total number of to-be-processed words and a preset proportional coefficient; and use the to-be-processed words in the first sorting result that are in the preset interval as target words.

[0080] As an optional implementation provided in an embodiment of the present application, the acquisition module 500 is specifically used to: divide the data sample set of the pre-trained language model into multiple data sample subsets; use any data sample subset to fine-tune the basic model parameters of the pre-trained language model to obtain initial model parameters; obtain the word units to be processed from other data sample subsets, and the other data sample subsets are data sample subsets other than any data sample subset among the multiple data sample subsets.

[0081] As an optional implementation provided in the embodiment of the present application, the device further includes an iteration module, which is used to: fine-tune the initial model parameters using target word units corresponding to other data sample subsets to obtain reference model parameters; Calculate the third loss function of the target word under the preset context window and basic model parameters, and the fourth loss function of the target word under the preset context window and reference model parameters; calculate the value evaluation score based on the difference between the fourth loss function and the third loss function; arrange the target word according to the value evaluation score, and screen the word in the preset interval for iterative training of the pre-trained language model until a converged pre-trained language model is obtained.

[0082] As an optional implementation provided in an embodiment of the present application, the iteration module is specifically used to: arrange the target word units according to the value evaluation scores to obtain a second sorting result; calculate the preset interval according to the total number of target word units and the preset proportional coefficient; filter out the word units in the preset interval from the second sorting result for iterative training of the pre-trained language model until a converged pre-trained language model is obtained; wherein the preset proportional coefficient is gradually reduced as the number of iteration steps increases during the iterative training of the pre-trained language model.

[0083] As an optional implementation provided in an embodiment of the present application, the iteration module is further used to: determine the current iteration step and the total iteration step of the pre-trained language model; calculate a preset proportional coefficient according to the first proportional coefficient boundary value, the second proportional coefficient boundary value, the current iteration step and the total iteration step, the first proportional coefficient boundary value is greater than the second proportional coefficient boundary value, so that the preset proportional coefficient is gradually reduced from the first proportional coefficient boundary value to the second proportional coefficient boundary value during the iterative training process of the pre-trained language model.

[0084] As an optional implementation provided in an embodiment of the present application, the device also includes a feedback module, which is used to: use the converged pre-trained language model to perform downstream tasks, determine the accuracy of the converged pre-trained language model in the downstream tasks; adjust the first proportional coefficient boundary value and the second proportional coefficient boundary value according to the accuracy to perform feedback iteration on the pre-trained language model.

[0085] The description of the features in the embodiment corresponding to the data cleaning device can refer to the relevant description of the embodiment corresponding to the data cleaning method, which will not be repeated here.

[0086] The embodiment of the present application also provides an electronic device, such as Figure 6 The device shown includes a memory 601 and a processor 602 , wherein the memory 601 stores a computer program, and the processor 602 is configured to run the computer program to execute the steps in any of the above data cleaning method embodiments.

[0087] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned data cleaning method embodiments when running.

[0088] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0089] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above data cleaning method embodiments are implemented.

[0090] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned data cleaning method embodiments are implemented.

[0091] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0092] The above is a detailed introduction to a data cleaning method provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A data cleaning method, characterized in that: include: Get the word to be processed; Based on the loss function of the pre-trained language model, calculating the value evaluation score of the word to be processed, wherein the value evaluation score is used to measure the prediction accuracy of different pre-trained language model parameters on the word to be processed; The word units to be processed are arranged according to the value evaluation scores, and target word units in a preset range are obtained by screening.

2. The method according to claim 1, characterized in that The step of obtaining a word to be processed includes: Acquire multimodal data; The multimodal data is segmented and / or encoded to obtain the word-unit to be processed.

3. The method according to claim 1, characterized in that The pre-trained language model includes a first model and a second model, and the performance of the second model is better than that of the first model; The loss function based on the pre-trained language model calculates the value evaluation score of the word to be processed, including: Calculating a first loss function of the word to be processed under a preset context window and a first model parameter, and a second loss function of the word to be processed under the preset context window and a second model parameter; The value assessment score is calculated based on a difference between the second loss function and the first loss function.

4. The method according to claim 3, characterized in that The calculating the value assessment score based on the difference between the second loss function and the first loss function includes: Obtaining the frequency of occurrence of the word to be processed in the corpus; The value assessment score is calculated according to the difference between the second loss function and the first loss function, and the occurrence frequency.

5. The method according to claim 1, characterized in that The step of arranging the word units to be processed according to the value evaluation scores and screening out target word units in a preset interval includes: Clustering the to-be-processed word-grams according to their parts of speech to obtain at least one part-of-speech clustering result; The word units to be processed in the at least one part-of-speech clustering result are arranged according to the value assessment score, and the target word units in the preset preceding interval are obtained by screening according to the preset deletable threshold.

6. The method according to claim 1, characterized in that The step of arranging the word units to be processed according to the value evaluation scores and screening out target word units in a preset interval includes: Arrange the to-be-processed words according to the value evaluation scores to obtain a first sorting result; Calculating the preset interval according to the total number of the word units to be processed and a preset proportional coefficient; The to-be-processed word-units in the preset interval in the first sorting result are used as the target word-units.

7. The method according to claim 1, characterized in that The obtaining of the word to be processed comprises: Dividing the data sample set of the pre-trained language model into multiple data sample subsets; Fine-tune basic model parameters of the pre-trained language model using any subset of data samples to obtain initial model parameters; The word-unit to be processed is obtained from other data sample subsets, where the other data sample subsets are data sample subsets other than any one of the data sample subsets among the multiple data sample subsets.

8. The method according to claim 7, characterized in that After arranging the to-be-processed words according to the value evaluation scores and selecting target words in a preset interval, the method further includes: Using the target word in the preset interval corresponding to the other data sample subset, fine-tuning the initial model parameters to obtain reference model parameters; Calculating a third loss function of the target word under a preset context window and basic model parameters, and a fourth loss function of the target word under a preset context window and reference model parameters; Calculating a value assessment score based on a difference between the fourth loss function and the third loss function; The target word-grams are arranged according to the value assessment scores, and word-grams in a preset interval are screened to be used for iterative training of the pre-trained language model until a converged pre-trained language model is obtained.

9. The method according to claim 8, characterized in that The step of arranging the target word-grams according to the value evaluation scores and screening out word-grams in a preset range includes: Arranging the target word-units according to the value evaluation scores to obtain a second ranking result; Calculating the preset interval according to the total number of the target word units and a preset proportionality coefficient; Filtering out word units in the preset interval from the second sorting results for iterative training of the pre-trained language model until a converged pre-trained language model is obtained; The preset proportionality coefficient is gradually reduced as the number of iteration steps increases during the iterative training of the pre-trained language model.

10. The method according to claim 9, characterized in that After arranging the target word-units according to the value evaluation scores to obtain a second sorting result, and before calculating the preset interval according to the total number of the target word-units and the preset proportional coefficient, the method further includes: Determine the current iteration step number and the total iteration step number of the pre-trained language model; The preset proportional coefficient is calculated according to the first proportional coefficient boundary value, the second proportional coefficient boundary value, the current iteration step number and the total iteration step number, wherein the first proportional coefficient boundary value is greater than the second proportional coefficient boundary value, so that during the iterative training process of the pre-trained language model, the preset proportional coefficient is gradually reduced from the first proportional coefficient boundary value to the second proportional coefficient boundary value.

11. The method according to claim 10, characterized in that After arranging the target word-grams according to the value evaluation scores and screening out word-grams in a preset interval for iterative training of the pre-trained language model until a converged pre-trained language model is obtained, the method further includes: Performing a downstream task using the converged pre-trained language model, and determining the accuracy of the converged pre-trained language model in the downstream task; The first proportional coefficient boundary value and the second proportional coefficient boundary value are adjusted according to the accuracy to perform feedback iteration on the pre-trained language model.

12. A data cleaning device, characterized in that: include: An acquisition module is used to acquire the word to be processed; A calculation module, used to calculate a value evaluation score of a word to be processed based on a loss function of a pre-trained language model, wherein the value evaluation score is used to measure the prediction accuracy of different pre-trained language model parameters on the word to be processed; The screening module is used to arrange the to-be-processed words according to the value evaluation scores, and screen out target words in a preset range.

13. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the data cleaning method according to any one of claims 1 to 11 when executing the computer program.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the data cleaning method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the data cleaning method according to any one of claims 1 to 11 are implemented.

Images