Multi-language data cleaning method, system, equipment and medium

Abstract

The invention relates to a multi-language data cleaning method, system and device and a medium. The method comprises the steps of collecting multi-language data; performing standardized preprocessing on the multi-language data to obtain first data; filtering the first data based on an expert rule to obtain second data; and performing quality evaluation on the second data by adopting an automatic data quality evaluation model based on a multi-language large model, and reserving data of which the accuracy rate of reserved language tags is higher than a set threshold value, the confusion degree is lower than the set threshold value and the model effect is improved after training to obtain a final multi-language data cleaning result. Compared with the prior art, the cleaning quality and cleaning efficiency of the multi-language data are improved.

Description

Technical Field

[0001] The present invention relates to the field of data cleaning, and in particular to a multilingual data cleaning method, system, device and medium. Background Art

[0002] In today's globalization process, cross-language communication and cooperation are becoming increasingly important. Especially in the context of the development of natural language processing (NLP) and large models, improving multilingual processing capabilities has far-reaching significance for promoting the application of technology on a global scale.

[0003] As big models are increasingly used in the field of artificial intelligence, companies and academic institutions around the world are working to develop and improve the multilingual capabilities of big models to help people around the world better use artificial intelligence technology. However, at this stage, big models mainly rely on a large amount of training corpus, most of which is English data, resulting in a serious shortage of data resources in non-English languages. Due to the diversity and complexity of languages, there are still many challenges in building high-quality multilingual datasets for model training.

[0004] In the current practice of multilingual data processing, the sources and processing methods of data mainly include the following categories:

[0005] (1) Multilingual data on the Internet: Public multilingual data is captured from the Internet through web crawlers and other technologies. These data come from a wide range of sources, covering a variety of genres and fields such as news, social media, and academic papers. Due to the diverse sources of data and the uneven quality of data, a lot of preprocessing and cleaning is usually required.

[0006] (2) Translation of English data: A large amount of English data is translated into other languages ​​through translation services (such as Google Translate or DeepL). However, the quality of the translated data depends on the accuracy of the translation system, and translation errors will affect the training effect of the model.

[0007] (3) Manually labeled multilingual data: High-quality multilingual data is collected through manual labeling, which is usually applied to specific fields such as medicine and law. However, manual labeling is costly and cannot be applied on a large scale, which limits the size of the data set.

[0008] After collecting multilingual data, existing technologies generally use rules designed by experts to screen and clean the data. These rules play a certain role in existing multilingual data processing, but because the rules are manually designed by experts, they cannot fully cover all types of corpora and language characteristics. In addition, different types of corpora often have their own unique quality screening standards, and a single rule is difficult to cope with diverse scenarios. For example, specific symbols in some news data may have linguistic significance, but may be regarded as noise in other types of corpora. This reliance on manual rules limits the application effect of existing technologies in multilingual data cleaning.

[0009] Therefore, there is an urgent need to design a more intelligent multilingual data cleaning solution to further improve the cleaning accuracy and quality of multilingual data. Summary of the invention

[0010] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a multilingual data cleaning method, system, equipment and medium with high cleaning quality and high efficiency.

[0011] The purpose of the present invention can be achieved by the following technical solutions:

[0012] According to a first aspect of the present invention, a multilingual data cleaning method is provided, the method comprising:

[0013] Collect multilingual data;

[0014] Performing standardized preprocessing on the multilingual data to obtain first data;

[0015] Performing filtering processing on the first data based on expert rules to obtain second data;

[0016] An automated data quality assessment model based on a large multilingual model is used to assess the quality of the second data. Data with language label accuracy higher than the set threshold, perplexity lower than the set threshold, and improved model effect after training are retained to obtain the final multilingual data cleaning results.

[0017] Preferably, the multilingual data includes multilingual public data from the Internet and manually translated multilingual data.

[0018] Preferably, the standardization preprocessing of the multilingual data includes:

[0019] Standardized language format processing: unify the language codes of all data to a unified standard;

[0020] Deduplication: Convert each piece of data into a unique hash code, use the hash code to determine whether the newly added data is duplicated with the previous data, and remove the duplicate data.

[0021] Preferably, the filtering process based on expert rules on the first data includes word adhesion recognition filtering, image link or HTML tag recognition filtering, invisible character recognition filtering, Chinese garbled character recognition filtering and sensitive word recognition filtering;

[0022] When bilingual data exists in the first data, the following expert rules are added: check whether the number of input lines of the two files is consistent; check whether the corresponding lines in the two files contain lines with empty content; check whether the corresponding lines in the two files contain lines with empty content; check whether the length difference of the corresponding lines in the two files exceeds the set threshold.

[0023] Preferably, the adopting of an automated data quality assessment mechanism based on a multilingual large model to perform quality assessment on the second data comprises:

[0024] Performing language tag detection on the second data, eliminating data in the current batch of data whose language tag matches the actual data language at a rate lower than a set threshold, to obtain third data;

[0025] Calculating the perplexity of the third data, eliminating the data whose perplexity is higher than the set threshold, and obtaining the fourth data;

[0026] The fourth data is used as training data to train a given large language model, and the performance of the trained large language model is verified on a given test set. If the performance of the large language model is improved, the fourth data is used as the final multilingual data cleaning result, otherwise this batch of data is removed.

[0027] Preferably, the perplexity calculation expression is:

[0028]

[0029] Where: P is the perplexity, M is the number of words in the current sentence, p(ω i ) is the i-th word in the current sentence ω i The probability of occurrence.

[0030] Preferably, the fourth data is used as training data to train a given large language model, specifically:

[0031] The fourth data is used as training data, and the causal language model is incrementally trained based on the decode-only Transformer model architecture.

[0032] According to a second aspect of the present invention, a multilingual data cleaning system is provided, the system comprising:

[0033] Data collection module, used to collect multi-language data;

[0034] A preprocessing module, used for performing standardized preprocessing on the multilingual data to obtain first data;

[0035] A filtering module based on expert rules, used for performing filtering processing based on expert rules on the first data to obtain second data;

[0036] The quality assessment and cleaning module is used to use an automated data quality assessment model based on a multilingual large model to perform quality assessment on the second data, retaining data with language label accuracy higher than a set threshold, perplexity lower than a set threshold, and improved model effect after training, to obtain the final multilingual data cleaning results.

[0037] According to a third aspect of the present invention, there is provided an electronic device, comprising a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, any one of the methods described above is implemented.

[0038] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, any one of the methods described above is implemented.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] (1) The present invention uses an automated data quality assessment model based on a multilingual large model to evaluate the data after expert rule filtering, and performs cleaning based on the language label accuracy threshold, perplexity threshold, and whether the trained model has improved the performance in a public multilingual test set, thereby making up for the limitations of a single rule method and improving the quality of data cleaning.

[0041] (2) By using high-quality filtered data in reverse for incremental training of the model, the adaptive optimization of the automated data quality assessment model based on a large multilingual model is achieved, which can gradually improve the multilingual capabilities of the model to meet the quality assessment requirements of different languages ​​and corpus types. Automated evaluation and training reduces dependence on professionals, which not only improves efficiency but also significantly reduces the overall cost of data processing, which has significant economic advantages, especially when facing large-scale multilingual data sets.

[0042] (3) Before using the automated data quality assessment mechanism based on a multilingual large model to assess the data quality, the data is tested for language tags and data with language tags that are different from the actual data language are eliminated, further improving the quality of data cleaning. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flow chart of the method of the present invention;

[0044] Figure 2It is a schematic diagram of the system framework of the present invention;

[0045] Figure 3 Schematic diagram of the cleaning process for quality assessment. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0047] Example

[0048] like Figure 1 As shown, this embodiment provides a multilingual data cleaning method, the method comprising:

[0049] S1. Collect multilingual data;

[0050] S2, performing standardization preprocessing on the multilingual data to obtain first data;

[0051] S3, filtering the first data based on expert rules to obtain second data;

[0052] S4. Use an automated data quality assessment model based on a large multilingual model to assess the quality of the second data, retain the language label accuracy higher than the set threshold, the perplexity lower than the set threshold, and the trained model has improved performance on the public multilingual test set, to obtain the final multilingual data cleaning results.

[0053] Next, the method of this embodiment is introduced in detail.

[0054] S1. Collect multilingual data.

[0055] The sources of multilingual data include obtaining multilingual public data from the Internet, translating English data into corresponding multilingual versions using translation models, and manually translated multilingual data.

[0056] This embodiment collects diverse corpora from multiple sources, such as news, social media, and public papers.

[0057] The corpus includes:

[0058] (1) CCMatrix: CCMatrix is ​​a multilingual parallel corpus developed by the Facebook AI research team. The dataset is extracted from Common Crawl web pages and uses alignment technology to generate parallel sentences between different languages. CCMatrix is ​​a very large dataset covering dozens of languages ​​and is very helpful for machine translation, cross-language retrieval, and multilingual model training. Its special feature is that it uses large-scale network data and achieves high-precision language alignment through machine learning technology.

[0059] (2) OpenSubtitles: OpenSubtitles is a multilingual parallel dataset based on movie and TV series subtitles, which is widely used in machine translation and natural language processing research. Since the content of subtitles covers daily conversations, cultural background, and scene descriptions, this dataset is particularly suitable for the research of dialogue systems and spoken language translation. The OpenSubtitles dataset contains subtitles in hundreds of languages ​​and provides sentence-level alignment information, which is widely used to train and evaluate multilingual models.

[0060] (3) MultiCCAligned: MultiCCAligned is a multilingual parallel dataset extracted and aligned from Common Crawl data, similar to CCMatrix. This dataset covers many languages ​​and the data comes from global web pages. MultiCCAligned provides cross-lingual sentence pairs and is an important resource for training and evaluating multilingual natural language processing models. Its construction goal is to provide large-scale and accurately aligned multilingual text to support research and development in translation, language modeling, and cross-lingual applications.

[0061] S2. Perform standardization preprocessing on the multilingual data to obtain first data.

[0062] (1) Standardized language format processing: Unify the language codes of all data to a unified standard.

[0063] Since the data comes from Internet corpora from different sources, some languages ​​have multiple corresponding language codes. One reason is that different corpora use different language code standards, including ISO 639-1, ISO 639-2, ISO 639-3 or custom language codes. Another reason is that some corpora append a regional ID to the end of the language code to distinguish the usage of the same language in different regions. In order to unify the language codes, if a language has corresponding language codes in ISO 639-1, ISO 639-2 and ISO 639-3, in this embodiment, the language codes of ISO 639-2 and ISO 639-3 are replaced with the language code of ISO 639-1.

[0064] (2) Deduplication: Each piece of data is converted into a unique hash code. The hash code is used to determine whether the newly added data is duplicated with the previous data, and the duplicate data is eliminated.

[0065] S3. Perform filtering processing on the first data based on expert rules to obtain second data.

[0066] In this embodiment, the filtering process based on expert rules includes word adhesion recognition filtering, image link or HTML tag recognition filtering, invisible character recognition filtering, Chinese garbled character recognition filtering and sensitive word recognition filtering, specifically:

[0067] (1) Word adhesion detection

[0068] Check if the number of characters in a word exceeds a set threshold;

[0069] Ignore URLs, file paths, and hexadecimal numbers;

[0070] Use English word splitter to split long words;

[0071] (2) Detect image links or HTML tags

[0072] If a picture link or HTML tag is detected, such as , or <o:p>, then the content is marked as irrelevant.

[0073] (3) Detecting invisible characters

[0074] Detects the presence of invisible Unicode characters in the content, such as special spaces or control characters, and marks the content as irrelevant if these characters are present.

[0075] (4) Detection of Chinese garbled characters

[0076] Detect garbled content in Chinese text;

[0077] Remove unwanted patterns and convert traditional characters to simplified characters.

[0078] Then check the word segmentation quality

[0079] (5) Sensitive word detection

[0080] Regularly update the multilingual sensitive word list.

[0081] If it is bilingual data, add the following rules:

[0082] 1) Check whether the number of input lines of the two files is consistent;

[0083] 2) Check whether the corresponding lines in the two files contain empty lines;

[0084] 3) Check whether the corresponding lines in the two files contain lines with the content "nan";

[0085] 4) Check whether the lengths of corresponding lines in the two files differ too much.

[0086] S4. Use an automated data quality assessment model based on a multilingual large model to assess the quality of the second data, retain the data with language label accuracy higher than the set threshold, perplexity lower than the set threshold, and improved model effect after training, and obtain the final multilingual data cleaning results, such as Figure 3 As shown, specifically including:

[0087] S41, performing language tag detection on the second data, eliminating data in the current batch of data whose language tag matches the actual data language at a rate lower than a set threshold, to obtain third data;

[0088] Since language tags of data on the Internet are often misspelled, especially between similar languages ​​(sharing some vocabulary), language tag detection is performed on the second data to eliminate data whose language tags are different from the actual data language. Specifically, tools such as fasttext can be used to detect whether the language tag is correct or to translate it into a specified language.

[0089] For data in a batch, the number of correct matches of its language tags should reach more than 90% (some data, such as code data, will contain some English content due to its characteristics, so it is not necessary to achieve 100% matching), and the batch of data with a language tag matching rate of less than 90% with the actual data language will be eliminated.

[0090] S42, calculating the perplexity of the third data, eliminating the data with perplexity higher than a set threshold, and obtaining the fourth data.

[0091] Perplexity is a commonly used indicator in the field of natural language processing to measure the prediction ability of a model, and is particularly suitable for language models.

[0092] Low perplexity indicates high-quality data: when the text in the data is clearly structured, well-annotated, and noise-free, the model is better able to learn and make accurate predictions. Therefore, lower perplexity indicates higher-quality data and the model is more certain about the language patterns in the data.

[0093] High perplexity may mean poor data quality: If the perplexity is high, it may indicate that the data is noisy, inconsistently labeled, or has a messy text structure, which makes it difficult for the model to learn language patterns. For example, if the data contains many grammatical errors, spelling errors, or randomly generated nonsense text, the perplexity will tend to rise, indicating poor data quality.

[0094] In this embodiment, the perplexity calculation expression is:

[0095]

[0096] Where: P is the perplexity, M is the number of words in the current sentence, p(ω i ) is the i-th word in the current sentence ω i In this embodiment, data with perplexity P>30 are eliminated.

[0097] S43: Use the fourth data as training data to train a given large language model, and perform performance verification on the trained large language model on a given test set. If the performance of the large language model is improved, use the fourth data as the final multilingual data cleaning result; otherwise, remove this batch of data.

[0098] In this embodiment, for the automated data quality assessment model based on a multilingual large model, high-quality data with language label accuracy higher than a set threshold and perplexity lower than a set threshold is used as training data in reverse, and the causal language model is incrementally trained based on the decode-only Transformer model architecture. The causal language model refers to predicting the next word of a text based on the content of the previous text.

[0099] If the model trained with this batch of data shows improved performance on the industry's publicly available multi-language test sets (such as FLORES101 and XNLI) after training, the data will be retained.

[0100] like Figure 2 As shown, this embodiment also provides a multilingual data cleaning system, the system comprising:

[0101] Data collection module, used to collect multi-language data;

[0102] A preprocessing module, used for performing standardized preprocessing on the multilingual data to obtain first data;

[0103] A filtering module based on expert rules, used for performing filtering processing based on expert rules on the first data to obtain second data;

[0104] The quality assessment and cleaning module is used to use an automated data quality assessment model based on a multilingual large model to perform quality assessment on the second data, retaining data with language label accuracy higher than a set threshold, perplexity lower than a set threshold, and improved model effect after training, to obtain the final multilingual data cleaning results.

[0105] Modularizing multilingual data processing reduces the frequency of manual rule maintenance and reduces reliance on professionals through automated model evaluation and loop training. Compared with existing technical solutions, this not only improves efficiency, but also significantly reduces the overall cost of data processing, especially when facing large-scale multilingual data sets, which has significant economic advantages.

[0106] The electronic device of the present invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0107] Multiple components in the device are connected to the I / O interface, including: input units, such as keyboards, mice, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as disks, optical disks, etc.; and communication units, such as network cards, modems, wireless communication transceivers, etc. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunication networks.

[0108] The processing unit performs the various methods and processes described above, such as methods S1 to S4. For example, in some embodiments, methods S1 to S7 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into the RAM and executed by the CPU, one or more steps of methods S1 to S4 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S1 to S4 in any other appropriate manner (e.g., by means of firmware).

[0109] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip systems (SOCs), load programmable logic devices (CPLDs), and the like.

[0110] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0111] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0112] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.< / o:p>

Claims

1. A multilingual data cleaning method, characterized in that: The method includes: Collect multilingual data; Performing standardized preprocessing on the multilingual data to obtain first data; Performing filtering processing on the first data based on expert rules to obtain second data; An automated data quality assessment model based on a large multilingual model is used to assess the quality of the second data. Data with language label accuracy higher than the set threshold, perplexity lower than the set threshold, and improved model effect after training are retained to obtain the final multilingual data cleaning results.

2. A multilingual data cleaning method according to claim 1, characterized in that: The multilingual data includes multilingual public data from the Internet and multilingual data translated manually.

3. A multilingual data cleaning method according to claim 1, characterized in that: The standardized preprocessing of multilingual data includes: Standardized language format processing: unify the language codes of all data to a unified standard; Deduplication: Convert each piece of data into a unique hash code, use the hash code to determine whether the newly added data is duplicated with the previous data, and remove the duplicate data.

4. A multilingual data cleaning method according to claim 1, characterized in that: The filtering process based on expert rules on the first data includes word adhesion recognition filtering, image link or HTML tag recognition filtering, invisible character recognition filtering, Chinese garbled character recognition filtering and sensitive word recognition filtering; When bilingual data exists in the first data, the following expert rules are added: check whether the number of input lines of the two files is consistent; check whether the corresponding lines in the two files contain lines with empty content; check whether the corresponding lines in the two files contain lines with empty content; check whether the length difference of the corresponding lines in the two files exceeds the set threshold.

5. The multilingual data cleaning method according to claim 1, characterized in that: The method of using an automated data quality assessment mechanism based on a multilingual large model to perform quality assessment on the second data includes: Performing language tag detection on the second data, eliminating data in the current batch of data whose language tag matches the actual data language at a rate lower than a set threshold, to obtain third data; Calculating the perplexity of the third data, eliminating the data whose perplexity is higher than the set threshold, and obtaining the fourth data; The fourth data is used as training data to train a given large language model, and the performance of the trained large language model is verified on a given test set. If the performance of the large language model is improved, the fourth data is used as the final multilingual data cleaning result, otherwise this batch of data is removed.

6. A multilingual data cleaning method according to claim 5, characterized in that: The perplexity calculation expression is: Where: P is the perplexity, M is the number of words in the current sentence, p(ω i ) is the i-th word in the current sentence i The probability of occurrence.

7. A multilingual data cleaning method according to claim 5, characterized in that: The fourth data is used as training data to train the given large language model, specifically: The fourth data is used as training data, and the causal language model is incrementally trained based on the decode-only Transformer model architecture.

8. A system based on the multilingual data cleaning method according to claim 1, characterized in that: The system comprises: Data collection module, used to collect multi-language data; A preprocessing module, used for performing standardized preprocessing on the multilingual data to obtain first data; A filtering module based on expert rules, used for performing filtering processing based on expert rules on the first data to obtain second data; The quality assessment and cleaning module is used to use an automated data quality assessment model based on a multilingual large model to perform quality assessment on the second data, retaining data with language label accuracy higher than a set threshold, perplexity lower than a set threshold, and improved model effect after training, to obtain the final multilingual data cleaning results.

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images