Corpus cleaning methods, systems, media and terminals

Abstract

The present invention provides a corpus cleaning method, system, medium, and terminal. The method includes: obtaining original corpus to be cleaned, pre-annotating the original corpus to generate pre-annotated corpus, parsing the pre-annotated corpus to generate first parsing information; optimizing the original corpus to obtain optimized corpus, parsing the optimized corpus to generate second parsing information; enhancing the original corpus to obtain enhanced corpus, parsing the enhanced corpus to generate third parsing information; generating target parsing information based on the first parsing information, the second parsing information, and the third parsing information, generating corresponding synthetic corpus based on the target parsing information; desensitizing the synthetic corpus based on a desensitization library to generate corresponding desensitized corpus, and standardizing the desensitized corpus to generate corresponding target corpus. The present invention effectively improves the efficiency and quality of corpus cleaning.

Description

Technical Field

[0001] The present invention belongs to the field of big data technology, and in particular relates to a corpus cleaning method, system, medium and terminal. Background Art

[0002] Large AI models are a type of AI model with a large number of parameters, constructed using artificial neural networks. They are typically pre-trained on massive amounts of data through self-supervised or semi-supervised learning. Their performance and capabilities are then further optimized through methods such as instruction fine-tuning and human alignment. Corpus is a crucial material used to train large AI models. Corpus generally refers to examples and datasets used in linguistic research and natural language processing. Corpus can be written text, spoken language recordings, or other structured data, and is typically used to analyze linguistic phenomena and support tasks such as machine translation, speech recognition, and automatic text summarization. Corpus is typically a set of collected, organized, and annotated text or speech data. In linguistic research, this data is used to analyze language usage patterns, vocabulary variation, and grammatical structure. In natural language processing, corpus is the fundamental data source for training and testing models, supporting functions such as machine translation, speech recognition, and sentiment analysis.

[0003] Because traditional corpora are complex, non-standard and diverse, they generally need to be cleaned before large-scale model training. Traditional cleaning methods mostly adopt a single processing method, which may cause the corpus to fail to meet the standards after cleaning or cause the content of the corpus to be inconsistent after cleaning, affecting the quality of the corpus and being unfavorable for large-scale model training. Summary of the Invention

[0004] In view of the above-mentioned shortcomings of the prior art, the object of the present invention is to provide a corpus cleaning method, system, medium and terminal for solving the problem of low efficiency of corpus cleaning in the prior art.

[0005] To achieve the above and other related purposes, the present invention provides a corpus cleaning method, comprising the following steps:

[0006] Acquiring original corpus to be cleaned, performing pre-annotation processing on the original corpus to generate pre-annotated corpus, and parsing the pre-annotated corpus to generate first parsing information;

[0007] Optimizing the original corpus to obtain optimized corpus, and parsing the optimized corpus to generate second parsing information;

[0008] performing enhancement processing on the original corpus to obtain enhanced corpus, and parsing the enhanced corpus to generate third parsing information;

[0009] generating target parsing information according to the first parsing information, the second parsing information, and the third parsing information, and generating corresponding synthetic corpus according to the target parsing information;

[0010] The synthetic corpus is desensitized according to the desensitization library to generate corresponding desensitized corpus, and the desensitized corpus is standardized to generate corresponding target corpus.

[0011] In one embodiment of the present invention, pre-annotating the original corpus to generate pre-annotated corpus, and parsing the pre-annotated corpus to generate first parsing information includes:

[0012] Labeling the original corpus to generate a tree structure, wherein the tree structure includes a plurality of structural blocks, each structural block corresponding to a field;

[0013] Copying the original corpus multiple times to obtain multiple backup corpora, and performing hierarchical annotation on each of the backup corpora according to the tree structure according to a preset rule to obtain multiple backup annotation information;

[0014] Performing differential comparison on the plurality of backup annotation information, and selecting the annotation word with the highest occurrence frequency at each position of the tree structure in turn as the target annotation word, and combining the plurality of target annotation words in order to form the pre-annotated corpus;

[0015] The pre-annotated corpus is parsed by a first parsing tool to obtain the first parsing information.

[0016] In one embodiment of the present invention, performing differential comparison on the plurality of backup annotation information, sequentially selecting the annotation word with the highest frequency of occurrence at each position in the tree structure as the target annotation word, and sequentially combining the plurality of target annotation words to form the pre-annotated corpus includes:

[0017] Determining a structural block corresponding to each field in the original corpus according to the tree structure;

[0018] At the position of each structural block, respectively calculate the occurrence frequency of the annotation words of each backup annotation information in the current structural block, and select the annotation word corresponding to the maximum occurrence frequency as the target annotation word;

[0019] According to the order of the structural blocks in the tree structure, the target annotated words are sequentially combined together to form the pre-annotated corpus.

[0020] In one embodiment of the present invention, the step of optimizing the original corpus to obtain an optimized corpus, and parsing the optimized corpus to generate second parsing information includes:

[0021] A hash algorithm is used to calculate the data unique identifier of the original corpus and the semantic similarity in the original corpus, and duplicate content and similar content are removed according to the data unique identifier and the semantic similarity to obtain a deduplicated corpus;

[0022] After parsing the deduplicated corpus through a syntax tree to remove logical errors, the marketing content is filtered based on a rule template;

[0023] The optimized corpus is obtained after removing existing URL characters, special characters and meaningless symbols;

[0024] The optimized corpus is parsed by a second parsing tool to obtain the second parsing information.

[0025] In one embodiment of the present invention, the enhancing the original corpus to obtain enhanced corpus, and parsing the enhanced corpus to generate third parsing information, include:

[0026] Determining the non-core fields corresponding to the original corpus according to the tree structure;

[0027] After replacing the non-core fields based on the domain vocabulary, a replacement corpus is generated;

[0028] Calling a multilingual API combination loop to translate the replacement corpus to generate the enhanced corpus;

[0029] The enhanced corpus is parsed using a third parsing tool to obtain the third parsing information.

[0030] In one embodiment of the present invention, generating target parsing information according to the first parsing information, the second parsing information, and the third parsing information, and generating corresponding synthetic corpus according to the target parsing information includes:

[0031] Splitting the first parsed information, the second parsed information, and the third parsed information according to the tree structure to obtain first information segments, second information segments, and third information segments at corresponding positions;

[0032] Calculate the information segment mean of each position according to the first information segment, the second information segment, and the third information segment;

[0033] Calculate the difference coefficient between the first information segment, the second information segment and the third information segment at each position and the mean of the information segments at the corresponding position respectively;

[0034] selecting the first information segment, the second information segment, and the third information segment with the smallest difference coefficient as the target information segment;

[0035] The target information segments at various positions are arranged and combined in sequence to form the target parsed information, and the corresponding synthetic corpus is generated according to the target parsed information.

[0036] In one embodiment of the present invention, performing desensitization processing on the synthetic corpus according to the desensitization library to generate corresponding desensitized corpus, and performing standardization processing on the desensitized corpus to generate corresponding target corpus includes:

[0037] Comparing the synthesized corpus with the desensitization library to determine sensitive words therein, and replacing the sensitive words with encrypted words to generate the desensitized corpus;

[0038] After the desensitized corpus is subjected to format unification, word form normalization processing and numerical standardization processing respectively, the corresponding target corpus is generated.

[0039] The present invention also discloses a corpus cleaning system, comprising:

[0040] A first parsing module is configured to obtain an original corpus to be cleaned, perform pre-annotation processing on the original corpus to generate a pre-annotated corpus, and parse the pre-annotated corpus to generate first parsing information;

[0041] A second parsing module is configured to perform corpus optimization on the original corpus to obtain an optimized corpus, and parse the optimized corpus to generate second parsing information;

[0042] a third parsing module, configured to perform enhancement processing on the original corpus to obtain enhanced corpus, and parse the enhanced corpus to generate third parsing information;

[0043] a synthesis module, configured to generate target parsing information according to the first parsing information, the second parsing information, and the third parsing information, and generate corresponding synthesized corpus according to the target parsing information;

[0044] The processing module is used to perform desensitization processing on the synthetic corpus according to the desensitization library to generate corresponding desensitized corpus, and to generate corresponding target corpus after performing standardization processing on the desensitized corpus.

[0045] The present invention provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned corpus cleaning method is implemented.

[0046] The present invention provides a terminal, comprising: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the computer program stored in the memory, so that the terminal executes the above-mentioned corpus cleaning method.

[0047] As described above, the corpus cleaning method, system, medium, and terminal of the present invention have the following beneficial effects:

[0048] The present invention performs pre-annotation processing, corpus optimization processing and enhancement processing on the original corpus to be cleaned and parses it to obtain first parsing information, second parsing information and third parsing information, cleans and optimizes the corpus from different directions, and subsequently generates target parsing information according to the first parsing information, the second parsing information and the third parsing information, and generates processed synthetic corpus according to the target parsing information, and finally obtains the target corpus through desensitization and standardization processing, thereby optimizing the original corpus from three levels: corpus annotation, corpus optimization and corpus enhancement, and finally selecting the optimal target corpus, thereby effectively realizing the cleaning of the original corpus, ensuring the efficiency of corpus cleaning, improving the quality of the original corpus, and improving the training efficiency of the large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Shown is a flow chart of corpus cleaning in one embodiment of the present invention.

[0050] Figure 2 It is a schematic diagram showing the specific process of step S400 in the corpus cleaning of the present invention.

[0051] Figure 3 Shown is a structural block diagram of a corpus cleaning system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following describes the embodiments of the present invention through specific embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0053] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. The illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0054] The corpus cleaning method, system, medium and terminal of the present invention perform pre-annotation processing, corpus optimization processing and enhancement processing on the original corpus to be cleaned and parse it to obtain first parsing information, second parsing information and third parsing information, clean and optimize the corpus from different directions, and subsequently generate target parsing information based on the first parsing information, the second parsing information and the third parsing information, and generate processed synthetic corpus based on the target parsing information, and finally obtain the target corpus through desensitization and standardization processing, thereby optimizing the original corpus from the three levels of corpus annotation, corpus optimization and corpus enhancement at the same time, and finally selecting the optimal target corpus, thereby effectively realizing the cleaning of the original corpus, ensuring the efficiency of corpus cleaning, improving the quality of the original corpus, and improving the training efficiency of the large model.

[0055] The storage medium of the present invention stores a computer program that, when executed by a processor, implements the following corpus cleaning method. The storage medium includes various media capable of storing program code, such as read-only memory (ROM), random access memory (RAM), a magnetic disk, a USB flash drive, a memory card, or an optical disk.

[0056] Any combination of one or more storage media may be used. The storage medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, RAM, ROM, an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in connection with an instruction execution system, apparatus, or device.

[0057] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0058] Program code embodied on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0059] Computer program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0060] The present invention will be described below with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, as well as combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine such that, when executed by the processor of the computer or other programmable data processing device, these computer program instructions generate a device that implements the functions / actions specified in one or more blocks in the flowcharts and / or block diagrams.

[0061] These computer program instructions may also be stored in a computer-readable medium, which causes a computer, other programmable data processing apparatus, or other device to operate in a specific manner, so that the instructions stored in the computer-readable medium produce an article of manufacture that includes instructions for implementing the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0062] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, such that the instructions executed on the computer or other programmable apparatus provide a process for implementing the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0063] The terminal of the present invention includes a processor and a memory.

[0064] The memory is used to store computer programs; preferably, the memory includes: ROM, RAM, magnetic disk, USB flash drive, memory card or optical disk and other media that can store program codes.

[0065] The processor is connected to the memory and is used to execute the computer program stored in the memory so that the terminal performs the following corpus cleaning method.

[0066] Preferably, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0067] like Figure 1 As shown, in one embodiment, the present invention discloses a corpus cleaning method, comprising the following steps:

[0068] S100: Acquire original corpus to be cleaned, perform pre-annotation processing on the original corpus to generate pre-annotated corpus, and parse the pre-annotated corpus to generate first parsing information.

[0069] In some embodiments, pre-annotating the original corpus to generate pre-annotated corpus, and parsing the pre-annotated corpus to generate first parsing information includes:

[0070] Labeling the original corpus to generate a tree structure, wherein the tree structure includes a plurality of structural blocks, each structural block corresponding to a field;

[0071] Copying the original corpus multiple times to obtain multiple backup corpora, and performing hierarchical annotation on each of the backup corpora according to the tree structure according to a preset rule to obtain multiple backup annotation information;

[0072] Performing differential comparison on the plurality of backup annotation information, and selecting the annotation word with the highest occurrence frequency at each position of the tree structure in turn as the target annotation word, and combining the plurality of target annotation words in order to form the pre-annotated corpus;

[0073] The pre-annotated corpus is parsed by a first parsing tool to obtain the first parsing information.

[0074] In this embodiment, the original corpus to be cleaned is first obtained, and then the original corpus is pre-annotated to obtain pre-annotated corpus, so that the pre-annotated corpus can be parsed to generate first parsing information, which is convenient for subsequent synthesis and cleaning of the original corpus based on the first parsing information to improve the quality of the original corpus.

[0075] Specifically, the original corpus is first labeled and graded according to its structure to generate a tree structure corresponding to the original corpus. The tree structure includes multiple structural blocks, each structural block corresponds to a field, and the fields include core fields and non-core fields. The structural blocks corresponding to the core fields serve as the trunk of the tree structure, and the structural blocks corresponding to the non-core fields serve as the branches of the structural blocks corresponding to the core fields.

[0076] The original corpus is then copied multiple times to obtain multiple backup corpora. Each backup corpus is hierarchically annotated according to a tree structure in accordance with preset annotation rules to obtain corresponding multiple backup annotation information. The multiple backup annotation information is then differentially compared, and the annotation words with the highest frequency of occurrence at each position in the tree structure are selected as target annotation words. These words are then combined in the order of the tree structure to form the pre-annotated corpus. The pre-annotated corpus is then parsed using a first parsing tool to obtain first parsed information, which facilitates subsequent cleaning of the original corpus based on the first parsed information.

[0077] Before the hierarchical annotation of each backup corpus according to the tree structure, the method further includes parsing the logical errors in the backup corpus according to the syntax tree and filtering out auxiliary words and interjections to ensure the accuracy of subsequent annotation results.

[0078] In the above process, the original corpus is copied to obtain multiple backup corpora, and then each backup corpus is graded and annotated according to the tree structure to obtain multiple backup annotation information, and then the multiple backup annotation information are differentially compared, and the annotation words with the highest frequency of occurrence are selected in turn as the target annotation words, and then the target annotation words at each position are combined together to form the final pre-annotated corpus. Compared with the traditional annotation method, the present application scheme ensures the accuracy of the pre-annotated corpus by performing multiple annotation processes on the original corpus and selecting the optimal annotation words to form a complete pre-annotated corpus, which is conducive to improving the quality of corpus cleaning.

[0079] In some other embodiments, performing differential comparison on the plurality of backup annotation information, sequentially selecting the annotation word with the highest occurrence frequency at each position of the tree structure as the target annotation word, and sequentially combining the plurality of target annotation words to form the pre-annotated corpus includes:

[0080] Determining a structural block corresponding to each field in the original corpus according to the tree structure;

[0081] At the position of each structural block, respectively calculate the occurrence frequency of the annotation words of each backup annotation information in the current structural block, and select the annotation word corresponding to the maximum occurrence frequency as the target annotation word;

[0082] According to the order of the structural blocks in the tree structure, the target annotated words are sequentially combined together to form the pre-annotated corpus.

[0083] In this embodiment, after hierarchically annotating the plurality of backup corpora to obtain a plurality of backup annotation information, the structural block corresponding to each field in the original corpus is first determined according to the tree structure. At the position of each structural block, the occurrence frequency of the annotation words of the plurality of backup annotation information in the current structural block is calculated respectively, and the annotation word with the largest occurrence frequency is selected as the target annotation word corresponding to the current structural block. After calculating and selecting one by one according to the order of the tree structure, the target annotation word corresponding to each structural block is obtained. Then, according to the order of the structural blocks in the tree structure, the target annotation words corresponding to the structural blocks are sequentially combined together to form the pre-annotated corpus.

[0084] The occurrence frequency is the ratio of the number of occurrences of the target word at the same position in the tree structure to the total number of the backup corpus.

[0085] S200: Optimize the original corpus to obtain optimized corpus, and parse the optimized corpus to generate second parsing information.

[0086] In some embodiments, the performing corpus optimization on the original corpus to obtain an optimized corpus, and parsing the optimized corpus to generate second parsing information, includes:

[0087] A hash algorithm is used to calculate the data unique identifier of the original corpus and the semantic similarity in the original corpus, and duplicate content and similar content are removed according to the data unique identifier and the semantic similarity to obtain a deduplicated corpus;

[0088] After parsing the deduplicated corpus through a syntax tree to remove logical errors, the marketing content is filtered based on a rule template;

[0089] The optimized corpus is obtained after removing existing URL characters, special characters and meaningless symbols;

[0090] The optimized corpus is parsed by a second parsing tool to obtain the second parsing information.

[0091] In this embodiment, in order to optimize the original corpus, a unique data identifier corresponding to each character in the original corpus is first calculated using a hash algorithm, and the semantic similarity of the characters in the original corpus is calculated using a MinHash algorithm or a SimHash algorithm. Duplicate content is then removed based on the unique data identifier, and similar content is removed based on the semantic similarity to obtain the deduplicated corpus. The deduplicated corpus is then parsed through syntax tree parsing to remove logical errors, and the marketing content of the deduplicated corpus is filtered based on a rule template. The URL characters, special characters, and meaningless symbols present in the deduplicated corpus are then removed to obtain the final optimized corpus. The optimized corpus is then semantically parsed using a second parsing tool to obtain second parsing information.

[0092] S300: Perform enhancement processing on the original corpus to obtain enhanced corpus, and parse the enhanced corpus to generate third parsing information.

[0093] In some other embodiments, the performing enhancement processing on the original corpus to obtain enhanced corpus, and parsing the enhanced corpus to generate third parsing information includes:

[0094] Determining the non-core fields corresponding to the original corpus according to the tree structure;

[0095] After replacing the non-core fields based on the domain vocabulary, a replacement corpus is generated;

[0096] Calling a multilingual API combination loop to translate the replacement corpus to generate the enhanced corpus;

[0097] The enhanced corpus is parsed using a third parsing tool to obtain the third parsing information.

[0098] In this embodiment, the non-core fields corresponding to the original corpus are first determined based on the tree structure, that is, the fields corresponding to the structural blocks of the branch parts corresponding to the trunk parts in the tree structure. After the non-core fields are replaced based on the domain vocabulary, a replacement corpus is generated, thereby ensuring that the original corpus is replaced while retaining the original semantics. At the same time, the multilingual API combination loop is called to translate the replacement corpus to generate the enhanced corpus. For example, if the original corpus is in Chinese, the multilingual API is called to translate the replacement corpus into English, then into Japanese, and then into the original Chinese, to produce a new sample whose semantics are consistent with those of the original corpus, that is, the enhanced corpus. Then, based on the obtained enhanced corpus, the third parsing tool is called to parse the enhanced corpus to obtain the third parsing information.

[0099] It should be noted that, in the aforementioned parsing processes, the parsing processes of the first parsing tool, the second parsing tool and the third parsing tool are the contents of the existing technology, which mainly obtain the semantic information of the corpus through parsing, and the selection of the first parsing tool, the second parsing tool and the third parsing tool can also be selected according to actual needs. This solution does not limit the tool itself and will not be repeated here.

[0100] S400: Generate target parsing information according to the first parsing information, the second parsing information, and the third parsing information, and generate corresponding synthetic corpus according to the target parsing information.

[0101] In some embodiments, reference Figure 2 The step of generating target parsing information according to the first parsing information, the second parsing information, and the third parsing information, and generating corresponding synthetic corpus according to the target parsing information includes:

[0102] S401: Split the first parsed information, the second parsed information, and the third parsed information according to the tree structure to obtain first information segments, second information segments, and third information segments at corresponding positions;

[0103] S402: Calculate the information segment mean at each position based on the first information segment, the second information segment, and the third information segment;

[0104] S403, respectively calculating the difference coefficient between the first information segment, the second information segment, and the third information segment at each position and the mean of the information segments at the corresponding position;

[0105] S404: Select the one with the smallest difference coefficient among the first information segment, the second information segment, and the third information segment as a target information segment;

[0106] S405: Arrange and combine the target information segments at various positions in order to form the target parsed information, and generate the corresponding synthetic corpus according to the target parsed information.

[0107] In this embodiment, after obtaining the first, second, and third parsed information corresponding to the original corpus, the first, second, and third parsed information are independently split according to the tree structure of the original corpus to obtain a plurality of first, second, and third information segments, respectively, equal to the number of the structural blocks. The mean value of the information segment corresponding to each structural block is then calculated based on the first, second, and third information segments. The coefficient of difference between the first, second, and third information segments at each structural block and the mean value of the information segment at the corresponding position is calculated. The information segment at each structural block position that results in the minimum coefficient of difference is then selected as the target information segment, thereby obtaining a number of target information segments equal to the number of the structural blocks. The target information segments are then sequentially arranged and combined according to the order of the structural blocks in the tree structure to form the target parsed information. Synthesized corpus is generated based on the target parsed information, thereby converting the corpus according to the parsed information and ensuring the quality of the resulting synthesized corpus.

[0108] It should be noted that, in the above-mentioned process of calculating the difference coefficient, in order to facilitate calculation, the first information segment, the second information segment, and the third information segment corresponding to each structural block are converted into corresponding coding values, such as ASCII coding, GB2312, GBK, and Unicode coding. When calculating the difference coefficient, the information segment mean corresponding to the current structural block is calculated by using the coding values ​​corresponding to the first information segment, the second information segment, and the third information segment respectively. The difference coefficient can be obtained by calculating the ratio of the absolute value of the difference between the coding values ​​corresponding to the first information segment, the second information segment, and the third information segment and the information segment mean to the information segment mean.

[0109] Exemplarily, the difference coefficient satisfies the following calculation formula:

[0110] , i is an integer from 1 to 3.

[0111] in, represents the coefficient of variation, represents the mean value of the information segment, represents the encoding value corresponding to the first information segment, The encoding value corresponding to the second information segment, The encoding value corresponding to the third information segment.

[0112] S500: Desensitizing the synthesized corpus according to the desensitization library to generate corresponding desensitized corpus, and normalizing the desensitized corpus to generate corresponding target corpus.

[0113] In some embodiments, performing desensitization processing on the synthetic corpus according to the desensitization library to generate corresponding desensitized corpus, and performing standardization processing on the desensitized corpus to generate corresponding target corpus, includes:

[0114] Comparing the synthesized corpus with the desensitization library to determine sensitive words therein, and replacing the sensitive words with encrypted words to generate the desensitized corpus;

[0115] After the desensitized corpus is subjected to format unification, word form normalization processing and numerical standardization processing respectively, the corresponding target corpus is generated.

[0116] In an embodiment, after obtaining the synthetic corpus, the synthetic corpus is desensitized according to the desensitization library, and the sensitive words are replaced with encrypted words to generate desensitized corpus. After the desensitized corpus is format-unified, magnetically normalized, and numerically normalized, the corresponding target corpus is generated, thereby completing the corpus cleaning work.

[0117] Among them, word form normalization processing is mainly aimed at text data in synthetic corpus, and numerical standardization processing is aimed at numerical data in synthetic corpus. Since word form normalization processing and numerical standardization processing are the contents of existing technologies, this solution does not make special limitations on them and will not be repeated here.

[0118] It should be noted that the protection scope of the method described in the present invention is not limited to the execution order of the steps listed in this embodiment. All solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present invention are included in the protection scope of the present invention.

[0119] The present invention also provides a corpus cleaning system, referring to FIG. 3 , comprising:

[0120] A first parsing module 301 is configured to obtain an original corpus to be cleaned, perform pre-annotation processing on the original corpus to generate pre-annotated corpus, and parse the pre-annotated corpus to generate first parsing information;

[0121] A second parsing module 302 is configured to perform corpus optimization on the original corpus to obtain an optimized corpus, and parse the optimized corpus to generate second parsing information;

[0122] A third parsing module 303 is configured to perform enhancement processing on the original corpus to obtain enhanced corpus, and parse the enhanced corpus to generate third parsing information;

[0123] a synthesis module 304, configured to generate target parsing information according to the first parsing information, the second parsing information, and the third parsing information, and generate corresponding synthesized corpus according to the target parsing information;

[0124] The processing module 305 is configured to perform desensitization processing on the synthetic corpus according to the desensitization library to generate corresponding desensitized corpus, and perform standardization processing on the desensitized corpus to generate corresponding target corpus.

[0125] Among them, since the structure and principles of the corpus cleaning system correspond one-to-one to the steps in the above-mentioned corpus cleaning method, they will not be repeated here.

[0126] It should be understood that the division of the modules in the above system is merely a division of logical functions. In actual implementation, they may be fully or partially integrated into a single physical entity or physically separated. Furthermore, these modules may be implemented entirely in software called by a processing element, or entirely in hardware. Alternatively, some modules may be implemented in software called by a processing element, while others may be implemented in hardware. For example, module x may be a separate processing element, or integrated into a chip in the above system. Furthermore, it may be stored in the form of program code in the memory of the above system, called by a processing element in the system to perform the functions of module x. The implementation of other modules is similar. Furthermore, these modules may be fully or partially integrated or implemented independently. The processing element described herein may be an integrated circuit with signal processing capabilities. During implementation, the steps of the above method or the modules above may be performed by hardware integrated logic circuits in the processor element or by software instructions.

[0127] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented by scheduling program code on a processing element, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor capable of calling program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0128] It should be noted that the corpus cleaning system of the present invention can implement the method of the present invention, but the implementation device of the corpus cleaning method of the present invention includes but is not limited to the structure of the corpus cleaning system listed in this embodiment. All structural deformations and replacements of the existing technology made according to the principles of the present invention are included in the protection scope of the present invention.

[0129] The present invention discloses a storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned corpus cleaning method is realized.

[0130] The present invention also discloses a terminal, comprising: a processor and a memory;

[0131] The memory is used to store computer programs;

[0132] The processor is used to execute the computer program stored in the memory, so that the terminal performs the above-mentioned corpus cleaning method.

[0133] In summary, the corpus cleaning method, system, medium and terminal of the present invention perform pre-annotation processing, corpus optimization processing and enhancement processing on the original corpus to be cleaned and parse to obtain first parsing information, second parsing information and third parsing information, clean and optimize the corpus from different directions, and subsequently generate target parsing information according to the first parsing information, the second parsing information and the third parsing information, and generate processed synthetic corpus according to the target parsing information, and finally obtain the target corpus through desensitization and standardization processing, thereby optimizing the original corpus from the three levels of corpus annotation, corpus optimization and corpus enhancement at the same time, and finally select the optimal target corpus, thereby effectively realizing the cleaning of the original corpus, ensuring the efficiency of corpus cleaning, improving the quality of the original corpus, and improving the training efficiency of the large model; therefore, the present invention effectively overcomes the various shortcomings in the prior art and has high industrial utilization value.

[0134] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical principles disclosed herein are intended to be covered by the claims of the present invention.

Claims

1. A corpus cleaning method, characterized in that: The steps include: Acquiring original corpus to be cleaned, performing pre-annotation processing on the original corpus to generate pre-annotated corpus, and parsing the pre-annotated corpus to generate first parsing information; Optimizing the original corpus to obtain optimized corpus, and parsing the optimized corpus to generate second parsing information; performing enhancement processing on the original corpus to obtain enhanced corpus, and parsing the enhanced corpus to generate third parsing information; generating target parsing information according to the first parsing information, the second parsing information, and the third parsing information, and generating corresponding synthetic corpus according to the target parsing information; Performing desensitization processing on the synthetic corpus according to the desensitization library to generate corresponding desensitized corpus, and performing standardization processing on the desensitized corpus to generate corresponding target corpus; Pre-annotating the original corpus to generate pre-annotated corpus, and parsing the pre-annotated corpus to generate first parsing information, including: Labeling the original corpus to generate a tree structure, wherein the tree structure includes a plurality of structural blocks, each structural block corresponding to a field; Copying the original corpus multiple times to obtain multiple backup corpora, and performing hierarchical annotation on each of the backup corpora according to the tree structure according to a preset rule to obtain multiple backup annotation information; Performing differential comparison on the plurality of backup annotation information, and selecting the annotation word with the highest occurrence frequency at each position of the tree structure in turn as the target annotation word, and combining the plurality of target annotation words in order to form the pre-annotated corpus; The pre-annotated corpus is parsed by a first parsing tool to obtain the first parsing information.

2. The corpus cleaning method according to claim 1, characterized in that: The differential comparison of the plurality of backup annotation information, selecting the annotation word with the highest occurrence frequency at each position of the tree structure in turn as the target annotation word, and combining the plurality of target annotation words in order to form the pre-annotated corpus includes: Determining a structural block corresponding to each field in the original corpus according to the tree structure; At the position of each structural block, respectively calculate the occurrence frequency of the annotation words of each backup annotation information in the current structural block, and select the annotation word corresponding to the maximum occurrence frequency as the target annotation word; According to the order of the structural blocks in the tree structure, the target annotated words are sequentially combined together to form the pre-annotated corpus.

3. The corpus cleaning method according to claim 1, characterized in that: The performing corpus optimization on the original corpus to obtain an optimized corpus, and parsing the optimized corpus to generate second parsing information, includes: A hash algorithm is used to calculate the data unique identifier of the original corpus and the semantic similarity in the original corpus, and duplicate content and similar content are removed according to the data unique identifier and the semantic similarity to obtain a deduplicated corpus; After parsing the deduplicated corpus through a syntax tree to remove logical errors, the marketing content is filtered based on a rule template; The optimized corpus is obtained after removing existing URL characters, special characters and meaningless symbols; The optimized corpus is parsed by a second parsing tool to obtain the second parsing information.

4. The corpus cleaning method according to claim 1, characterized in that: The performing enhancement processing on the original corpus to obtain enhanced corpus, and parsing the enhanced corpus to generate third parsing information, includes: Determining the non-core fields corresponding to the original corpus according to the tree structure; After replacing the non-core fields based on the domain vocabulary, a replacement corpus is generated; Calling a multilingual API combination loop to translate the replacement corpus to generate the enhanced corpus; The enhanced corpus is parsed using a third parsing tool to obtain the third parsing information.

5. The corpus cleaning method according to claim 1, characterized in that: Generating target parsing information according to the first parsing information, the second parsing information, and the third parsing information, and generating corresponding synthetic corpus according to the target parsing information includes: Splitting the first parsed information, the second parsed information, and the third parsed information according to the tree structure to obtain first information segments, second information segments, and third information segments at corresponding positions; Calculate the information segment mean of each position according to the first information segment, the second information segment, and the third information segment; Calculate the difference coefficient between the first information segment, the second information segment and the third information segment at each position and the mean of the information segments at the corresponding position respectively; selecting the first information segment, the second information segment, and the third information segment with the smallest difference coefficient as the target information segment; The target information segments at various positions are arranged and combined in sequence to form the target parsed information, and the corresponding synthetic corpus is generated according to the target parsed information.

6. The corpus cleaning method according to claim 1, characterized in that: The desensitizing the synthetic corpus according to the desensitizing library to generate corresponding desensitized corpus, and standardizing the desensitized corpus to generate corresponding target corpus, including: Comparing the synthesized corpus with the desensitization library to determine sensitive words therein, and replacing the sensitive words with encrypted words to generate the desensitized corpus; After the desensitized corpus is subjected to format unification, word form normalization processing and numerical standardization processing respectively, the corresponding target corpus is generated.

7. A corpus cleaning system, using the corpus cleaning method according to claim 1, characterized in that: include: A first parsing module is configured to obtain an original corpus to be cleaned, perform pre-annotation processing on the original corpus to generate a pre-annotated corpus, and parse the pre-annotated corpus to generate first parsing information; A second parsing module is configured to perform corpus optimization on the original corpus to obtain an optimized corpus, and parse the optimized corpus to generate second parsing information; a third parsing module, configured to perform enhancement processing on the original corpus to obtain enhanced corpus, and parse the enhanced corpus to generate third parsing information; a synthesis module, configured to generate target parsing information according to the first parsing information, the second parsing information, and the third parsing information, and generate corresponding synthesized corpus according to the target parsing information; The processing module is used to perform desensitization processing on the synthetic corpus according to the desensitization library to generate corresponding desensitized corpus, and to generate corresponding target corpus after performing standardization processing on the desensitized corpus.

8. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the corpus cleaning method according to any one of claims 1 to 6 is implemented.

9. A terminal, characterized in that: include: processor and memory; The memory is used to store computer programs; The processor is configured to execute the computer program stored in the memory, so that the terminal executes the corpus cleaning method according to any one of claims 1 to 6.

Images