Quality improvement method and equipment based on corpus cleaning, medium and product

Abstract

The embodiment of the invention relates to the technical field of natural language processing, and discloses a corpus cleaning-based quality improvement method and device, a medium and a product. Processing the corpus data through a multi-level corpus cleaning strategy, wherein the multi-level corpus cleaning strategy comprises basic cleaning, deep cleaning and semantic cleaning; irrelevant information of the corpus data is removed through the basic cleaning; adjusting a word and sentence structure of the corpus data through the deep cleaning; identifying and correcting semantic errors of the corpus data through the semantic cleaning; and performing data enhancement, noise filtering and context optimization on the corpus data processed by the multi-level corpus cleaning strategy. And through a multi-level cleaning strategy and a quality improvement technology, the purification degree of the corpora is improved, and noise and redundant information are reduced. The method and the device can at least solve the problem of insufficient purity and accuracy of corpus data.

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and in particular to a quality improvement method, device, medium and product based on corpus cleaning. Background Art

[0002] With the rapid development of artificial intelligence technology, especially the widespread application of natural language processing (NLP) technology, massive text data has become a key resource for model training and application. However, the original corpus often contains a lot of noise, incomplete or redundant information. These problems seriously affect the quality of the data, and thus restrict the training effect and actual application performance of the model. In the prior art, although some basic corpus cleaning methods such as removing stop words, word segmentation, spelling correction, etc. have alleviated these problems to a certain extent, there are still significant deficiencies. For example, these methods are difficult to effectively identify and process context-related errors, and cannot fully understand and process complex semantic information, resulting in limited cleaning effects and unable to meet the needs of high-quality corpus.

[0003] In order to solve these problems, advanced corpus cleaning technology has gradually introduced more intelligent and automated means. By combining machine learning and deep learning methods, new corpus cleaning technology can more efficiently identify and filter noise, correct grammatical and spelling errors, and ensure data consistency and integrity at the semantic level. For example, context-based language models can effectively identify and correct semantic errors in corpora, while the use of large-scale pre-trained models (such as BERT, GPT, etc.) can more accurately process complex semantic information and improve data quality. In addition, combining artificial intelligence and the feedback loop of human experts can continuously optimize and improve the cleaning algorithm, gradually improve the overall quality of the corpus, and provide a solid foundation for model training and practical application. Summary of the invention

[0004] One purpose of the present application is to provide a quality improvement method, device, medium and product based on corpus cleaning, at least to solve the problem of insufficient purity and accuracy of corpus data.

[0005] To achieve the above objectives, some embodiments of the present application provide the following aspects:

[0006] In the first aspect, some embodiments of the present application also provide a quality improvement method based on corpus cleaning, including processing corpus data through a multi-level corpus cleaning strategy, the multi-level corpus cleaning strategy including basic cleaning, deep cleaning and semantic cleaning; removing irrelevant information of the corpus data through the basic cleaning; adjusting the word and sentence structure of the corpus data through the deep cleaning; identifying and correcting semantic errors of the corpus data through the semantic cleaning; and performing data enhancement, noise filtering and context optimization on the corpus data after being processed by the multi-level corpus cleaning strategy.

[0007] In a second aspect, some embodiments of the present application further provide an electronic device, comprising: one or more processors; and a memory storing computer program instructions, wherein the computer program instructions, when executed, cause the processor to perform the steps of the method described above.

[0008] In a third aspect, some embodiments of the present application further provide a computer-readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the method as described above.

[0009] In a fourth aspect, some embodiments of the present application further provide a computer program product, comprising a computer program / instruction, which implements the steps of the method described above when executed by a processor.

[0010] Compared with the related technology, the solution provided in the embodiment of the present application improves the degree of purification of the corpus, reduces noise and redundant information, and significantly improves the accuracy and robustness of the training model through multi-level cleaning strategies and quality improvement technologies; high-quality corpus improves the model training effect, reduces repeated debugging and correction work caused by data quality issues, and reduces development costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0012] Figure 1 A flowchart of a quality improvement method based on corpus cleaning provided according to an embodiment of the present application;

[0013] Figure 2 The present invention is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0015] First embodiment

[0016] The first embodiment of the present application relates to a quality improvement method based on corpus cleaning. Figure 1 As shown, the method may include the following steps:

[0017] S101, processing the corpus data through a multi-level corpus cleaning strategy, wherein the multi-level corpus cleaning strategy includes basic cleaning, deep cleaning and semantic cleaning;

[0018] S102, removing irrelevant information of the corpus data by the basic cleaning; adjusting the word and sentence structure of the corpus data by the deep cleaning; identifying and correcting semantic errors of the corpus data by the semantic cleaning;

[0019] S103, performing data enhancement, noise filtering and context optimization on the corpus data processed by the multi-level corpus cleaning strategy.

[0020] The following is a detailed description of each of the above steps.

[0021] For step S101, in natural language processing, a large amount of text data is required to train and apply the model, and these original corpus data may contain various noises, such as spelling errors, incomplete sentences, irrelevant information, etc. These problems will affect the quality of the data, thereby restricting the training effect and actual application performance of the model. It is necessary to clean these original corpus data to obtain high-quality corpus data for application and model training. These original corpus data are cleaned through a multi-level corpus cleaning strategy to systematically solve the noise and redundant information problems in the corpus, thereby improving the corpus quality, improving the model training effect and application performance. The multi-level corpus cleaning strategy includes three parts: basic cleaning, deep cleaning and semantic cleaning, which deal with different types of corpus problems.

[0022] For step S102, basic cleaning mainly includes removing stop words, punctuation marks, and duplicate content. Stop words refer to words that often appear in a specific language environment but contribute little to semantics, such as "de", "shi", "zai", etc. By removing stop words, the interference of irrelevant information can be reduced, and the effective information density of the corpus can be increased; punctuation mark processing aims to remove or standardize the use of punctuation marks to avoid syntactic misunderstandings caused by punctuation errors; removing duplicate content can reduce the impact of redundant information on model training. Through basic cleaning, the noise in the corpus data is significantly reduced, the interference of irrelevant information on data processing and model training is reduced, which helps to improve the accuracy of subsequent cleaning and analysis.

[0023] Deep cleaning involves spelling correction, word segmentation, and part-of-speech tagging. Spelling correction uses predefined spelling check tools or model-based error correction algorithms to identify and correct spelling mistakes. The word segmentation process divides continuous text into independent words, which is applicable to languages without obvious word boundaries such as Chinese and German. Part-of-speech tagging assigns corresponding part-of-speech tags to each word, such as nouns, verbs, adjectives, etc., to facilitate understanding of the function of words in a sentence. Through deep cleaning, the accuracy of vocabulary and sentence structure in the corpus can be ensured, which helps to improve the comprehensibility and readability of the text and lays a foundation for subsequent semantic analysis.

[0024] Semantic cleaning can use pre-trained language models (such as BERT, GPT) to identify and correct semantic errors in the corpus through context relevance analysis. These models can capture context information, detect and correct semantic inconsistencies or errors in sentences, such as ambiguity problems caused by synonym replacement or misuse of polysemous words. The introduction of semantic cleaning improves the accuracy of the corpus at the semantic level, reduces ambiguity and errors in context understanding, and ensures the consistency and logic of the cleaned corpus at the semantic level.

[0025] For step S103, for the corpus data processed by the multi-level corpus cleaning strategy, data augmentation, noise filtering, and context optimization are performed. In this embodiment, the diversity of the corpus is increased by means of synonym replacement and sentence reconstruction. For example, replacing "kuaisu" with "xunsu", or converting an active sentence into a passive sentence. This process uses a pre-constructed thesaurus and sentence structure rules to achieve the expansion of the diversity of the corpus. Data augmentation increases the diversity of the corpus, enabling the model to learn more diverse expressions during training and improving the generalization ability and robustness of the model.

[0026] Introduce a noise recognition model. By analyzing the characteristics of low-quality texts in the corpus, it automatically identifies and filters out noisy data. These noisy data usually include incomplete content, incorrect information, or text fragments irrelevant to the theme. Noise filtering effectively reduces the negative impact of low-quality corpus on model training, ensures the high quality of input data, and improves the training effect and performance of the model.

[0027] Use a deep learning model to optimize the context of the corpus. In this embodiment, a fine-tuned BERT model is used to optimize the context of the sentences in the corpus to enhance the coherence and readability of the text. Context optimization improves the user experience by enhancing text coherence, making the processed text more fluent and natural in applications, and helping to improve the model's understanding and response ability to complex contexts.

[0028] It is not difficult to find that, compared with the related technologies, in the solution provided by the embodiment of the present application, through a multi-level corpus cleaning strategy and quality improvement technology, irrelevant information and noise in the corpus are effectively removed, the sentence structure is adjusted, and semantic errors are identified and corrected. This method significantly improves the purity and semantic consistency of the corpus, ensuring high-quality corpus input. In addition, the introduction of data augmentation, noise filtering, and context optimization technologies further increases the diversity, reliability, and coherence of the corpus, thereby improving the training effect and practical application performance of the NLP model. Generally speaking, this solution realizes more efficient and accurate corpus processing, and improves the accuracy and robustness of the model.

[0029] Second Embodiment

[0030] The second embodiment of the present application relates to a quality improvement method based on corpus cleaning. The second implementation is an improvement based on the first embodiment. The specific improvement lies in:

[0031] Basic cleaning removes the irrelevant information in the corpus data, including removing stop words, punctuation marks, and duplicate content. Stop words refer to words that frequently appear in a specific language environment but contribute little to semantics, such as "de", "le", "zai" in Chinese. Removing stop words can reduce the interference of irrelevant information on corpus analysis and increase the density of effective information. By constructing a stop word dictionary and traversing the corpus data, the matching stop words are removed.

[0032] Punctuation marks are usually used for sentence separation and pauses, but in some analysis scenarios, they contribute limited semantics. By removing redundant punctuation marks, the text structure can be simplified and the unnecessary syntax parsing burden can be reduced. The removal of punctuation marks is achieved through regular expression matching, and necessary separators are retained to maintain the sentence structure.

[0033] The corpus data may contain a large amount of duplicate content, which will affect the training efficiency of the model and the accuracy of the results. Removing duplicate content can be achieved through text comparison algorithms (such as hash comparison, text fingerprint, etc.) to ensure the uniqueness and independence of each corpus.

[0034] The word and sentence structure of the corpus data is adjusted through the deep cleaning, including adjusting the word and sentence structure of the corpus data through spelling correction, word segmentation and part-of-speech tagging. There are often spelling errors in corpus data, especially data collected from user-generated content (UGC). The existence of spelling errors will affect the quality of the corpus and the training effect of the model. Spelling correction identifies spelling errors in the corpus through spell checking algorithms (such as edit distance algorithms, dictionary matching) and automatically corrects them. For example, "app l ee" is corrected to "apple le".

[0035] In some languages ​​(such as Chinese and German), there are no obvious separators between words. Word segmentation is to divide continuous text into independent words for subsequent processing and analysis. Word segmentation algorithms (such as maximum matching method, hidden Markov model, etc.) are used to segment text to ensure the correct division of word boundaries.

[0036] Part-of-speech tagging is to assign a corresponding part-of-speech tag to each word in the corpus, such as noun, verb, adjective, etc. By using a pre-trained part-of-speech tagging model (such as CRF, BiLSTM-CRF, etc.), the part of speech of each word can be automatically tagged to ensure the accuracy of the sentence structure. For example, the sentence "Apples are delicious" is tagged as "apple / noun very / adverb delicious / verb".

[0037] Semantic cleaning identifies and corrects semantic errors in the corpus data, including identifying and correcting semantic errors in the corpus data through contextual relevance analysis. The core of semantic cleaning is contextual relevance analysis, which identifies semantic errors by analyzing the contextual relationship between words in the corpus. Pre-trained language models (such as BERT and GPT) can capture contextual information in the text and identify words or sentences in the corpus that do not conform to the context. For example, in the sentence "Cats like to eat bones", the context analysis model can identify the inconsistency between "bones" and "cats" and suggest changing them to "fish". Once semantic errors in the corpus are identified, correction suggestions will be provided based on the contextual information. Semantic correction can be achieved by replacing words that do not conform to the context, adjusting sentence structure, or reconstructing sentences. The model can generate new semantically appropriate words based on known collocation rules or through a deep learning model. For example, the part that does not conform to the context in "She likes to read newspapers and he likes to drink tea" is corrected to "She likes to drink tea and he likes to read newspapers".

[0038] A multi-level corpus cleaning model is constructed to process the corpus data; the multi-level corpus cleaning model performs the basic cleaning through regularization; the multi-level corpus cleaning model detects the error proportion and error type of the sentence structure through spelling, grammar and part of speech, and adjusts the word and sentence structure of the corpus data according to the error proportion and error type; the multi-level corpus cleaning model performs context relevance analysis on the corpus data through random sampling.

[0039] Data enhancement includes: expanding the diversity of the corpus data by performing synonym replacement and sentence reconstruction, and further optimizing the replacement effect by combining context and word frequency information. Data enhancement aims to improve the model's generalization ability for different expressions by increasing the diversity of the corpus. Data enhancement is achieved through two methods: synonym replacement and sentence reconstruction, and further optimizing the replacement effect by combining context and word frequency information.

[0040] Using the synonym library, some words in the corpus are replaced with synonyms according to the context and word frequency information, thereby increasing the diversity of the corpus. The specific steps are as follows: Establish a rich synonym library, which contains common words and their synonyms. The selection of synonyms in the library is based on word frequency and usage scenarios to ensure that the replaced corpus still conforms to the natural language expression habits; first, the corpus is contextually analyzed to identify target words that can be replaced. According to word frequency information and context, the most suitable synonym is selected for replacement. For example, in "rapid development", "rapid" may be chosen to replace "rapid" because "rapid" is more in line with the context in some contexts;

[0041] By changing the sentence structure, the diversity of the corpus can be further increased. Sentence reconstruction can include converting active sentences into passive sentences, changing the word order of sentences, etc. The specific steps are as follows: Use a grammar parsing tool to parse the sentence and identify the subject, predicate, object and other grammatical components of the sentence; based on the parsing results, generate a new sentence form by changing the sentence structure, for example, converting "the company has released a new product" to "the new product was released by the company". This reconstruction maintains the original semantics of the sentence, but changes the grammatical form and enriches the corpus expression.

[0042] Noise filtering includes: filtering low-quality corpus of the corpus data through a noise recognition model; the noise recognition model is trained according to the noise generated in the process of processing the corpus data according to the multi-level corpus cleaning strategy, and setting standards for corpus quality.

[0043] Build and train a noise recognition model that can detect and filter low-quality corpora. The training of the model relies on the noise data collected during the multi-level corpus cleaning process. The specific steps are as follows: During the multi-level cleaning process, record the noise features such as grammatical errors, spelling errors, semantic inconsistencies, etc. These feature data are used to train the noise recognition model so that it can automatically recognize similar noise patterns; according to the collected noise features, set the corpus quality standards, such as the allowable number of spelling errors, grammatical error types, and contextual consistency. The model uses these standards to judge the quality of the corpus and filter out low-quality corpora that do not meet the standards.

[0044] When processing new corpus data, the noise identification model analyzes the corpus line by line, marks low-quality parts and removes them. For example, paragraphs with a large number of spelling errors and grammatical incoherence will be identified as noise and filtered out, ensuring that subsequent processing is only for high-quality data.

[0045] Context optimization includes: context optimization of the corpus data through a deep learning model; the deep learning model is fine-tuned and trained according to the vertical field corpus. Use a pre-trained deep learning language model (such as BERT, GPT, etc.) and fine-tune it to adapt to the corpus processing needs of specific fields (such as finance, education, life sciences). The specific steps are as follows: Collect high-quality corpus data in vertical fields for fine-tuning deep learning models. The fine-tuning process enables the model to better understand the language habits and expressions in a specific field and improve its context processing capabilities in this field; use the fine-tuned model to perform context analysis on the corpus data to identify possible semantic inconsistencies or logical contradictions. For example, when processing financial field corpus, the model can more accurately determine whether the financial terms and expressions in the sentence meet the standards of the field.

[0046] Through the context analysis capabilities of the deep learning model, the sentences in the corpus can be optimized and adjusted. For example, the order of sentences can be adjusted to improve logical coherence, or certain words can be replaced to eliminate ambiguity. The optimized corpus will be more semantically coherent and easier to understand and apply.

[0047] It is not difficult to find that in the embodiment of the present application, stop words, punctuation marks and repeated content are systematically removed to ensure the simplicity and effectiveness of the corpus data. The deep cleaning step makes the word and sentence structure of the corpus more accurate and standardized through spelling correction, word segmentation and part-of-speech tagging. Semantic cleaning identifies and corrects semantic errors in the corpus through contextual relevance analysis. Using pre-trained deep learning models (such as BERT, GPT), the coherence and logical consistency of the corpus at the semantic level are ensured, and contextual misunderstandings and ambiguous phenomena are reduced. Data enhancement is achieved through synonym replacement and sentence reconstruction, which expands the diversity of corpus data. This method enables the model to adapt to different language expressions, enhances the model's generalization ability to different contexts, and further improves the training effect of the model. The noise feature data accumulated by the noise recognition model during the multi-level cleaning process is used to train and set corpus quality standards. By automatically identifying and filtering low-quality corpora, the purity of the input data is ensured, and the accuracy and efficiency of subsequent processing are improved.

[0048] The step division of the above methods is only for the purpose of clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this patent.

[0049] In addition, some embodiments of the present application also provide an electronic device. The electronic device may be a digital computer in various forms, such as a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, etc. The electronic device may also be a mobile device in various forms, such as a personal digital processing, a cellular phone, a smart phone, a wearable device, and other similar computing devices.

[0050] The electronic device includes: one or more processors; and a memory storing computer program instructions, wherein when the computer program instructions are executed, the processor executes the steps of the method provided in any one or more of the above embodiments. Figure 2 An exemplary structural diagram of the electronic device is disclosed. Figure 2As shown, the electronic device includes: one or more processors 1101, memory 1102, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Among them, the components shown in this article, their connections and relationships, and their functions are only used as examples, and are not intended to limit the implementation of the present application described and / or required herein.

[0051] The electronic device may further include: an input device 1103 and an output device 1104. The processor 1101, the memory 1102, the input device 1103 and the output device 1104 may be connected via a bus or other means. Figure 2 The example of connecting through bus is taken in the following.

[0052] The input device 1103 can receive input digital or character information, and generate key signal input related to the user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 1104 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0053] To provide interaction with a user, the electronic device may be a computer. The computer has: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball), through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0054] In the embodiments of the present application, a computer program / instruction is stored on a computer-readable medium, and when the computer program / instruction is executed by a processor, the steps of the method provided by any one or more of the above embodiments are implemented. The computer-readable medium may be included in the electronic device described in the above embodiments; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more computer-readable instructions.

[0055] The memory 1102 can be used as a non-transient computer-readable storage medium, which can be used to store non-transient software programs, non-transient computer executable programs and modules. The processor 1101 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 1102, so as to implement the program instructions / modules corresponding to the method provided by any one or more embodiments in the embodiments of the present application.

[0056] The memory 1102 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 1102 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 1102 may optionally include a memory remotely arranged relative to the processor 1101, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0057] It should be noted that the computer-readable medium described in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0058] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0059] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet).

[0060] In the above-described embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. For example, it can be implemented by using an application specific integrated circuit (ASIC), a general-purpose computer or any other similar hardware device. In certain embodiments, the software program of the present application can be executed by a processor to implement the above steps or functions. Similarly, the software program of the present application (including relevant data structures) can be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and similar devices. In addition, some steps or functions of the present application can be implemented by hardware, for example, as a circuit that cooperates with a processor to perform each step or function.

[0061] The computer program product provided in the embodiment of the present application includes one or more computer programs / instructions, and when the computer program / instructions are executed by the processor, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive so lid state di sk (SSD)), etc.

[0062] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0063] The scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes that fall within the meaning and scope of the equivalent elements of the claims are included in the present application. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the device claim may also be implemented by one unit or device through software or hardware. The words "first", "second", etc. are only used to distinguish the description, and do not indicate any particular order, nor can they be understood as indicating or implying relative importance.

[0064] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily mention changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims, and the above embodiments should be regarded as exemplary and non-restrictive.

Claims

1. A quality improvement method based on corpus cleaning, characterized in that: The method comprises: Processing the corpus data through a multi-level corpus cleaning strategy, wherein the multi-level corpus cleaning strategy includes basic cleaning, deep cleaning and semantic cleaning; Removing irrelevant information of the corpus data by the basic cleaning; adjusting the word and sentence structure of the corpus data by the deep cleaning; identifying and correcting semantic errors of the corpus data by the semantic cleaning; The corpus data processed by the multi-level corpus cleaning strategy is subjected to data enhancement, noise filtering and context optimization.

2. The method according to claim 1, characterized in that: The multi-level corpus cleaning strategy includes: The removing irrelevant information of the corpus data by the basic cleaning includes removing stop words, punctuation marks and repeated contents; The adjusting the word and sentence structure of the corpus data by the deep cleaning includes adjusting the word and sentence structure of the corpus data by spelling correction, word segmentation and part-of-speech tagging; The semantic cleaning identifies and corrects semantic errors in the corpus data, including identifying and correcting semantic errors in the corpus data through context relevance analysis.

3. The method according to claim 2, characterized in that The method further comprises: Construct a multi-level corpus cleaning model to process the corpus data; The multi-level corpus cleaning model performs the basic cleaning through regularization; The multi-level corpus cleaning model detects the error ratio and error type of the sentence structure in spelling, grammar and part of speech, and adjusts the word and sentence structure of the corpus data according to the error ratio and error type; The multi-level corpus cleaning model performs context relevance analysis on the corpus data through random sampling.

4. The method according to claim 1, characterized in that: The data enhancement includes: By performing synonym replacement and sentence reconstruction, the diversity of the corpus data is expanded.

5. The method according to claim 1, characterized in that The noise filtering includes: Low-quality corpus of the corpus data is filtered out through a noise recognition model; the noise recognition model is trained according to the noise generated during the process of processing the corpus data according to the multi-level corpus cleaning strategy, and a standard is set for the corpus quality.

6. The method according to claim 1, characterized in that The context optimization includes: The corpus data is context-optimized through a deep learning model; the deep learning model is fine-tuned and trained based on the vertical domain corpus.

7. The method according to any one of claims 1 to 6, characterized in that: The method also includes: the data enhancement includes synonym replacement based on context and word frequency.

8. An electronic device, characterized in that: The electronic device comprises: one or more processors; and A memory storing computer program instructions, which, when executed, cause the processor to perform the steps of the method as claimed in any one of claims 1 to 7.

9. A computer readable medium having a computer program / instructions stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images