Preprocessing method of pre-training data and electronic equipment

Abstract

The invention discloses a pre-training data preprocessing method and electronic equipment, and belongs to the field of data processing. The method comprises the following steps: a data input module receives to-be-processed data and a target processing demand Prompt, and sends the target processing demand Prompt to a data brain module; the data brain module receives the target processing demand Prompt, analyzes the target processing demand Prompt by using a large language model, and determines a first dynamic processing strategy corresponding to the processing task; each intelligent Agent module processes the to-be-processed data according to the first dynamic processing strategy to obtain a corresponding processing result; and the data output module obtains the processing result and outputs the processing result. Different intelligent Agent modules are adopted for targeted processing according to different information and requirements, the processing efficiency is improved, harmful information is avoided, privacy information leakage is avoided, normative data meeting the requirements can be obtained, and the method is suitable for various scenes.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and in particular relates to a preprocessing method and electronic device for pre-training data. Background Art

[0002] Currently, the training data of large language models (LLM) mainly comes from Internet web pages, books, encyclopedias, academic papers, codes and other content. These data sources include but are not limited to CommonCrawl, Wikipedia, arXiv and GitHub. Due to the diverse sources of data, its content format is complex and diverse, and the forms of redundant information, harmful information and privacy information are also different. Therefore, the processing of pre-training data faces the following challenges: 1) It is difficult to process redundant data: the repetition rate of data from different sources is high, and the content quality is uneven, which affects the training effect of the model. 2) Diversification of harmful information and leakage of privacy information: it is impossible to remove sensitive information, harmful content, privacy leakage information, etc. 3) Poor format standardization: data cleaning and unified formatting require a lot of rules and manual intervention.

[0003] In order to solve the above problems, a pre-processing method and electronic device for pre-training data are proposed in the present application. Summary of the invention

[0004] In order to address the deficiencies of the prior art, the present application provides a pre-processing method and electronic device for pre-training data, which solve the problems of difficulty in processing redundant data, diversification of harmful information, leakage of privacy information, and poor format standardization in the processing of pre-training data in the prior art.

[0005] The technical effects to be achieved by this application are achieved through the following solutions: In a first aspect, the present application provides a method for preprocessing pretraining data, the method comprising: The data input module receives the data to be processed and the target processing requirement Prompt, and sends the target processing requirement Prompt to the data brain module; The data brain module receives the target processing requirement Prompt, and uses the large language model to parse the target processing requirement Prompt to determine a first dynamic processing strategy corresponding to the processing task; Each intelligent Agent module processes the data to be processed according to the first dynamic processing strategy to obtain a corresponding processing result; The data output module obtains the processing result and outputs the processing result.

[0006] In some embodiments, determining a first dynamic processing strategy corresponding to the processing task includes: Determine the logical flow corresponding to the processing task; Through dynamic programming algorithms or logic rules, the various intelligent Agent modules that need to be used are determined, and the calling sequence and parameter configuration of the various intelligent Agent modules are generated.

[0007] In some embodiments, the types of the intelligent agent modules include: basic operators, content analysis operators and advanced operators; the intelligent agent modules are the following multiple modules: HTML cleaning Agent module, used to remove HTML tags and retain the text content; Harmful content filtering Agent module, used to filter harmful content; Language detection Agent module, used to detect the text language and match it with the target language; Duplicate data monitoring agent module, used to detect and remove semantically duplicate text; The privacy information identification Agent module is used to identify and shield privacy information; Sentence sorting agent module, used to sort by length or grammatical complexity; and User-defined Agent module for special needs processing.

[0008] In some embodiments, each of the intelligent Agent modules processes the data to be processed according to the first dynamic processing strategy to obtain corresponding processing results, including: Each intelligent Agent module processes the data to be processed in turn according to the calling order and parameter configuration to obtain the corresponding processing results; or The various intelligent Agent modules process the data to be processed in parallel according to the calling sequence and parameter configuration to obtain corresponding processing results.

[0009] In some embodiments, the processing result includes target data and an execution log corresponding to the processing task; the data output module obtains the processing result and outputs the processing result, including: The data output module obtains the target data and the execution log corresponding to the processing task; outputting the target data to a user; Record the execution log corresponding to the processing task, and the execution log is used for subsequent tracking and debugging.

[0010] In some embodiments, the user obtains the target data and verifies whether the target data is the required data. If the target data is the required data, the data is saved. If the target data is not the required data, the execution log corresponding to the processing task is obtained from the data output module, and the preprocessing process is tracked and debugged based on the execution log.

[0011] In some embodiments, the data input module receives data to be processed, including: The data input module receives multiple pieces of data in batches and dynamically adjusts the task allocation strategy according to the task scale.

[0012] In some embodiments, the formats of the data to be processed include JSON, HTML, and CSV.

[0013] In a second aspect, the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the preprocessing method of pre-training data of the first aspect when executing the computer program.

[0014] In a third aspect, the present application provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the pre-processing method of pre-training data of the first aspect.

[0015] The pre-processing method and electronic device for pre-trained data provided by the present application parse prompts through a large language model to determine a dynamic processing strategy for processing tasks, and then perform corresponding processing on the pre-trained data based on multiple intelligent agent modules, thereby reducing the difficulty of processing redundant data; different intelligent agent modules are used for targeted processing according to different information and needs, thereby improving processing efficiency, avoiding the appearance of harmful information and avoiding the leakage of privacy information, and finally obtaining normative data that meets the needs and is suitable for a variety of scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present application or the existing technical solutions, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 This is a flowchart of a method for preprocessing pre-training data in one embodiment of the present application; Figure 2It is a schematic block diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0019] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in one or more embodiments of the present application should be understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in one or more embodiments of the present application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0020] In the related technologies, the processing solutions for pre-training data include DataJuicer, FlagData, Redpajama, Fineweb, and Fineweb-edu, etc. Most of the above solutions are implemented by integrating 90 to 100 basic functional operators. The implementation and use of these operators have the following problems: Operators based on artificial rules (such as regular expressions): 1) The precision and recall rates are insufficient, making it difficult to adapt to a variety of scenarios; 2) Data engineers are required to manually design rules, which consumes a lot of manpower.

[0021] Operators based on small models (such as FastText and n-gram): There is an out-of-distribution problem, which affects applicability.

[0022] In terms of operator usage, there are problems such as unclear operator objectives and functions relying on manual experience definition: 1) The threshold needs to be set manually, and the thresholds of different data sources are inconsistent; 2) Operator users cannot determine whether to use the operator.

[0023] For example, the following example is unclear: Example 1: Filter sentences with an average word length of x; Example 2: Filter sentences with more / less than x words.

[0024] Therefore, it is necessary to adopt the pre-processing method of pre-training data of the present application to solve the problems of high manual design cost, poor adaptability, unclear operator objectives, etc. in the existing pre-training data processing process.

[0025] This application proposes an automated preprocessing framework based on a large language model (LLM) + intelligent agent, which combines the strong generalization ability of LLM with the high efficiency of traditional methods to achieve multi-scenario, multi-task, and multi-language pre-training data cleaning and screening.

[0026] Prompt in this application refers to the input information provided to LLM, including text, instructions or context, which is used to trigger the calculation process of the model and generate responses or output content related to the input. Agent refers to an independent software entity or module with autonomous decision-making and execution capabilities, used to complete specific tasks or functions. Agent receives input data, processes it according to predefined logic or algorithms, and outputs results, and can be used as the core execution unit in the intelligent system.

[0027] Various non-limiting implementations of the present application are described in detail below in conjunction with the accompanying drawings.

[0028] First, refer to Figure 1 , the preprocessing method of the pre-training data of this application is described in detail.

[0029] The present application provides a method for preprocessing pretraining data, the method comprising: S1: The data input module receives the data to be processed and the target processing requirement Prompt, and sends the target processing requirement Prompt to the data brain module; S2: The data brain module receives the target processing requirement Prompt, and uses the large language model to parse the target processing requirement Prompt to determine a first dynamic processing strategy corresponding to the processing task; S3: Each intelligent Agent module processes the data to be processed according to the first dynamic processing strategy to obtain a corresponding processing result; S4: The data output module obtains the processing result and outputs the processing result.

[0030] The pre-processing method of pre-trained data provided in the present application parses prompts through a large language model to determine the dynamic processing strategy of the processing task, and then can perform corresponding processing on the pre-trained data based on multiple intelligent agent modules, thereby reducing the difficulty of processing redundant data; different intelligent agent modules are used for targeted processing according to different information and needs, thereby improving processing efficiency, avoiding the occurrence of harmful information and avoiding the leakage of privacy information, and finally obtaining normative data that meets the needs and is suitable for a variety of scenarios.

[0031] For example, the Data Brain (LM) module mainly implements the following functions: Core function: A small but efficient large language model (about 100M parameters) for reasoning about the logical flow of data processing and dynamically generating processing strategies.

[0032] Core features: Dynamic task parsing: parse the input processing requirements (Prompt) and generate the calling sequence and parameter configuration of the intelligent Agent module.

[0033] Specifically, the data brain module has the characteristics of multimodal adaptability: it supports the parsing and processing of multiple data types such as text, HTML structure, and code.

[0034] For example, the intelligent agent module is the core execution unit that implements specific data processing functions, which is driven by LLM or implemented by traditional algorithms. The design goal of each intelligent agent module is to complete a clear data cleaning, screening or labeling task, etc., which can improve the processing expertise of pre-trained data and improve the processing efficiency.

[0035] Exemplarily, the data input module receiving the data to be processed may include: single text input: single text data; the data to be processed means pre-trained data, which has a variety of data files: such as JSON, HTML, CSV and other file formats. At the same time, the processing requirement prompt may be received.

[0036] In some embodiments, determining a first dynamic processing strategy corresponding to the processing task includes: Determine the logical flow corresponding to the processing task; Through dynamic programming algorithms or logic rules, the various intelligent Agent modules that need to be used are determined, and the calling sequence and parameter configuration of the various intelligent Agent modules are generated.

[0037] In some embodiments, the types of the intelligent agent modules include: basic operators, content analysis operators and advanced operators; the intelligent agent modules are or include the following multiple modules: The HTML cleaning Agent module is used to remove HTML tags and retain the text content; for example, it is also used to remove special symbols, meaningless characters, etc.

[0038] The harmful content filtering Agent module is used to filter harmful content; harmful content includes related content that is not suitable for display, such as uncivilized content.

[0039] The language detection agent module is used to detect the text language and match it with the target language; ensuring that the data is consistent with the target language.

[0040] The duplicate data monitoring agent module is used to detect and remove semantically duplicate texts; for example, it can remove duplicates based on similarity (such as Jaccard, TF-IDF) and MD5 verification.

[0041] The private information identification Agent module is used to identify and shield private information; for example, it can identify mobile phone numbers, email addresses, ID card numbers, etc. based on regular expressions and NER models.

[0042] Sentence sorting agent module, used to sort by length or grammatical complexity; and User-defined Agent modules are used for special needs processing. Users can customize Agent modules according to actual needs, for example, they can add user-defined Agent modules to adapt to multi-language processing, especially low-resource languages. This application supports users to upload user-defined Agent modules, which are automatically loaded and registered by the system.

[0043] Exemplarily, the intelligent Agent module may further include one or more of the following modules: Sentence complexity evaluation agent module: used to evaluate the readability and grammatical complexity of sentences.

[0044] Topic consistency detection Agent module: used to detect whether the text deviates from the specified topic.

[0045] Sentiment Analysis Agent Module: used to label text sentiment (e.g., positive / negative / neutral).

[0046] Format normalization agent module: used to unify punctuation, encoding (such as UTF-8), and language spelling.

[0047] For example, the functions, algorithm implementations, and applicable scenarios of some core operators in the intelligent Agent module are shown in the following table:

[0048] For example, among the above-mentioned intelligent agent modules, the basic operators are: HTML cleaning agent module and duplicate data monitoring agent module; the content analysis operators are: privacy information identification agent module, harmful content filtering agent module; and the advanced operators are: sentence complexity evaluation agent module, topic consistency detection agent module, sentiment analysis agent module and format normalization agent module. The basic logic is that the basic operators perform routine tasks; the advanced operators perform complex tasks.

[0049] Exemplarily, each intelligent agent module implements the following functions: Operator library management: The operator library includes basic operators, content analysis operators, and advanced operators.

[0050] Operator execution: Call specific operators to process data according to the task assignment of the data brain.

[0051] Task parallelization: supports parallel execution between operators to improve processing efficiency.

[0052] Exemplarily, the Prompt-driven data task is parsed through the large language module, wherein the Prompt can be designed to include information such as data cleaning goals, expected output format, and processing constraints. The example is as follows: Input: Clean the text containing HTML tags, remove redundant content, and ensure that the sentence length is 10-50 words.

[0053] Output: Plain text format, each data entry does not exceed 100 words, encoded in UTF-8.

[0054] In some embodiments, a large language model is used to parse the prompt, converting natural language requirements into specific operator calls and configurations. The example parsing process is as follows: Input Prompt: Cleans text containing HTML tags, removes harmful content, and sorts by sentence length.

[0055] Parsing results: Call: HTML cleaning Agent (parameter: keep the text content).

[0056] Call: Harmful content filtering Agent (parameter: use a filter with a high safety level).

[0057] Call: Text Sorting Agent (parameter: sort in ascending order by sentence length).

[0058] Dynamic task adaptation: The data brain module generates a specific task flow based on the prompt and decomposes it into subtasks handled by multiple intelligent agent modules.

[0059] In some embodiments, each of the intelligent Agent modules processes the data to be processed according to the first dynamic processing strategy to obtain corresponding processing results, including: Each intelligent Agent module processes the data to be processed in turn according to the calling order and parameter configuration to obtain the corresponding processing results; or The various intelligent Agent modules process the data to be processed in parallel according to the calling sequence and parameter configuration to obtain corresponding processing results.

[0060] In some embodiments, the processing result includes target data and an execution log corresponding to the processing task; the data output module obtains the processing result and outputs the processing result, including: The data output module obtains the target data and the execution log corresponding to the processing task; outputting the target data to a user; Record the execution log corresponding to the processing task, and the execution log is used for subsequent tracking and debugging.

[0061] Exemplarily, the output content may include: Target data: formatted and output according to user requirements.

[0062] Execution log: records the execution process and results of each intelligent agent module for auditing and improvement.

[0063] In the modular framework design of this application, the modules are decoupled: each module (data input module, data brain module, each intelligent agent module, data output module) runs independently, and the modules communicate through interfaces; this application has strong scalability: it supports custom agents, new data types and processing tasks.

[0064] In some embodiments, the user obtains the target data and verifies whether the target data is the required data. If the target data is the required data, the data is saved. If the target data is not the required data, the execution log corresponding to the processing task is obtained from the data output module, and the preprocessing process is tracked and debugged based on the execution log.

[0065] In some embodiments, the data input module receives data to be processed, including: The data input module receives multiple data in batches and dynamically adjusts the task allocation strategy according to the task scale. Specifically, the data input module supports input format parsing: supports parsing and preprocessing of multiple data formats such as JSON, TXT, HTML, etc.; it also supports multi-task concurrent processing: receives multiple data in batches and dynamically adjusts the task allocation strategy according to the task scale.

[0066] In some embodiments, the formats of the data to be processed include JSON, HTML, and CSV.

[0067] The following technical effects can be achieved through the preprocessing method of pre-training data of this application: 1) Improve efficiency: Through feature hashing and efficient distribution matching, the time consumption of the screening process is reduced by more than 50%.

[0068] 2) Improved accuracy: The relevance of the selected data subset to the target task is increased by 30%.

[0069] 3) Resource saving: significantly reduce the loading and training time of redundant data and reduce training costs.

[0070] 4) High degree of automation: no manual intervention required, strong adaptability, suitable for a variety of tasks.

[0071] The preprocessing method of pre-training data in this application can also support the following technical directions: 1. Multimodal data screening: supports feature analysis and screening of images and texts.

[0072] 2. Multilingual expansion: Cross-language data screening is achieved through multilingual embedding models.

[0073] 3. Incremental optimization: Update the screening strategy in real time to adapt to changes in task requirements.

[0074] It should be noted that the method of one or more embodiments of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of one or more embodiments of the present application, and the multiple devices will interact with each other to complete the described method.

[0075] It should be noted that the above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0076] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also discloses an electronic device.

[0077] Specifically, Figure 2 A hardware structure diagram of an electronic device for a pre-processing method of pre-training data provided in this embodiment is shown, and the device may include: a processor 410, a memory 420, an input / output interface 430, a communication interface 440, and a bus 450. The processor 410, the memory 420, the input / output interface 430, and the communication interface 440 are connected to each other in communication within the device through the bus 450.

[0078] The processor 410 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0079] The memory 420 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 420 may store an operating system and other application programs. When the technical solution provided in the embodiment of the present application is implemented by software or firmware, the relevant program code is stored in the memory 420 and is called and executed by the processor 410.

[0080] The input / output interface 430 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0081] The communication interface 440 is used to connect a communication module (not shown in the figure) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired manner (for example, USB, network cable, etc.) or a wireless manner (for example, mobile network, Wi-Fi, Bluetooth, etc.).

[0082] The bus 450 includes a path that transmits information between the various components of the device (eg, the processor 410 , the memory 420 , the input / output interface 430 , and the communication interface 440 ).

[0083] It should be noted that, although the above device only shows the processor 410, the memory 420, the input / output interface 430, the communication interface 440 and the bus 450, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiment of the present application, and does not necessarily include all the components shown in the figure.

[0084] The electronic device of the above embodiment is used to implement the corresponding pre-processing method of pre-training data in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0085] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, one or more embodiments of the present application also provide a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the preprocessing method of pre-training data as described in any of the above embodiments.

[0086] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0087] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the preprocessing method of pre-training data as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0088] A person skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of different aspects of one or more embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0089] In addition, to simplify the description and discussion, and in order not to make one or more embodiments of the present application difficult to understand, the well-known power / ground connections to the integrated circuit (IC) chip and other components may or may not be shown in the provided figures. In addition, the device may be shown in the form of a block diagram to avoid making one or more embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which one or more embodiments of the present application will be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that one or more embodiments of the present application can be implemented without these specific details or with changes in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0090] Although the present application has been described in conjunction with specific embodiments of the present application, many alternatives, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the discussed embodiments.

[0091] One or more embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application should be included in the scope of protection of the present application.

Claims

1. A preprocessing method for pretraining data, characterized in that: The method comprises: The data input module receives the data to be processed and the target processing requirement Prompt, and sends the target processing requirement Prompt to the data brain module; The data brain module receives the target processing requirement Prompt, and uses the large language model to parse the target processing requirement Prompt to determine a first dynamic processing strategy corresponding to the processing task; Each intelligent Agent module processes the data to be processed according to the first dynamic processing strategy to obtain a corresponding processing result; The data output module obtains the processing result and outputs the processing result.

2. The preprocessing method for pretraining data according to claim 1, characterized in that: The determining of a first dynamic processing strategy corresponding to the processing task includes: Determine the logical flow corresponding to the processing task; Through dynamic programming algorithms or logic rules, the various intelligent Agent modules that need to be used are determined, and the calling sequence and parameter configuration of the various intelligent Agent modules are generated.

3. The preprocessing method for pretraining data according to claim 2, characterized in that: The types of intelligent agent modules include: basic operators, content analysis operators and advanced operators; the intelligent agent modules are the following modules: HTML cleaning Agent module, used to remove HTML tags and retain the text content; Harmful content filtering Agent module, used to filter harmful content; Language detection Agent module, used to detect the text language and match it with the target language; Duplicate data monitoring agent module, used to detect and remove semantically duplicate text; The privacy information identification Agent module is used to identify and shield privacy information; Sentence sorting agent module, used to sort by length or grammatical complexity; and User-defined Agent module for special needs processing.

4. The preprocessing method for pretraining data according to claim 3, characterized in that: Each of the intelligent Agent modules processes the data to be processed according to the first dynamic processing strategy to obtain corresponding processing results, including: Each intelligent Agent module processes the data to be processed in turn according to the calling order and parameter configuration to obtain the corresponding processing results; or The various intelligent Agent modules process the data to be processed in parallel according to the calling sequence and parameter configuration to obtain corresponding processing results.

5. The preprocessing method for pretraining data according to claim 1, characterized in that: The processing result includes target data and an execution log corresponding to the processing task; the data output module obtains the processing result and outputs the processing result, including: The data output module obtains the target data and the execution log corresponding to the processing task; outputting the target data to a user; Record the execution log corresponding to the processing task, and the execution log is used for subsequent tracking and debugging.

6. The preprocessing method for pretraining data according to claim 5, characterized in that: The user obtains the target data and verifies whether the target data is the required data. If the target data is the required data, the data is saved. If the target data is not the required data, the execution log corresponding to the processing task is obtained from the data output module, and the preprocessing process is tracked and debugged based on the execution log.

7. The preprocessing method for pretraining data according to claim 1, characterized in that: The data input module receives the data to be processed, including: The data input module receives multiple pieces of data in batches and dynamically adjusts the task allocation strategy according to the task scale.

8. The preprocessing method for pretraining data according to claim 1, characterized in that: The formats of the data to be processed include JSON, HTML and CSV.

9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the preprocessing method for pre-training data as described in any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the pre-processing method of pre-training data as described in any one of claims 1 to 8.

Images