Preprocessing method, system and equipment of large model sample and storage medium

Abstract

The invention discloses a large model sample preprocessing method, system and device and a storage medium, and the method is realized based on the system, and comprises the steps: collecting original data from a plurality of sample sources through a sample collection module, and storing the original data in an original sample library provided by a sample storage module; acquiring the collected original data from the original sample library by utilizing a sample processing module, performing corresponding analysis, cleaning and quality inspection work on different types of sample data according to file types, and storing the sample data into a high-quality sample library; and on the basis of user requirements, maintenance, retrieval and display of original and high-quality samples are realized through sample directory creation / modification, requirement reporting, sample visualization and sample application functions provided by the sample management module. Through the automatic sample preprocessing process, the sample preparation time is remarkably shortened, and the large model training efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and big data processing technology, and specifically to a sample preprocessing method, system, device and storage medium for a machine learning model (referred to as a "big model") with large-scale parameters and complex computing structures. Background Art

[0002] As a research hotspot in the field of deep learning, large models, with their billions or even hundreds of billions of parameters, have demonstrated amazing complex representation capabilities and cross-domain generalization performance. Behind this is the extreme reliance on massive, high-quality sample data. Faced with such a huge parameter space, relying solely on limited or low-quality data for training can easily lead to problems such as model overfitting and poor generalization ability, which seriously restricts the practicality and reliability of large models. Therefore, how to efficiently collect, organize, annotate and expand sample data sets that meet the training needs of large models has become a major challenge in current research.

[0003] Data preprocessing is a key step in improving model performance. In existing technologies, data preprocessing often focuses on improving data quality and avoiding negative impacts through steps such as data cleaning, deduplication, and sensitive information filtering. From the overall process point of view, the current sample preprocessing systems and methods can no longer meet the needs of large-scale training. There is no standardized process guidance for the process from raw data to providing samples for model training, and no full-process support tools have been built. Therefore, it is particularly urgent and critical to develop an efficient, process-based, and modular large-model sample processing system and method. Summary of the invention

[0004] Purpose of the invention: The present invention provides a method, system, device and storage medium for preprocessing large model samples. Through an automated sample preprocessing process, the sample preparation time can be significantly shortened and the efficiency of large model training can be improved.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0006] A large model sample preprocessing method is implemented based on a large model sample preprocessing system, and the method comprises the following steps:

[0007] The sample collection module collects raw data from multiple sample sources and stores the raw data in the raw sample library provided by the sample storage module;

[0008] The sample processing module is used to obtain the collected raw data from the original sample library, and different types of sample data are analyzed, cleaned and quality-checked according to the file type. Sample cleaning mainly includes deduplication, classification, noise reduction, topic analysis and labeling, and the cleaned sample data, label and topic information, and quality inspection information are stored in the high-quality sample library provided by the sample storage module;

[0009] Based on user needs, the sample management module provides functions such as creating / modifying sample catalogs, submitting requirements, sample visualization, and sample application, so as to achieve maintenance, retrieval, and display of original and high-quality samples.

[0010] Furthermore, the workflow of the sample collection module includes:

[0011] Step 1-1: Add a new data source, configure the data source, and test whether it is connected normally;

[0012] Step 1-2: Configure the relevant parameters of the collection task, including collection method, collection time, and collection element information;

[0013] Step 1-3: Perform data collection and monitor the status of the collection task;

[0014] Step 1-4: Complete data collection and store it in the original sample library.

[0015] Furthermore, the workflow of the sample storage module includes:

[0016] Step 2-1: Create corresponding storage space or table structure according to the collection task configuration information;

[0017] Step 2-2: Collect data into the database and generate corresponding metadata;

[0018] Step 2-3: Sample data is taken out of the database and corresponding samples are extracted according to the connection task.

[0019] Furthermore, the workflow of the sample processing module includes:

[0020] Step 3-1: Extract samples from the original sample library;

[0021] Step 3-2: Perform sample analysis and process them according to the file type: txt, word, etc. are processed through data standardization and converted into sample data in a standard format; pdf and ceb are recognized through OCR to obtain the content in the file and then converted into sample data in a standard format; image types are resized, cropped or format converted; video files are decoded into video streams and audio streams, and the audio streams are stored in the original sample library as audio sample data;

[0022] Step 3-3: Clean the sample data in the standard format: Use automated tools for preliminary screening, including removing noise data, duplicate data, and related confidential information, and then perform key extraction, topic analysis, and labeling on the sample data;

[0023] Step 3-4: The processed sample data, labels, and subject information are written into a high-quality sample library to achieve the conversion of sample data from unstructured text to structured data;

[0024] Step 3-5: Set the sampling ratio according to actual needs, and extract the cleaned samples according to the ratio for sample quality inspection.

[0025] Step 3-6: Carry out sample quality inspection, including the overall process of sample distribution, manual quality inspection, content optimization and content evaluation. Sample distribution is to assign sample data to corresponding quality inspectors, manual quality inspection and content optimization are to conduct detailed inspection and improvement of samples, and content evaluation is to score the sample content quality.

[0026] Step 3-7: Store the quality inspection information into the high-quality sample library. The quality inspection information includes the quality inspector, quality inspection time, and sample quality score.

[0027] Furthermore, in step 3-3, the sample cleaning of text type mainly involves noise reduction, deduplication, labeling and feature extraction; the sample cleaning of image type mainly involves noise reduction, deduplication, conversion, labeling and feature extraction; the sample cleaning of video type mainly involves noise reduction, conversion, labeling; the sample cleaning of audio type mainly involves noise reduction, conversion, labeling and feature extraction.

[0028] Furthermore, in steps 3-5, samples are extracted from a high-quality sample library, or sample quality inspection is performed after sample cleaning.

[0029] Furthermore, in steps 3-6, content evaluation also includes automatically deleting or shielding low-quality samples whose content quality scores are lower than the specified threshold by setting a threshold.

[0030] A preprocessing system for large model samples includes the following modules:

[0031] The sample collection module is used to collect raw data from multiple sample sources and store the collected raw data in the raw sample library provided by the sample storage module;

[0032] The sample storage module provides an original sample library and a high-quality sample library. The original sample library is used to store the original data collected by the sample collection module, and the high-quality sample library is used to store the sample data, labels, subject information, and quality inspection information after being cleaned by the sample processing module.

[0033] The sample processing module is used to obtain the collected raw data from the original sample library, and perform corresponding analysis, cleaning and quality inspection on different types of sample data according to the file type. Sample cleaning mainly includes deduplication, classification, noise reduction, topic analysis and labeling, and the cleaned sample data, label and topic information, and quality inspection information are stored in the high-quality sample library built by the sample storage module;

[0034] The sample management module is used to maintain, retrieve and display original and high-quality samples based on user needs through the creation / modification of sample catalogs, demand submission, sample visualization and sample application functions provided by this module.

[0035] According to a third aspect, a computer device is provided, comprising one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of a method for preprocessing large model samples as described in the first aspect of the present invention are implemented.

[0036] In a fourth aspect, a computer storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of a preprocessing method for a large model sample as described in the first aspect of the present invention are implemented.

[0037] Compared with the prior art, the invention has the following beneficial effects: the present invention proposes a large model sample preprocessing system and method, which significantly shortens the sample preparation time and improves the efficiency of large model training through an automated sample preprocessing process. By processing the sample data in three stages: sample analysis, sample cleaning, and sample quality inspection, and designing a preprocessing process that better meets the needs for different sample data types, it reduces manual requirements and reduces costs, while effectively improving the quality and processing efficiency of large model samples, and improving the efficiency and accuracy of large model training. The invention can be flexibly applied to various types of large model training tasks, and has strong versatility and scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 The overall architecture diagram of the large model sample preprocessing system provided by the embodiment of the present invention;

[0039] Figure 2 A technical architecture diagram of a large model sample preprocessing system provided by an embodiment of the present invention;

[0040] Figure 3 An overall flow chart of a sample collection module of a large model sample preprocessing system provided by an embodiment of the present invention;

[0041] Figure 4An overall flow chart of a sample storage module of a large model sample preprocessing system provided by an embodiment of the present invention;

[0042] Figure 5 An overall flow chart of a sample processing module of a large model sample preprocessing system provided by an embodiment of the present invention;

[0043] Figure 6 This is an overall flow chart of the sample management module of the large model sample preprocessing system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings.

[0045] There is no standardized process standard for the current sample preprocessing work. The main contents include sample collection, storage, processing, and management. Each link is relatively independent and carried out on demand, which can hardly meet the needs of large-scale training. There is no standardized process guidance for the process from raw data to providing for model training, and no full-process support tools have been built. The specific manifestations are the following problems in the process of sample collection, storage, processing, and management:

[0046] (1) Sample collection: Manual sample collection makes it impossible to solidify and control the collection process, and there are no systematic tools, which leads to uncontrollable sample sources, unsafe transmission processes, and low work efficiency.

[0047] (2) Sample storage: Failure to use a dedicated file storage system to effectively manage samples has made it difficult or impossible to carry out sample statistics, analysis, and display, and has also been detrimental to subsequent sample management and services.

[0048] (3) Sample processing: No standardized sample processing process has been established, and various technical components are scattered everywhere, resulting in a lack of control over the progress of sample processing and the inability to trace back operations.

[0049] (4) Sample management: Samples exist in the form of files, which makes it impossible to implement hierarchical and classified management of samples. Services can only be provided in the form of files, which is a single means and cannot meet the diversified service requirements. At the same time, there is no standardized sample demand submission and sample authority management process, and there is no visual sample display method, which can easily lead to low efficiency in sample demand response and unsafe sample sharing process.

[0050] In view of this, the present invention has developed a preprocessing system suitable for large model samples, covering the entire process from sample collection, storage, processing to management. It aims to significantly improve the quantity, quality and diversity of sample data through automation and process-based means, and build a large-scale, normalized, full-chain high-quality sample service to meet users' diverse data needs and effectively support large model scenario applications.

[0051] like Figure 1 As shown, the system includes a sample collection module, a sample storage module, a sample processing module, and a sample management module.

[0052] The sample collection module can realize the collection and upload of multi-source data, provide data source configuration and automatic data collection functions, and the collected original data is stored in the original sample library. At the same time, when the amount of data is scarce, sample augmentation can be achieved by transforming and processing the data, expanding the number and diversity of training samples, thereby improving the generalization ability and accuracy of the model.

[0053] The sample storage module is divided into the original sample library and the high-quality sample library: the original sample library is used to store original multimodal unstructured sample data, such as text, pictures, videos and audio; the high-quality sample library is used to store structured data preprocessed by the sample processing module, including sample content, label information, quality inspection information, etc.

[0054] The sample processing module is divided into three parts: sample analysis, sample cleaning, and sample quality inspection, which realize the full life cycle of sample processing. Sample analysis mainly realizes the extraction and standardization of sample content; sample cleaning realizes sample deduplication, classification, noise reduction and labeling, and the generated samples are mainly used for pre-training of large models; sample quality inspection samples some samples for sample content optimization to improve sample quality, which can be used for supervised fine-tuning, reinforcement learning and other stages of large models.

[0055] The sample management module is based on sample metadata, and builds a complete sample data operation process to achieve management of the entire system. It is specifically divided into resource management, permission management, sample library management, processing flow management, and processing task management. By maintaining the original and high-quality sample directory, it provides sample users with intuitive sample display and retrieval capabilities. Supporting one-stop sample service processes such as demand submission, sample application, and problem feedback, it supports sample users to use it conveniently. By managing the entire process of sample collection, processing, and service, the stability and reliability of sample data are guaranteed.

[0056] The application service module provides a variety of applications and services such as sample visualization and unified data service (API). Sample visualization can display the statistical information of the samples in the current sample workbench, including the number of samples, sample processing overview, sample library overview, etc. The unified data service (API) provides a data interface in a customizable format, which can provide high-quality sample data for large models for training.

[0057] The sample data is preprocessed by the system of the present invention, thereby realizing the conversion of the sample data from unstructured to structured.

[0058] The specific technical architecture of the system in the embodiment of the present invention is as follows Figure 2 As shown in the figure, it is divided into four layers. The collection layer corresponds to the sample collection module in the overall architecture, and realizes the unified adaptation and data collection of different types of sample data sources through ETL, OCR and web crawlers. The storage layer corresponds to the sample storage module in the overall architecture, which is mainly divided into two parts: the original sample library module is implemented by object storage OSS; the high-quality sample library module is implemented by building the MPP library Doris. The processing layer corresponds to the sample processing module in the overall architecture, which is implemented by a variety of tool libraries and components. NLTK, JioNLP, etc. are used to process text samples; CNN and Embedding are used to process image and video samples; MFCC, etc. are used to process audio samples. The service layer and the display layer correspond to the application service layer in the overall architecture. The service layer includes search engines, statistical engines, processing engines and visualization engines, and provides external connection services such as API interfaces; the display layer mainly realizes the interface operation of the entire system, using a distributed front-end and back-end separation architecture, and is implemented through frameworks such as Vue and Springboot.

[0059] The focus of this invention is to achieve unified management of the entire life cycle of sample data from collection, processing, storage to application, as well as the conversion of multimodal sample data from unstructured to structured. The specific steps are:

[0060] The first step, sample collection, is implemented by the sample collection module. The workflow of the sample collection module includes the following steps:

[0061] Step 1: Add a new data source, configure the data source, and test whether it is connected normally;

[0062] Step 2: Configure the relevant parameters of the collection task, such as collection method, collection time, collection elements and other information;

[0063] Step 3: Collect data and monitor the status of the collection task;

[0064] Step 4: Complete data collection and store it in the sample library.

[0065] At the same time, the sample collection module can also achieve sample augmentation by transforming and processing the original data (such as rotation, scaling, translation, etc. for images; synonym replacement, random insertion, random exchange, etc. for text data), thereby expanding the number and diversity of training samples and improving the generalization ability and accuracy of the model.

[0066] In the embodiment of the present invention, a safe, portable and efficient technical tool is used to complete the one-stop collection of sample data for the Internet of Things, industry applications, academic research, etc. from multiple sample sources such as public data sets, books, business databases, and special corpora by purchasing data sets, downloading, web crawling, and manual entry. The workflow of the sample collection module is as follows: Figure 3 shown.

[0067] The second step, sample storage, is completed by the sample storage module. The workflow of the sample storage module includes the following steps:

[0068] Step 1: Create the corresponding storage space or table structure according to the collection task configuration information;

[0069] Step 2: Collect data into the database and generate corresponding metadata;

[0070] Step 3: Sample data is taken out of the database and corresponding samples are extracted according to the connection task.

[0071] In the embodiment of the present invention, two types of sample libraries and metadata systems are constructed: original samples and high-quality samples. Rapid adaptation and integration with object storage, relationship library, analysis library and search service are supported. Metadata collection and integration capabilities for multi-source heterogeneous data storage are provided. The workflow of the sample storage module is as follows: Figure 4 shown.

[0072] The third step, sample analysis, is completed by the sample processing module. The steps of sample analysis are:

[0073] Step 1: Extract samples from the original sample library.

[0074] Step 2: Perform sample analysis and process them according to the file type: txt, word, etc. are converted into sample data in a standard format through data standardization; pdf, ceb, etc. are recognized through OCR to obtain the content in the file and then converted into sample data in a standard format; image types are resized, cropped, or format converted; video files are decoded into video streams and audio streams, and video streams are decoded, framed, and data standardized; audio type sample analysis mainly performs audio data standardization, such as cutting into multiple audio segments of fixed length or converting into text data through language recognition. The audio stream is stored in the original sample library as audio sample data. The standard format can be a user-specified or user-defined format, and the standardization process converts the file into a specified or defined format.

[0075] The fourth step is sample cleaning, which is completed by the sample processing module. The steps of sample cleaning are:

[0076] Step 1: Clean the sample data in standard format: Use automated tools for preliminary screening, including removing noise data, duplicate data, and related confidential information. Then perform key extraction, topic analysis, labeling, etc. on the sample data to achieve deep mining, information extraction, structured representation, and value enhancement of the sample data;

[0077] In the embodiment of the present invention, the sample cleaning of text type mainly carries out noise reduction, deduplication, label marking and feature extraction; the sample cleaning of picture type mainly carries out noise reduction, deduplication, conversion (rotation, flipping, etc.), label marking and feature extraction; the sample cleaning of video type mainly carries out noise reduction, conversion (cropping, rotation, flipping, etc.), label marking and feature extraction; the sample cleaning of audio type mainly carries out noise reduction, conversion (language to text, etc.), label marking and feature extraction.

[0078] Step 2: The cleaned sample data and information such as labels and topics are written into the high-quality sample library to achieve the conversion of sample data from unstructured text to structured data. The cleaned sample data is stored in the high-quality sample library for pre-training of large models and can also be transmitted to the sample quality inspection link.

[0079] The fifth step is sample quality inspection, which is completed by the sample processing module. The steps of sample quality inspection are:

[0080] Step 1: Set the sampling ratio according to actual needs, and extract cleaned samples according to the ratio for sample quality inspection. Samples can be extracted from a high-quality sample library, or sample quality inspection can be continued after sample cleaning;

[0081] Step 2: Carry out sample quality inspection, including the overall process of sample distribution, manual quality inspection, content optimization and content evaluation. Sample distribution is to assign sample data to corresponding quality inspectors. Manual quality inspection and content optimization are to conduct detailed inspection and improvement of samples. Content evaluation is to score the sample content quality. For example, low-quality samples can be automatically deleted or blocked by setting thresholds.

[0082] Step 3: Store the quality inspection information (quality inspector, quality inspection time, sample quality score, etc.) in the high-quality sample library. The samples after quality inspection can be used as training data for fine-tuning the large model.

[0083] For the above four types of samples, a portion of sample data is selected in each batch according to the sampling ratio for sample distribution, and then distributed to multiple quality inspectors for manual quality inspection. After the quality inspection, experts optimize and evaluate the content, and finally form high-quality SFT (Supervised Fine-Tuning) samples stored in the high-quality sample library for supervised fine-tuning and reinforcement learning of large models. The workflow of the sample processing module is as follows: Figure 5 shown.

[0084] The above technical solutions are all standardized preprocessing processes provided by the system. Optionally, users can also quickly complete the customized configuration of the process by customizing the process.

[0085] The sixth step, sample management, is completed by the sample management module. Sample management includes the following steps:

[0086] Step 1: Create or modify the sample directory based on the sample library and user needs, so that users can query the sample directory and visualize the required sample content.

[0087] Step 2: Manage data collection tasks and sample processing tasks to achieve periodic scheduling and real-time monitoring. Manage sample service processes and tasks to ensure task stability.

[0088] Step 3: Sample users can submit sample collection and processing requirements through the workbench based on query results and sample requirements. After the requirements are responded to, they can apply for samples and, after application, they can feedback any problems with the samples to the sample construction team.

[0089] In the embodiment of the present invention, sample management is carried out based on sample metadata to build a complete sample data operation process. By maintaining the original and high-quality sample directory, intuitive sample display and retrieval capabilities are provided to sample users. One-stop sample service processes such as supporting demand submission, sample application, and problem feedback are provided to support sample users to use them conveniently. By managing the entire process of sample collection, processing, and service, the stability and reliability of sample data are guaranteed. The workflow of the sample management module is as follows: Figure 6 shown.

[0090] The large model sample preprocessing system provided by the present invention covers the entire process from sample collection, storage, processing to management. Through automated and process-based processing methods, it significantly improves the quantity, quality and diversity of sample data, and effectively supports large model scenario applications.

[0091] A large model sample preprocessing method according to the present invention is implemented based on the large model sample preprocessing system, and the method comprises the following steps:

[0092] The sample collection module collects raw data from multiple sample sources and stores the raw data in the raw sample library provided by the sample storage module;

[0093] The sample processing module is used to obtain the collected raw data from the original sample library, and different types of sample data are analyzed, cleaned and quality-checked according to the file type. Sample cleaning mainly includes deduplication, classification, noise reduction, topic analysis and labeling, and the cleaned sample data, label and topic information, and quality inspection information are stored in the high-quality sample library provided by the sample storage module;

[0094] Based on user needs, the sample management module provides functions such as creating / modifying sample catalogs, submitting requirements, sample visualization, and sample application, so as to achieve maintenance, retrieval, and display of original and high-quality samples.

[0095] The present invention also provides a computer device, comprising one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of a preprocessing method for a large model sample as described above are implemented.

[0096] The present invention also provides a computer storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the preprocessing method of a large model sample as described above are implemented.

[0097] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems (systems), computer devices or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0098] The present invention is described with reference to a flowchart of a method according to an embodiment of the present invention. It should be understood that each process in the flowchart and a combination of processes in the flowchart can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A system that specifies functions within a process or multiple processes.

[0099] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction system, which is implemented in the process Figure 1 A function specified in a process or multiple processes.

[0100] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 The steps of a specified function in a process or multiple processes.

Claims

1. A preprocessing method for a large model sample, characterized in that: The method is implemented based on a preprocessing system of a large model sample and comprises the following steps: The sample collection module collects raw data from multiple sample sources and stores the raw data in the raw sample library provided by the sample storage module; The sample processing module is used to obtain the collected raw data from the original sample library, and different types of sample data are analyzed, cleaned and quality-checked according to the file type. Sample cleaning mainly includes deduplication, classification, noise reduction, topic analysis and labeling, and the cleaned sample data, label and topic information, and quality inspection information are stored in the high-quality sample library provided by the sample storage module; Based on user needs, the sample management module provides functions such as creating / modifying sample catalogs, submitting requirements, sample visualization, and sample application, so as to achieve maintenance, retrieval, and display of original and high-quality samples.

2. The method according to claim 1, characterized in that The workflow of the sample collection module includes: Step 1-1: Add a new data source, configure the data source, and test whether it is connected normally; Step 1-2: Configure the relevant parameters of the collection task, including collection method, collection time, and collection element information; Step 1-3: Perform data collection and monitor the status of the collection task; Step 1-4: Complete data collection and store it in the original sample library.

3. The method according to claim 1, characterized in that The workflow of the sample storage module includes: Step 2-1: Create corresponding storage space or table structure according to the collection task configuration information; Step 2-2: Collect data into the database and generate corresponding metadata; Step 2-3: Sample data is taken out of the database and corresponding samples are extracted according to the connection task.

4. The method according to claim 1, characterized in that The workflow of the sample processing module includes: Step 3-1: Extract samples from the original sample library; Step 3-2: Perform sample analysis and process them according to the file type: txt and word files are processed through data standardization and converted into sample data in a standard format; pdf and ceb files are recognized through OCR to obtain the content in the file and then converted into sample data in a standard format; image files are resized, cropped or format converted; video files are decoded into video streams and audio streams, and the audio streams are stored in the original sample library as audio sample data; Step 3-3: Clean the sample data in the standard format: Use automated tools for preliminary screening, including removing noise data, duplicate data, and related confidential information, and then perform key extraction, topic analysis, and labeling on the sample data; Step 3-4: The processed sample data, labels, and subject information are written into a high-quality sample library to achieve the conversion of sample data from unstructured text to structured data; Step 3-5: Set the sampling ratio according to actual needs, and extract the cleaned samples according to the ratio for sample quality inspection; Step 3-6: Carry out sample quality inspection, including the overall process of sample distribution, manual quality inspection, content optimization and content evaluation. Sample distribution is to assign sample data to corresponding quality inspectors, manual quality inspection and content optimization are to conduct detailed inspection and improvement of samples, and content evaluation is to score the sample content quality. Step 3-7: Store the quality inspection information into the high-quality sample library. The quality inspection information includes the quality inspector, quality inspection time, and sample quality score.

5. The method according to claim 4, characterized in that In step 3-3, the sample cleaning of text type mainly involves noise reduction, deduplication, label marking and feature extraction; the sample cleaning of image type mainly involves noise reduction, deduplication, conversion, label marking and feature extraction; the sample cleaning of video type mainly involves noise reduction, conversion, label marking; the sample cleaning of audio type mainly involves noise reduction, conversion, label marking and feature extraction.

6. The method according to claim 4, characterized in that In steps 3-5, samples are extracted from a high-quality sample library, or sample quality inspection is performed after sample cleaning.

7. The method according to claim 4, characterized in that In steps 3-6, content evaluation also includes automatically deleting or blocking low-quality samples whose content quality scores are lower than the specified threshold by setting a threshold.

8. A preprocessing system for large model samples, characterized in that: The sample collection module is used to collect raw data from multiple sample sources and store the collected raw data in the raw sample library provided by the sample storage module; The sample storage module provides an original sample library and a high-quality sample library. The original sample library is used to store the original data collected by the sample collection module, and the high-quality sample library is used to store the sample data, labels, subject information, and quality inspection information after being cleaned by the sample processing module. The sample processing module is used to obtain the collected raw data from the original sample library, and perform corresponding analysis, cleaning and quality inspection on different types of sample data according to the file type. Sample cleaning mainly includes deduplication, classification, noise reduction, topic analysis and labeling, and the cleaned sample data, label and topic information, and quality inspection information are stored in the high-quality sample library built by the sample storage module; The sample management module is used to maintain, retrieve and display original and high-quality samples based on user needs through the creation / modification of sample catalogs, demand submission, sample visualization and sample application functions provided by this module.

9. A computer device, characterized in that: The device includes one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of a preprocessing method for a large model sample as described in any one of claims 1 to 7 are implemented.

10. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a method for preprocessing a large model sample according to any one of claims 1 to 7 are implemented.

Images