Large-scale data set processing method and device, electronic equipment and storage medium

By selecting the data import method in the data management interface and processing large-scale data sets in combination with data cloning and ID replacement, the problems of slow processing speed and low storage performance in the existing technology are solved, and the flexibility and efficiency of data processing are achieved. Improves the speed of data retrieval and storage.

CN119088868APending Publication Date: 2024-12-06GUANGZHOU QUYAN NETWORK TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411286171.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-13
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The existing large-scale data set processing methods face the problems of slow processing speed, low storage performance, and inflexible data processing when facing the ever-growing data scale and complexity.

Method used

Provide a large-scale data set processing method, which selects data import methods in the data management interface, processes the sampled data in combination with data cloning and ID replacement, and stores the processed data using preset storage methods, and supports multiple data import methods For example, the platform already has data set import, local import, SQL import and URL import, which store metadata and actual data separately, and use file preloader for concurrent loading and data retrieval.

Benefits of technology

It realizes the flexibility and efficiency of data processing, improves the speed of data retrieval and storage, reduces storage costs, and enhances the flexibility and efficiency of data processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119088868A_ABST
    Figure CN119088868A_ABST
Patent Text Reader

Abstract

The invention discloses a large-scale data set processing method and device, electronic equipment and a storage medium, which are used for solving the technical problems that the existing large-scale data set processing is not flexible enough and the storage optimization effect is not good enough. The method comprises the steps of determining a data import mode in response to a selection operation for an import mode in a data management interface; when the data import mode is platform existing data set import, sampling conditions are determined according to sampling combination selection operation for data configuration options in the data management interface, and a sampling version is generated according to the sampling conditions; whether an existing sampling table exists in the sampling version or not is judged, and if yes, sampling data in the existing sampling table are retrieved in batches; and processing the sampled data in combination with data cloning and ID replacement, and storing the processed sampled data in a preset storage mode.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a large-scale data set processing method, device, electronic device and storage medium. Background Art

[0002] With the widespread application of machine learning and deep learning, large-scale language models, such as GPT-3 (Generative Pre-trained Transformer 3, the third generation of pre-trained language model), BERT (Bidirectional Encoder Representations from Transformers, a pre-trained language model), etc., have performed better and better in natural language processing tasks. In order to train these complex models, a large number of high-quality data sets are required. These data sets usually contain massive amounts of text, pictures, audio and other data. Not only are the capacity huge, but also the types are numerous. Before using these data for model training and evaluation, in order to ensure the quality and diversity of the data, they must be effectively processed and labeled. In addition, these data need to be properly stored for quick access and efficient processing.

[0003] The processing methods of large-scale data sets mainly include data collection, data cleaning, data labeling, data storage and data distribution. In the process of data processing, a variety of technical means are usually involved. For example, text processing, image processing, audio processing, etc. In order to improve processing efficiency, some methods also use parallel computing and distributed storage technology. However, in the face of the growing scale and complexity of data, traditional data processing methods have gradually exposed problems such as slow processing speed, low storage performance, and inflexible data processing. At present, the means involved in large-scale data set processing mainly include Apache Hadoop (an open source framework for large-scale data set storage and processing), Apache Spark (a fast and general cluster computing system) and TFX (TensorFlow Extended, an end-to-end platform for production-level machine learning workflows).

[0004] Among them, Hadoop uses the MapReduce (a programming model and related software framework) programming model, which can efficiently process large-scale data sets. However, Hadoop focuses more on data storage and batch processing, and its support for real-time data processing and complex data labeling is relatively limited. Spark's in-memory computing capabilities enable it to perform well in processing large-scale data sets and support real-time data processing. However, Spark still has room for improvement in data labeling and storage optimization. TFX covers the entire process of data preprocessing, model training, and model deployment. However, TFX is mainly aimed at the pipeline management of machine learning models, and its support for detailed processing and labeling of large-scale data sets is relatively preliminary. Summary of the invention

[0005] The present invention provides a large-scale data set processing method, device, electronic device and storage medium, which are used to solve or partially solve the technical problems that the existing large-scale data set processing is not flexible enough and the storage optimization effect is not good enough.

[0006] The present invention provides a large-scale data set processing method, the method comprising:

[0007] In response to a selection operation on an import method in a data management interface, determining a data import method;

[0008] When the data import method is to import an existing data set on the platform, a sampling condition is determined according to a sampling combination selection operation for a data configuration option in the data management interface, and a sampling version is generated according to the sampling condition;

[0009] Determine whether there is an existing sampling table in the sampling version, and if so, retrieve the sampling data in the existing sampling table in batches;

[0010] The sampled data is processed in combination with data cloning and ID replacement, and the processed sampled data is stored in a preset storage method.

[0011] Optionally, the processing of the sampled data by combining data cloning and ID replacement, and storing the processed sampled data in a preset storage method, includes:

[0012] Extracting a list of file storage paths to which the sampled data belongs;

[0013] Constructing and starting a file preloader, and concurrently loading a plurality of first splicing-related data corresponding to the file storage path list through the file preloader, each of the first splicing-related data corresponding to a first globally unique ID and an ID-data mapping relationship;

[0014] Acquire existing imported data of the first globally unique ID according to the ID-data mapping relationship, where the existing imported data corresponds to the first metadata;

[0015] Performing data cloning on all the first metadata to obtain cloned metadata of each piece of the first metadata;

[0016] For each piece of the existing imported data, a corresponding second globally unique ID is regenerated, and the ID of the cloned metadata is replaced based on the second globally unique ID to obtain second metadata;

[0017] splicing the second globally unique ID with the actual content of the existing imported data to perform data association and obtain second spliced ​​associated data;

[0018] Writing the second metadata into a first metadata buffer area, and writing the second splicing-related data into a first I / O buffer area corresponding to the storage path of the existing imported data;

[0019] When the amount of second splicing-related data cached in the first I / O cache area reaches a preset amount threshold, or when no new write data is detected within a preset time interval, the second metadata cached in the first metadata cache area is batch-written to the database, and the second splicing-related data cached in the first I / O cache area is written to the cloud storage disk.

[0020] Optionally, the method further comprises:

[0021] When it is determined that there is no existing sampling table for the sampling version, creating a new sampling table;

[0022] Data is retrieved according to the sampling condition, and the retrieved sampling data is imported into the new sampling table.

[0023] Optionally, the method further comprises:

[0024] Generate a sampling data overview according to a current sampling table, and generate a current data overview of a current data set version that has not been sampled, wherein the current sampling table is an existing sampling table or a new sampling table;

[0025] The sampling data overview and the current data overview are combined to form a final data overview of the sampling conditions, and are displayed in the data management interface in a preview form.

[0026] Optionally, the data importing method further includes local importing, SQL importing and URL importing, and the method further includes:

[0027] When the data import mode is local import, obtaining a local file uploaded by the user as the first file to be imported;

[0028] When the data import method is SQL import, obtaining the SQL statement input by the user, and exporting the data according to the SQL statement as the second file to be imported;

[0029] When the data import method is URL import, obtaining the URL address input by the user, and downloading the data file of the URL address as the third file to be imported;

[0030] Convert the first file to be imported, and / or the second file to be imported, and / or the third file to be imported into a JSONL file;

[0031] Data processing is performed on the JSONL file to obtain processed imported data, and the processed imported data is stored according to a preset storage rule.

[0032] Optionally, the processed imported data includes a plurality of pieces of data to be imported, and storing the processed imported data according to a preset storage rule includes:

[0033] For each piece of data to be imported, generate a corresponding third globally unique ID;

[0034] Based on the third globally unique ID and in combination with tag information of the data to be imported, construct third metadata corresponding to the data to be imported, wherein the tag information includes a storage path, token statistics, and a label of the data to be imported;

[0035] splicing the third globally unique ID and the actual content of the data to be imported to perform data association to obtain third spliced ​​associated data;

[0036] Writing the third metadata into a second metadata buffer area, and writing the third splicing-related data into a second I / O buffer area corresponding to the storage path of the data to be imported;

[0037] When the cumulative amount of the third splicing-related data cached in the second I / O cache area reaches a preset quantity threshold, or when no new write data is detected within a preset time interval, the third metadata cached in the second metadata cache area is batch-written to the database, and the third splicing-related data cached in the second I / O cache area is written to the cloud storage disk.

[0038] Optionally, the method further comprises:

[0039] Receive a data retrieval instruction, and extract from a database a plurality of target metadata corresponding to the data retrieval instruction and a target file path list corresponding to the plurality of target metadata;

[0040] Construct and start a file preloader, and concurrently load multiple target splicing associated data corresponding to the target file path list through the file preloader, each of the target splicing associated data corresponds to a target global unique ID and a target ID-data mapping relationship;

[0041] According to the target ID-data mapping relationship, the actual search content corresponding to the target global unique ID is extracted from the target splicing associated data to obtain the search result.

[0042] The present invention also provides a large-scale data set processing device, comprising:

[0043] A data import mode determination module, used to determine a data import mode in response to a selection operation on an import mode in a data management interface;

[0044] A sampling version generation module, used for determining sampling conditions according to the sampling combination selection operation for the data configuration options in the data management interface when the data import mode is to import the existing data set of the platform, and generating a sampling version according to the sampling conditions;

[0045] A sampling data batch retrieval module is used to determine whether there is an existing sampling table for the sampling version, and if so, to batch retrieve the sampling data in the existing sampling table;

[0046] The data processing and storage module is used to process the sampled data in combination with data cloning and ID replacement, and to store the processed sampled data in a preset storage method.

[0047] The present invention also provides an electronic device, the device comprising a processor and a memory:

[0048] The memory is used to store program code and transmit the program code to the processor;

[0049] The processor is used to execute any of the large-scale data set processing methods described above according to the instructions in the program code.

[0050] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the large-scale data set processing method as described in any one of the above items.

[0051] It can be seen from the above technical solutions that the present invention has the following advantages:

[0052] A large-scale data set processing method is provided. First, in response to the selection operation of the import method in the data management interface, the data import method is determined; when the data import method is to import the existing data set of the platform, the sampling conditions are determined according to the sampling combination selection operation of the data configuration option in the data management interface, and the sampling version is generated according to the sampling conditions; then it is determined whether there is an existing sampling table in the sampling version, and if so, the sampling data in the existing sampling table is retrieved in batches; then the sampling data is processed in combination with data cloning and ID replacement, and the processed sampling data is stored in a preset storage method. Therefore, by importing the sampling combination of the existing data set of the platform, the data set required for model training and evaluation can be quickly retrieved without regenerating a new data set, making the processing of the data set more flexible and efficient. Furthermore, by optimizing the data storage in combination with data cloning, ID replacement and preset storage methods, better storage effects can be achieved, making it more convenient and quick when the subsequent process needs to retrieve or call data, thereby improving data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 A flowchart of the steps of a large-scale data set processing method;

[0055] Figure 2 A schematic diagram of the preview process of sampling and combining existing data sets on a platform;

[0056] Figure 3 A schematic diagram of a data overview of the current dataset version;

[0057] Figure 4 A schematic diagram of the final data overview of the sampling conditions generated after the merging process;

[0058] Figure 5 A schematic diagram of the import process of sampling and combining existing data sets on a platform;

[0059] Figure 6 is a schematic diagram of an import process of data to be imported;

[0060] Figure 7 A schematic diagram of a data storage process for data to be imported;

[0061] Figure 8 It is a schematic diagram of a quick retrieval and data reading process;

[0062] Fig. 9 The figure is a structural block diagram of a large-scale data set processing device. DETAILED DESCRIPTION

[0063] The embodiments of the present invention provide a large-scale data set processing method, device, electronic device and storage medium, which are used to solve or partially solve the technical problems that the existing large-scale data set processing is not flexible enough and the storage optimization effect is not good enough.

[0064] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] As an example, currently, the means of processing large-scale data sets mainly include Apache Hadoop, Apache Spark and TFX. Among them, Hadoop adopts the MapReduce programming model, which can efficiently process large-scale data sets. However, Hadoop focuses more on data storage and batch processing, and its support for real-time data processing and complex data labeling is relatively limited. Spark's in-memory computing capabilities enable it to perform well in processing large-scale data sets and support real-time data processing. However, Spark still has room for improvement in data labeling and storage optimization. TFX covers the entire process of data preprocessing, model training and model deployment. However, TFX is mainly aimed at the pipeline management of machine learning models, and its support for detailed processing and labeling of large-scale data sets is relatively preliminary.

[0066] Therefore, one of the core inventive points of the embodiment of the present invention is: in view of the deficiencies in the prior art, a method for processing large-scale data sets is proposed. First, a variety of data import methods are provided, including importing existing data sets on the platform, local import, SQL (Structured Query Language) import, and URL (Uniform Resource Locator) import, so that through diversified data import methods, the system can easily integrate data from different sources and improve the flexibility and efficiency of data collection. Secondly, metadata and actual data are stored separately, so that when data is retrieved and read, data can be quickly retrieved through metadata, and the actual content of the data can be quickly retrieved through the file preloader, which can reduce storage costs and retrieve data efficiently.

[0067] In particular, when the data import method is to import the existing data sets on the platform, the data sets required for model training and evaluation can be quickly retrieved by importing the sampling combination of the existing data sets on the platform, without the need to regenerate new data sets, making the processing of data sets more flexible and efficient. Furthermore, by optimizing data storage by combining data cloning, ID replacement and preset storage methods, better storage effects can be achieved, making it more convenient and faster to retrieve or call data in subsequent processes, thereby improving data processing efficiency.

[0068] Reference Figure 1 , shows a flowchart of a large-scale data set processing method provided by an embodiment of the present invention, which may specifically include the following steps:

[0069] Step 101, in response to a selection operation on an import method in a data management interface, determining a data import method;

[0070] In actual applications, the data management interface (also known as the dataset version management page) provides an import button. Users can enter the data import page by clicking the import button. The system provides multiple data import methods, and correspondingly sets multiple import buttons. Such as local import, SQL import, URL import, import of existing datasets on the platform, etc.

[0071] Step 102, when the data import method is to import an existing data set on the platform, determine the sampling conditions according to the sampling combination selection operation for the data configuration options in the data management interface, and generate a sampling version according to the sampling conditions;

[0072] The platform has built-in rich labeled data sets. Users can choose to combine and generate new data sets according to actual needs. For example, filter and combine by data type, label category, scenario, label, sampling ratio, sampling quantity and other conditions. In other words, the platform can preset a variety of data sets. Through the platform data set import, users can quickly build data sets including but not limited to training scenarios and model evaluation scenarios according to sampling conditions. In addition, the platform also supports previewing and generating combined data visualization charts, and automatically identifies which label data is too high, so that users can easily judge whether the current conditions meet user expectations. The platform's existing data set import page provides the ability to sample data from multiple data sets according to sampling conditions such as labels and data content to combine into new labels and generate sampled versions. When the import is confirmed, the data is actually copied according to the user's sampling conditions for subsequent use.

[0073] Step 103, determining whether there is an existing sampling table in the current sampling version, and if so, batch searching the sampling data in the existing sampling table;

[0074] When it is determined that an existing sampling table currently exists for the sampling version (ie, a sampling version has been generated based on the same sampling condition before, and a corresponding sampling table has been created and stored), the sampling data in the existing sampling table is retrieved in batches.

[0075] In another case, when it is determined that the sampling version does not currently have an existing sampling table, a new sampling table is created, data is retrieved according to the sampling conditions, and the retrieved sampling data is imported into the new sampling table.

[0076] As an optional embodiment, the system also provides a function of real-time preview of the sampling data overview.

[0077] Specifically, first, a sampling data overview can be generated based on the current sampling table, and at the same time, a current data overview of the current data set version that has not been sampled can be generated. Among them, the current sampling table is an existing sampling table, or a new sampling table. Then the sampling data overview and the current data overview can be merged to form a final data overview of the sampling conditions, and displayed in the data management interface in a preview form. In this way, not only can the data set overview be viewed conveniently and quickly to achieve data visualization, but also the data indicators of the current data set can be intelligently annotated, including abnormal points of session type, scene label, content label, and main content label.

[0078] For example, Figure 2 A schematic diagram of a preview process of sampling and combining existing data sets on a platform provided by an embodiment of the present invention is shown. Specifically, it may include steps 201 to 209:

[0079] Step 201: The user selects sampling conditions on the existing data set import page of the platform (i.e., the data configuration option interface in the data management interface);

[0080] Step 202: The server generates a sampling version by using a HASH algorithm according to the sampling conditions;

[0081] Step 203: Determine whether there is an existing sampling table in the sampling version; if so, directly jump to step 207; if not, execute step 204;

[0082] Step 204: Create a new sampling table;

[0083] Step 205: Retrieve data according to sampling conditions;

[0084] Step 206: Import the retrieved sampling data into a new sampling table;

[0085] Step 207: Generate a sampling data overview according to the sampling table;

[0086] Step 208: Generate a current data overview of the current data set version;

[0087] Step 209: Merge the sampling data overview and the current data overview to form a final data overview of the sampling conditions.

[0088] The current dataset version may already contain some data. For example, when you select 0 to sample, the current dataset version already has 1,000 data items. The data overview diagram is as follows: Figure 3 As shown in the figure, the vertical axis in each bar graph represents the number of data containing the corresponding label, and the horizontal axis represents the label.

[0089] Furthermore, assuming that the data of the task scenario label source_3 is in line with expectations, we want to import 20% of the data of source_3 based on this version. After setting the sampling conditions, 175 data items containing 20% ​​of the source_3 label are sampled. Together with the original 1000 items, the total number is 1175. The final data overview diagram of the sampling conditions generated after the merging process is as follows: Figure 4 shown.

[0090] Step 104, the sampled data is processed in combination with data cloning and ID replacement, and the processed sampled data is stored in a preset storage method.

[0091] Specifically, the sampled data is processed in combination with data cloning and ID replacement, and the processed sampled data is stored in a preset storage method, which can be achieved by executing the following sub-steps S01 to S08:

[0092] Step S01: extracting a list of file storage paths to which sample data belongs;

[0093] Step S02: construct and start a file preloader, and concurrently load multiple first splicing-related data corresponding to the file storage path list through the file preloader, each of which corresponds to a first globally unique ID and an ID-data mapping relationship;

[0094] Step S03: Obtain the existing imported data of the first globally unique ID according to the ID-data mapping relationship, and the existing imported data corresponds to the first metadata; wherein metadata refers to the description information of the data, and for a data set, each metadata includes but is not limited to data ID, token statistics, labels, actual data content storage path, etc.

[0095] Step S04: performing data cloning on all first metadata to obtain cloned metadata of each first metadata;

[0096] Step S05: for each piece of existing imported data, a corresponding second globally unique ID is regenerated, and the ID of the cloned metadata is replaced based on the second globally unique ID to obtain second metadata;

[0097] Step S06: splicing the second globally unique ID with the actual content of the existing imported data to perform data association and obtain second splicing associated data;

[0098] Step S07: writing the second metadata into the first metadata buffer area, and writing the second splicing-related data into the first I / O buffer area corresponding to the storage path of the existing imported data;

[0099] Step S08: When the amount of second splicing-related data cached in the first I / O cache area reaches a preset amount threshold, or when no new write data is detected within a preset time interval, the second metadata cached in the first metadata cache area is batch-written to the database, and the second splicing-related data cached in the first I / O cache area is written to the cloud storage disk.

[0100] In combination with the above introduction, illustratively, Figure 5 A schematic diagram of a process for importing a sampling combination of existing data sets on a platform provided by an embodiment of the present invention is shown. Specifically, it may include steps 301 to 3013:

[0101] Step 301: The user selects sampling conditions on the existing data set import page of the platform (i.e., the data configuration option interface in the data management interface);

[0102] Step 302: The server generates a sampling version by using a HASH algorithm according to the sampling conditions;

[0103] Step 303: Determine whether there is an existing sampling table in the sampling version; if so, directly jump to step 307; if not, execute step 304;

[0104] Step 304: Create a new sampling table;

[0105] Step 305: Retrieve data according to sampling conditions;

[0106] Step 306: Import the retrieved sampling data into a new sampling table;

[0107] Step 307: retrieve sample data in batches;

[0108] Step 308: extracting a storage file path list to which the sampled data belongs;

[0109] Step 309: Build and start a file preloader to load files concurrently;

[0110] Among them, the file preloader is a technology that can improve application performance and supports loading multiple files at the same time. Its purpose is to reduce application startup time and file access latency by loading key files into memory in advance. In actual data processing, the reasonable use of file preloaders can significantly improve application response speed and user experience.

[0111] Step 3010: concurrently load multiple pieces of splicing-related data corresponding to the file path list through a file preloader, and store the mapping relationship between the ID and the actual content of the data in the memory;

[0112] Step 3011: clone the sampled data and replace the ID to avoid duplication of the ID with other data sets; use the old ID to obtain the actual content of the data loaded in step 3010 and splice the new ID;

[0113] Step 3012: write the metadata into the metadata cache, and write the actual content of the data with the new ID spliced ​​into the I / O cache corresponding to the storage path;

[0114] Step 3013: When the cumulative number of data reaches 10,000, or there is no other written data within the preset time interval, the metadata cached in the metadata cache is batch written to the database (as the current dataset version table), and the data files cached in the I / O cache are written to the cloud storage disk.

[0115] As an optional embodiment, the data import method may also include local import, SQL import, and URL import. Local import means that the user selects a local data set file to upload, the server receives the file, and the uploaded file is uniformly processed into a JSONL (JavaScript Object Notation Lines, a data exchange format) file. SQL import means that the server collects data from the data platform according to SQL instructions and exports it as a JSONL file. URL import means that the server downloads the corresponding open source data set file according to the URL address and converts it into a JSONL file.

[0116] In addition, the data import method can also be adjusted according to the actual data source. In the embodiment of the present invention, not only local upload, platform existing data sampling combination import, SQL import and URL import are supported, but also import through data governance platform and other methods. It is understandable that the present invention does not limit this.

[0117] Then, you can label the data in combination with the functional annotations and add a unique ID to each piece of data. In terms of storage, you can use a database to store IDs and tags, token statistics, file paths to which data content belongs, and other related information (metadata), and use cloud storage to store IDs and actual data content information. Labeling data in combination with functional annotations refers to the process of labeling data according to different functional or task requirements.

[0118] Specifically, Figure 6 The following is a schematic diagram of a process flow for importing data to be imported provided by an embodiment of the present invention. Specifically, it includes steps 401 to 404:

[0119] Step 401: The user selects a corresponding method to upload a file;

[0120] When the data import method is local import, the user selects a local file, and the server obtains the local file uploaded by the user as the first file to be imported.

[0121] When the data import method is SQL import, the user enters the SQL statement, the server obtains the SQL statement entered by the user, and exports the data according to the SQL statement and stores it as a JSONL file, and downloads the file as the second file to be imported.

[0122] When the data import method is URL import, the user enters the URL address of the data set, the server obtains the URL address entered by the user, and downloads the data file at the URL address as the third file to be imported.

[0123] Step 402: After the server obtains the uploaded file, it converts the file into a JSONL file;

[0124] The first file to be imported, and / or the second file to be imported, and / or the third file to be imported are uniformly converted into JSONL files.

[0125] Step 403: Use the injected plug-in to process data;

[0126] Process the JSONL file and import the processed data. The processing plug-in referred to in the present invention refers to a processing function. For example, denoising according to rules (removing garbled characters, expressions, etc. in the text), formatting, etc. It can also be an http request for requesting that the current data be transferred to the microservice for data processing. It can also be a python script, etc.

[0127] For data processing, the embodiment of the present invention mainly adopts a plug-in method. In practical applications, the data processing method can be adjusted according to the actual scenario. For example, other annotation platform replacement methods, or token calculation algorithms, and so on can also be used. Among them, token calculation can use the TikToken (a Python library for processing and analyzing text data) tool, and uniformly use GPT-3.5-Turbo encoding. However, in practical applications, different model manufacturers have their own encoding methods, and those skilled in the art can make adjustments according to actual conditions, and the present invention does not limit this.

[0128] Step 404: Store the processed imported data according to preset storage rules.

[0129] Furthermore, the processed imported data may include multiple pieces of data to be imported. The process of storing the processed imported data according to the preset storage rules may be implemented by executing the following sub-steps S11 to S15:

[0130] S11: For each piece of data to be imported, generate a corresponding third globally unique ID;

[0131] S12: constructing third metadata corresponding to the data to be imported based on the third globally unique ID and in combination with tag information of the data to be imported, where the tag information includes a storage path, token statistics, and a label of the data to be imported;

[0132] S13: splicing the third globally unique ID and the actual content of the data to be imported to perform data association to obtain third splicing associated data;

[0133] S14: writing the third metadata into the second metadata buffer area, and writing the third splicing-related data into the second I / O buffer area corresponding to the storage path of the data to be imported;

[0134] S15: When the amount of the third splicing-related data cached in the second I / O cache area reaches a preset amount threshold, or when no new write data is detected within a preset time interval, the third metadata cached in the second metadata cache area is batch-written to the database, and the third splicing-related data cached in the second I / O cache area is written to the cloud storage disk.

[0135] Based on the above introduction, illustratively, Figure 7 A schematic diagram of a data storage process for data to be imported provided by an embodiment of the present invention is shown. Specifically, steps 501 to 505 are included:

[0136] Step 501: Obtain a single piece of data to be imported;

[0137] Step 502: Generate a globally unique ID for the data to be imported;

[0138] Step 503: construct metadata of the current data to be imported; concatenate an ID before the actual data content of the data to be imported for associating the data;

[0139] Step 504: write the metadata into the metadata cache, and write the actual data content with the spliced ​​ID into the file I / O cache corresponding to the storage path;

[0140] Step 505: When the accumulated data reaches 10,000, or there is no other written data within a preset time interval, the metadata cached in the metadata cache is batch-written to the database, and the data files cached in the I / O cache are written to the cloud storage disk.

[0141] Data is collected from a variety of sources and converted into a system-specified format based on the characteristics of data from different sources. At the same time, data is labeled through intelligent annotation, and a unique ID is added to each piece of data. In terms of storage, the database and distributed cloud storage are combined. The database uses a unique ID to associate data tags, and the storage file uses a unique ID to associate data content, thereby improving data retrieval efficiency and reducing storage costs. Therefore, through a variety of data import methods, the system can easily integrate data from different sources, improving the flexibility and efficiency of data collection.

[0142] At the same time, metadata is stored separately from actual data, and mongoDB (an open source document database) is used to store metadata in shard sets. Cloud storage is used to divide files into small files for storage. For data retrieval and reading, metadata can be used to quickly retrieve data, and the actual data content can be quickly retrieved through the file preloader, which can reduce storage costs and retrieve data efficiently.

[0143] Based on the above introduction, as an optional embodiment, the embodiment of the present invention also provides a method for fast retrieval and data reading.

[0144] Specifically, Figure 8 A schematic diagram of a fast search and data reading process provided by an embodiment of the present invention is shown. Specifically, it includes steps 601 to 605:

[0145] Step 601: Retrieve a metadata list based on tags or other conditions

[0146] The server can receive data retrieval instructions (such as tag search) input by the user and extract multiple target metadata corresponding to the data retrieval instructions from the database.

[0147] In addition, metadata can be retrieved by other conditions, such as dataset ID, dataset version, compliance, etc.

[0148] Step 602: extracting a list of storage file paths to which the metadata belongs;

[0149] The location where the original content of each piece of data is stored will be recorded in the database. Therefore, after searching for metadata, it is necessary to read the original data according to the file path. Thus, it is also possible to obtain a list of target file paths corresponding to multiple target metadata through retrieval and extraction.

[0150] Step 603: Build and start a file preloader for concurrently loading files. The preloader supports loading multiple files simultaneously.

[0151] Step 604: the preloader concurrently loads the files and stores the mapping relationship between the ID and the actual content of the data in the memory;

[0152] Build and start the file preloader, and concurrently load multiple target splicing associated data corresponding to the target file path list through the file preloader, each target splicing associated data corresponds to the target global unique ID and the target ID-data mapping relationship.

[0153] Step 605: Obtain the data content loaded in step 604 according to the ID to obtain the search result.

[0154] According to the target ID-data mapping relationship, the actual search content corresponding to the target global unique ID is extracted from the target splicing associated data to obtain the search results.

[0155] It should be noted that, in order to enable those skilled in the art to better distinguish data of the same type but with different actual meanings, the embodiments of the present invention use the first, second, and third terms to distinguish and illustrate some technical features. The first, second, and third terms are only used to distinguish data and have no other special meanings. It is understood that the present invention does not limit this.

[0156] In an embodiment of the present invention, a large-scale data set processing method is provided. First, a variety of data import methods are provided, including platform existing data set import, local import, SQL import and URL import, so that through the diversified data import methods, the system can easily integrate data from different sources and improve the flexibility and efficiency of data collection. Secondly, metadata and actual data are stored separately, so that when data retrieval and reading are performed, data can be quickly retrieved through metadata, and the actual content of the data can be quickly retrieved through the file preloader, which can reduce storage costs and retrieve data efficiently.

[0157] Reference Fig. 9 , shows a structural block diagram of a large-scale data set processing device provided by an embodiment of the present invention, which may specifically include:

[0158] The data import mode determination module 901 is used to determine the data import mode in response to the selection operation of the import mode in the data management interface;

[0159] The sampling version generating module 902 is used to determine the sampling conditions according to the sampling combination selection operation for the data configuration options in the data management interface when the data import mode is to import the existing data set of the platform, and generate the sampling version according to the sampling conditions;

[0160] The sampling data batch retrieval module 903 is used to determine whether there is an existing sampling table in the current sampling version, and if so, batch retrieve the sampling data in the existing sampling table;

[0161] The data processing and storage module 904 is used to process the sampled data in combination with data cloning and ID replacement, and store the processed sampled data in a preset storage method.

[0162] In an optional embodiment, the data processing and storage module 904 includes:

[0163] A file storage path list extraction module, used to extract the file storage path list to which the sample data belongs;

[0164] A first splicing associated data loading module is used to construct and start a file preloader, and concurrently load multiple first splicing associated data corresponding to the file storage path list through the file preloader, each of the first splicing associated data corresponds to a first globally unique ID and an ID-data mapping relationship;

[0165] An existing imported data acquisition module, used for acquiring the existing imported data of the first globally unique ID according to the ID-data mapping relationship, wherein the existing imported data corresponds to the first metadata;

[0166] A data cloning module, used to clone all the first metadata to obtain cloned metadata of each piece of the first metadata;

[0167] An ID replacement module, configured to regenerate a corresponding second globally unique ID for each piece of the existing imported data, and replace the ID of the cloned metadata based on the second globally unique ID to obtain second metadata;

[0168] A first data association module, used for splicing the second globally unique ID with the actual content of the existing imported data to perform data association and obtain second splicing associated data;

[0169] A first data cache module, used for writing the second metadata into a first metadata cache area, and writing the second splicing-related data into a first I / O cache area corresponding to the storage path of the existing imported data;

[0170] The data writing first module is used to batch write the second metadata cached in the first metadata cache area to the database and write the second splicing associated data cached in the first I / O cache area to the cloud storage disk when the cumulative amount of the second splicing associated data cached in the first I / O cache area reaches a preset amount threshold, or when no new write data is detected within a preset time interval.

[0171] In an optional embodiment, the device further includes:

[0172] A new sampling table creation module, used for creating a new sampling table when it is determined that there is no existing sampling table for the sampling version;

[0173] The sampling data import module is used to retrieve data according to the sampling conditions and import the retrieved sampling data into the new sampling table.

[0174] In an optional embodiment, the device further includes:

[0175] A data overview generation module, used to generate a sampling data overview according to a current sampling table, and to generate a current data overview of a current data set version that has not been subjected to sampling processing, wherein the current sampling table is an existing sampling table or a new sampling table;

[0176] The data preview module is used to merge the sampling data overview and the current data overview to form a final data overview of the sampling conditions, and display it in the data management interface in a preview form.

[0177] In an optional embodiment, the data import method further includes local import, SQL import and URL import, and the device further includes:

[0178] A local file import module, used to obtain a local file uploaded by a user as the first file to be imported when the data import mode is local import;

[0179] An SQL file import module, used for obtaining an SQL statement input by a user when the data import mode is SQL import, and exporting data according to the SQL statement as a second file to be imported;

[0180] A URL file import module, used for obtaining a URL address input by a user and downloading a data file at the URL address as a third file to be imported when the data import mode is URL import;

[0181] A file format conversion module, used for uniformly converting the first file to be imported, and / or the second file to be imported, and / or the third file to be imported into a JSONL file;

[0182] The import data storage module is used to perform data processing on the JSONL file, obtain the processed import data, and store the processed import data according to preset storage rules.

[0183] In an optional embodiment, the processed imported data includes a plurality of pieces of data to be imported, and the imported data storage module includes:

[0184] A third globally unique ID generating module, used for generating a corresponding third globally unique ID for each piece of data to be imported;

[0185] A third metadata construction module, configured to construct third metadata corresponding to the data to be imported based on the third globally unique ID and in combination with tag information of the data to be imported, wherein the tag information includes a storage path, token statistics, and a label of the data to be imported;

[0186] A second data association module is used to splice the third globally unique ID with the actual content of the data to be imported to perform data association and obtain third splicing association data;

[0187] A second data cache module, used for writing the third metadata into a second metadata cache area, and writing the third splicing-related data into a second I / O cache area corresponding to the storage path of the data to be imported;

[0188] The data is written into the second module, which is used to batch write the third metadata cached in the second metadata cache area into the database and write the third splicing associated data cached in the second I / O cache area into the cloud storage disk when the cumulative amount of the third splicing associated data cached in the second I / O cache area reaches a preset quantity threshold, or when no new write data is detected within a preset time interval.

[0189] In an optional embodiment, the device further includes:

[0190] A data retrieval module, used for receiving a data retrieval instruction, and extracting from a database a plurality of target metadata corresponding to the data retrieval instruction, and a target file path list corresponding to the plurality of target metadata;

[0191] A target splicing associated data loading module is used to build and start a file preloader, and concurrently load multiple target splicing associated data corresponding to the target file path list through the file preloader, each of which corresponds to a target global unique ID and a target ID-data mapping relationship;

[0192] The search content extraction module is used to extract the actual search content corresponding to the target global unique ID from the target splicing associated data according to the target ID-data mapping relationship to obtain the search result.

[0193] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the aforementioned method embodiment.

[0194] An embodiment of the present invention further provides an electronic device, the device comprising a processor and a memory:

[0195] The memory is used to store the program code and transmit the program code to the processor;

[0196] The processor is used to execute the large-scale data set processing method of any embodiment of the present invention according to the instructions in the program code.

[0197] An embodiment of the present invention further provides a computer-readable storage medium, which is used to store program code, and the program code is used to execute the large-scale data set processing method of any embodiment of the present invention.

[0198] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0199] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0200] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0201] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0202] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0203] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing large-scale data sets, It is characterized in that include: In response to a selection operation on an import method in a data management interface, determining a data import method; When the data import method is to import an existing data set on the platform, a sampling condition is determined according to a sampling combination selection operation for a data configuration option in the data management interface, and a sampling version is generated according to the sampling condition; Determine whether there is an existing sampling table in the sampling version, and if so, retrieve the sampling data in the existing sampling table in batches; The sampled data is processed in combination with data cloning and ID replacement, and the processed sampled data is stored in a preset storage method.

2. The large-scale data set processing method according to claim 1, It is characterized in that The processing of the sampled data by combining data cloning and ID replacement, and storing the processed sampled data in a preset storage method, includes: Extracting a list of file storage paths to which the sampled data belongs; Constructing and starting a file preloader, and concurrently loading a plurality of first splicing-related data corresponding to the file storage path list through the file preloader, each of the first splicing-related data corresponding to a first globally unique ID and an ID-data mapping relationship; Acquire existing imported data of the first globally unique ID according to the ID-data mapping relationship, where the existing imported data corresponds to the first metadata; Performing data cloning on all the first metadata to obtain cloned metadata of each piece of the first metadata; For each piece of the existing imported data, a corresponding second globally unique ID is regenerated, and the ID of the cloned metadata is replaced based on the second globally unique ID to obtain second metadata; splicing the second globally unique ID with the actual content of the existing imported data to perform data association and obtain second spliced ​​associated data; Writing the second metadata into a first metadata buffer area, and writing the second splicing-related data into a first I / O buffer area corresponding to the storage path of the existing imported data; When the amount of second splicing-related data cached in the first I / O cache area reaches a preset amount threshold, or when no new write data is detected within a preset time interval, the second metadata cached in the first metadata cache area is batch-written to the database, and the second splicing-related data cached in the first I / O cache area is written to the cloud storage disk.

3. The large-scale data set processing method according to claim 1, It is characterized in that Also includes: When it is determined that there is no existing sampling table for the sampling version, creating a new sampling table; Data is retrieved according to the sampling condition, and the retrieved sampling data is imported into the new sampling table.

4. The large-scale data set processing method according to claim 3, It is characterized in that include: Generate a sampling data overview according to a current sampling table, and generate a current data overview of a current data set version that has not been sampled, wherein the current sampling table is an existing sampling table or a new sampling table; The sampling data overview and the current data overview are combined to form a final data overview of the sampling conditions, and are displayed in the data management interface in a preview form.

5. The large-scale data set processing method according to claim 1, It is characterized in that The data importing method also includes local importing, SQL importing and URL importing, and the method also includes: When the data import mode is local import, obtaining a local file uploaded by the user as the first file to be imported; When the data import method is SQL import, obtaining the SQL statement input by the user, and exporting the data according to the SQL statement as the second file to be imported; When the data import method is URL import, obtaining the URL address input by the user, and downloading the data file of the URL address as the third file to be imported; Convert the first file to be imported, and / or the second file to be imported, and / or the third file to be imported into a JSONL file; Data processing is performed on the JSONL file to obtain processed imported data, and the processed imported data is stored according to a preset storage rule.

6. The large-scale data set processing method according to claim 5, It is characterized in that The processed imported data includes a plurality of pieces of data to be imported, and storing the processed imported data according to a preset storage rule includes: For each piece of data to be imported, generate a corresponding third globally unique ID; Based on the third globally unique ID and in combination with tag information of the data to be imported, construct third metadata corresponding to the data to be imported, wherein the tag information includes a storage path, token statistics, and a label of the data to be imported; splicing the third globally unique ID and the actual content of the data to be imported to perform data association to obtain third spliced ​​associated data; Writing the third metadata into a second metadata buffer area, and writing the third splicing-related data into a second I / O buffer area corresponding to the storage path of the data to be imported; When the cumulative amount of the third splicing-related data cached in the second I / O cache area reaches a preset quantity threshold, or when no new write data is detected within a preset time interval, the third metadata cached in the second metadata cache area is batch-written to the database, and the third splicing-related data cached in the second I / O cache area is written to the cloud storage disk.

7. The method for processing a large-scale data set according to any one of claims 1 to 6, It is characterized in that Also includes: Receive a data retrieval instruction, and extract from a database a plurality of target metadata corresponding to the data retrieval instruction and a target file path list corresponding to the plurality of target metadata; Construct and start a file preloader, and concurrently load multiple target splicing associated data corresponding to the target file path list through the file preloader, each of the target splicing associated data corresponds to a target global unique ID and a target ID-data mapping relationship; According to the target ID-data mapping relationship, the actual search content corresponding to the target global unique ID is extracted from the target splicing associated data to obtain the search result.

8. A large-scale data set processing device, It is characterized in that include: A data import mode determination module, used to determine a data import mode in response to a selection operation on an import mode in a data management interface; A sampling version generation module, used for determining sampling conditions according to the sampling combination selection operation for the data configuration options in the data management interface when the data import mode is to import the existing data set of the platform, and generating a sampling version according to the sampling conditions; A sampling data batch retrieval module is used to determine whether there is an existing sampling table for the sampling version, and if so, to batch retrieve the sampling data in the existing sampling table; The data processing and storage module is used to process the sampled data in combination with data cloning and ID replacement, and to store the processed sampled data in a preset storage method.

9. An electronic device, It is characterized in that The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the large-scale data set processing method described in any one of claims 1-7 according to the instructions in the program code.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium is used to store program code, and the program code is used to execute the large-scale data set processing method according to any one of claims 1-7.