Data management method and apparatus

Abstract

Embodiments of the present application provide a data management method and device, wherein the method comprises: obtaining training sample data uploaded by a user and a data type and a data format of the training sample data; in response to a first interaction operation, generating and running a target script, processing the training sample data through the target script, the target script being determined by a target key-value pair specified by the first interaction operation and a script template; in response to a second interaction operation, extracting target sample data from the training sample data according to a target proportion, converting the target sample data into model training data, the target data type and the target proportion of the target sample data being specified by the second interaction operation, the model training data representing a training set used for training a target large model. Through the present application, the technical problem of low sample data management efficiency of a large model in the related art is solved, and the management efficiency of sample data is improved.

Description

Technical Field

[0001] This application relates to the field of computers, and more specifically, to a data management method and apparatus. Background Technology

[0002] With the widespread application of large-scale models, enterprises across various industries have raised the need to train their own large-scale models, which are updated according to specific needs or over time. Since pre-training large-scale models requires a large amount of text data, enterprises typically obtain datasets from open-source websites for training each time the model is updated. However, due to differences in the content and composition of different datasets, the trained models will vary in capability, and the methods for processing the original data also differ. Therefore, there is an urgent need for a management system specifically designed for pre-training datasets to more efficiently support model pre-training tasks. A common model pre-training process first obtains open-source datasets or self-collected data. The original data formats are diverse, such as JSON and TXT. Scripts are needed to extract the required text from the original data and store it in new files. The processing method is manually recorded for reference during the next large-scale model update.

[0003] However, raw data typically comes from diverse sources, resulting in varying formats and structures, including types such as JSON and TXT. Each format requires different processing methods, often necessitating the writing of numerous custom scripts. However, manually rewriting these scripts each time new data is acquired is prone to omissions or mishandling, leading to the loss or incorrect removal of crucial data, ultimately impacting model performance. Furthermore, if the dataset is large, the manual processing workload is enormous, further increasing the risk of errors and potentially leading to a significant increase in data processing time.

[0004] Furthermore, the diversity of raw data is a key factor in ensuring that large-scale model pre-training tasks can cover a wide range of scenarios and complex situations. During each large-scale model pre-training session, relevant data information is manually recorded and stored for future model updates. However, as datasets are continuously updated and expanded over time, the workload of manual recording increases accordingly. Updated datasets may contain new categories, language variations, contextual information, or different data formats, placing higher demands on the accuracy of manual recording. With the increase in data volume and complexity, the probability of errors during manual recording also rises significantly. For example, recorders may make incorrect annotations or omissions due to fatigue, cognitive biases, or inaccurate understanding, thus affecting the quality and effectiveness of the pre-training task. In addition, standardization issues may exist during manual recording; differences in understanding and recording methods among different personnel can lead to data inconsistencies, which in turn affect the training performance of subsequent models.

[0005] Therefore, there is a technical problem in the related technologies where the sample data management efficiency of large models is relatively low. Summary of the Invention

[0006] This application provides a data management method and apparatus to at least solve the problem of low efficiency in managing sample data of large models in related technologies.

[0007] According to one embodiment of this application, a data management method is provided, comprising: acquiring training sample data uploaded by a user, as well as the data type and data format of the training sample data; in response to a first interactive operation, generating and running a target script to process the training sample data through the target script, wherein the target script is determined by a target key-value pair and a script template specified by the first interactive operation, and the script template is a script template corresponding to the data format; in response to a second interactive operation, extracting target sample data from the training sample data according to a target ratio, and converting the target sample data into model training data, wherein the target data type and the target ratio of the target sample data are specified by the second interactive operation, the data type includes the target data type, and the model training data represents the training set used by the target large model.

[0008] In an exemplary embodiment, after obtaining the training sample data uploaded by the user and the data type and data format of the training sample data, the method further includes: storing the training sample data to a target node; generating a storage path in response to storing the training sample data to the target node, wherein the storage path is used to indicate the storage address of the training sample data on the target node; and storing the data identifier, the data type, the data format and the storage path corresponding to the training sample data to a database.

[0009] In an exemplary embodiment, storing the training sample data to a target node includes: obtaining the amount of training sample data; searching for the target node on a target server based on the amount of data, wherein the target node represents a free node with sufficient storage space to accommodate the amount of data; and storing the training sample data to the target node in response to finding the target node.

[0010] In one exemplary embodiment, the step of generating and running a target script in response to a first interactive operation, and processing the training sample data through the target script, includes: determining a script template based on the data format in response to the first interactive operation; determining a target key-value pair based on key fields in the training sample data in response to the first interactive operation; replacing target fields in the script template with the target key-value pair to generate the target script; and running the target script to process the training sample data through the target script.

[0011] In one exemplary embodiment, different data formats correspond to different target scripts. Running the target script and processing the training sample data through the target script includes at least one of the following: running the target script to perform sampling operations on the training sample data when sampling operations need to be performed on the training sample data; running the target script to perform cleaning operations on the training sample data when cleaning operations need to be performed on the training sample data; and running the target script to perform concatenation operations on the training sample data when concatenation operations need to be performed on the training sample data.

[0012] In one exemplary embodiment, the step of extracting target sample data from the training sample data according to a target ratio in response to the second interactive operation, and converting the target sample data into model training data, includes: determining a dataset identifier and the target ratio in response to the second interactive operation; extracting target sample data from the training sample data according to the dataset identifier and the target ratio; and calling a format conversion module to perform format conversion on the target sample data to obtain the model training data.

[0013] In one exemplary embodiment, the method further includes: after acquiring training sample data uploaded by a user and the data type and data format of the training sample data, acquiring the data volume of the training sample data; searching for the target node on a target server based on the data volume, wherein the target node represents a free node with storage space sufficient for the data volume; in response to finding the target node, storing the training sample data to the target node; in response to storing the training sample data to the target node, generating a storage path, wherein the storage path indicates the storage address of the training sample data on the target node; storing the data identifier, the data type, the data format, and the storage path corresponding to the training sample data in a database; and in response to the first interactive operation, determining the script template based on the data format and according to the training sample data... The target key-value pair is determined based on the key fields in the template; the target key-value pair is used to replace the target fields in the script template to automatically generate the target script and create a data processing task; the target script is run to execute the data processing task, and the processing results of the data processing task are reported to the database in real time for task status marking; in response to the second interactive operation, the data identifier, data type, data format, and storage path corresponding to the training sample data whose task status is marked as completed are displayed in the database; the dataset identifier and the target ratio are determined, wherein the dataset identifier represents the identifier that meets the preset identifier conditions; target sample data is extracted from the training sample data according to the dataset identifier and the target ratio; the format conversion module is called to convert the format of the target sample data to obtain the model training data.

[0014] According to another embodiment of this application, a data management device is provided, comprising: an acquisition module, configured to acquire training sample data uploaded by a user, as well as the data type and data format of the training sample data; a first execution module, configured to generate and run a target script in response to a first interactive operation, and process the training sample data through the target script, wherein the target script is determined by a target key-value pair and a script template specified by the first interactive operation, and the script template is a script template corresponding to the data format; and a second execution module, configured to extract target sample data from the training sample data according to a target ratio in response to a second interactive operation, and convert the target sample data into model training data, wherein the target data type and the target ratio of the target sample data are specified by the second interactive operation, the data type includes the target data type, and the model training data represents the training set used by the training target large model.

[0015] According to yet another embodiment of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and the computer program is configured to perform the steps in any of the above method embodiments when it is run.

[0016] According to yet another embodiment of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0017] This application employs structured data management and automated data processing technologies. The system first acquires user-uploaded training sample data, along with related data type and format information. Responding to the user's first interactive operation, the system automatically generates and runs target scripts. These scripts are determined by user-specified target key-value pairs and preset script templates. The templates correspond to the data format, ensuring standardized and automated data processing. Furthermore, responding to the user's second interactive operation, the system extracts target sample data from the training sample data according to a user-specified target ratio and converts it into model training data, ensuring the flexibility and accuracy of data allocation. This process not only improves the efficiency and accuracy of data management but also maximizes the utilization of computing resources, reduces labor costs, and achieves standardized and accurate data recording. It achieves the goal of efficiently managing and processing large-scale datasets, thereby improving the technical effect of data processing efficiency and accuracy, and solving the technical problem of low efficiency in managing sample data for large models in related technologies. Attached Figure Description

[0018] Figure 1 This is a hardware structure block diagram of a server device for a data management method according to an embodiment of this application;

[0019] Figure 2 This is a flowchart of a data management method according to an embodiment of this application;

[0020] Figure 3 This is a schematic diagram of the hardware environment for a data management method according to an embodiment of this application;

[0021] Figure 4 This is a schematic diagram of the logical environment of the data management method according to an embodiment of this application;

[0022] Figure 5 This is a schematic diagram of the data upload process according to the data management method of the embodiment of this application;

[0023] Figure 6 This is a schematic diagram of the data processing flow of the data management method according to an embodiment of this application;

[0024] Figure 7 This is a schematic diagram of the data extraction process of the data management method according to an embodiment of this application;

[0025] Figure 8 This is a structural block diagram of a data management device according to an embodiment of this application. Detailed Implementation

[0026] The embodiments of this application will be described in detail below with reference to the accompanying drawings and examples.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0028] The methods and embodiments provided in this application can be executed on a server device or a similar computing device. Taking running on a server device as an example, Figure 1 This is a hardware structure block diagram of a server device for a data management method according to an embodiment of this application. Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The server device may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the server equipment described above. For example, the server equipment may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0029] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the data management method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to server devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0030] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the server device. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0031] This embodiment provides a data management method. Figure 2 This is a flowchart of a data management method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0032] Step S202: Obtain the training sample data uploaded by the user, as well as the data type and data format of the training sample data;

[0033] Step S204: In response to the first interactive operation, generate and run the target script to process the training sample data. The target script is determined by the target key-value pairs and script template specified by the first interactive operation. The script template is a script template corresponding to the data format.

[0034] Step S206: In response to the second interactive operation, target sample data is extracted from the training sample data according to the target ratio, and the target sample data is converted into model training data. The target data type and target ratio of the target sample data are specified by the second interactive operation. The data type includes the target data type. The model training data represents the training set used by the target large model for training.

[0035] The entities that perform the above steps can be servers, terminals, etc., but are not limited to these.

[0036] The execution order of steps S202 and S204 can be interchanged; that is, step S204 can be executed first, and then step S202 can be executed.

[0037] Optionally, in this embodiment, some of the terms involved in this application are first explained:

[0038] TB: Terabyte, trillion bytes.

[0039] JSON is an open standard file and data exchange format.

[0040] txt: A text format.

[0041] key / value: Data key-value pairs, where the key is the value used to uniquely identify the data, and the value is the data associated with the key.

[0042] bin: In this application, it specifically refers to the input file format required for pre-training of large models.

[0043] Optionally, in this embodiment, the "training sample data" mentioned above may include, but is not limited to, raw datasets uploaded by users to train large models. These datasets typically contain a large amount of text information for training and optimizing machine learning models. These datasets may come from different fields, such as news articles, social media posts, academic papers, etc. They can be structured data, such as records in a database, or unstructured data, such as text files. For example, training sample data may be a series of JSON format files containing user comments and rating information, or it may be TXT format files containing a large amount of book text content.

[0044] Optionally, in this embodiment, the aforementioned "data type and data format" may include, but are not limited to, the way data is organized and stored, which is crucial for determining how to correctly process and parse the data. Data types can be categorized according to Wikipedia's standards, including arts and culture, geography, humanities and social sciences, natural sciences, engineering and applied sciences, etc. Data format refers to the physical structure of the data, such as JSON, TXT, etc. For example, if the data type is "engineering and applied sciences," the data format might be TXT, representing the text content of a series of technical documents; if it is "humanities and social sciences," it might be JSON, containing survey results and statistical data from social science research.

[0045] Optionally, in this embodiment, the "target script" may include, but is not limited to, a series of predefined code templates. These templates are used to process data in a specific format and can be customized according to user-specified key-value pairs. The purpose of the target script is to extract, clean, and transform data from the raw data to meet the needs of subsequent model training. For example, if the data format is JSON, the target script may contain code to parse the JSON object and extract specific fields; if it is TXT format, the script may contain logic for text cleaning and word segmentation. The generation and execution of the target script are based on the key-value pairs specified by the user through a first interactive operation, which tell the system what specific information needs to be extracted from the data.

[0046] Optionally, in this embodiment, the "model training data" mentioned above may include, but is not limited to, datasets used to train the target large model after processing and transformation. These datasets are carefully selected and proportioned to ensure that the model can learn diverse and balanced information. The model training data not only includes the target data type but also reflects the diversity and complexity of the data. For example, if the target large model is a language model, the model training data may include text data in different languages ​​to ensure that the model can understand and generate content in multiple languages; if it is an image recognition model, the model training data may include images of different categories and styles to improve the model's generalization ability. Through the second interactive operation, the user can specify the required data type and proportion, and the system will extract and transform a dataset suitable for model training from the training sample data accordingly.

[0047] It should be noted that the training sample data uploaded by users can come from a variety of sources, including but not limited to open-source datasets, internal enterprise data accumulation, web page content crawled by web crawlers, and social media data collected by users themselves. The types and formats of this data are also diverse, such as text data, image data, audio data, or structured data such as CSV files. This application does not impose any limitations on this.

[0048] Furthermore, the target script generated and executed in response to the first interactive operation can employ different processing logics depending on the data format. For example, for text data, the script may include preprocessing steps such as word segmentation and stop word removal; for image data, it may include operations such as image resizing and normalization; and for audio data, it may involve noise reduction and feature extraction. The specification of target key-value pairs can also vary depending on the data content; for example, text data may focus on keywords and themes, while image data may focus on objects and scenes. This application does not impose any limitations on this.

[0049] On the other hand, the process of extracting target sample data from the training sample data in response to the second interactive operation can be adjusted according to different model training needs. For example, for training a multilingual translation model, it may be necessary to extract text data according to the proportion of different languages; for training an image recognition model, it may be necessary to extract image data according to the proportion of different categories. The process of converting target sample data into model training data can also be customized according to the specific needs of the model. For example, for deep learning models, it may be necessary to convert the data into a specific tensor format; for traditional machine learning models, it may be necessary to convert the data into feature vectors. This application does not limit this.

[0050] For example, suppose you are building a data preprocessing system for training large natural language processing (NLP) models. This system is capable of processing large-scale text datasets and converting them into a format suitable for model training.

[0051] S1: The user uploads training sample data. The user uploaded a 1TB text dataset through the system interface. This dataset contains files in various formats, such as JSON and TXT. The user specified the data type as "Humanities and Social Sciences" and the data format as JSON and TXT.

[0052] S2: Responding to the first interactive operation, the system generates the target script. The user has performed the first interactive operation through the system interface, specifying target key-value pairs. For example, in JSON data, specifying "title" and "content" as key-value pairs indicates that the title and content need to be extracted from the JSON file. Based on the user-specified key-value pairs and data format, the system automatically generates two target scripts: one for processing JSON format data and the other for processing TXT format data. These script templates are predefined and capable of handling common data cleaning and format conversion tasks.

[0053] S3: Run the target scripts to process the training sample data. The system begins executing these two target scripts. For JSON format data, the script parses each JSON object, extracts the "title" and "content" fields, and saves this information to a new TXT file. For TXT format data, the script performs text cleaning operations, such as removing special characters and extra spaces. This processed data is stored in a specified path on the server.

[0054] S4: Respond to the second interactive operation to extract target sample data. The user performed a second interactive operation, specifying that target sample data should be extracted from the processed data in a ratio of 60% news articles and 40% social media posts. The system extracted 60% of the data from the news article dataset and 40% of the data from the social media post dataset according to the user-specified ratio.

[0055] S5: Converting target sample data into model training data. The system merges the extracted target sample data into a single result file and converts this data into the format required for NLP model training, based on the model's training needs. For example, if the model requires input data as a fixed-length sequence, the system will perform word segmentation and padding operations on the text to ensure that the length of each sample is consistent.

[0056] Through the above process, the system achieves efficient management and processing of large-scale text datasets. First, automated script generation and execution significantly reduce the workload of manual script writing and debugging, improving the speed and accuracy of data processing. Second, automated data extraction and format conversion enable the system to quickly respond to user needs and generate datasets suitable for model training. This not only improves data processing efficiency but also ensures dataset quality and consistency. Finally, through structured data management and optimization of distributed computing resources, the system can handle large-scale datasets while maximizing resource utilization and reducing costs. These technological effects collectively enhance the efficiency and effectiveness of large-scale model pre-training, laying a solid foundation for building high-performance NLP models.

[0057] This application's embodiments employ structured data management and automated data processing technologies. The system first acquires user-uploaded training sample data, along with related data type and format information. Responding to the user's first interactive operation, the system automatically generates and runs target scripts. These scripts are determined by user-specified target key-value pairs and preset script templates. The templates correspond to the data format, ensuring standardized and automated data processing. Furthermore, responding to the user's second interactive operation, the system extracts target sample data from the training sample data according to a user-specified target ratio and converts it into model training data, ensuring the flexibility and accuracy of data allocation. This process not only improves the efficiency and accuracy of data management but also maximizes the utilization of computing resources, reduces labor costs, and achieves standardized and accurate data recording. It achieves the goal of efficiently managing and processing large-scale datasets, thereby improving the technical effect of data processing efficiency and accuracy, and solving the technical problem of low management efficiency for large model sample data in related technologies.

[0058] In an exemplary embodiment, after obtaining the training sample data uploaded by the user and the data type and data format of the training sample data, the method further includes: storing the training sample data to a target node; generating a storage path in response to storing the training sample data to the target node, wherein the storage path is used to indicate the storage address of the training sample data on the target node; and storing the data identifier, data type, data format and storage path corresponding to the training sample data to a database.

[0059] Optionally, in this embodiment, the "target node" may include, but is not limited to, one or more servers, cloud storage service nodes, nodes in a distributed file system, or any computing resource capable of storing data. These target nodes are responsible for receiving and storing the uploaded training sample data. For example, the target node may be a server cluster within Inspur Electronic Information Industry Co., Ltd., an S3 bucket in Amazon AWS cloud service, or a storage node in Google Cloud Platform.

[0060] Optionally, in this embodiment, the aforementioned data identifier, data type, data format, and storage path can constitute a sample label. This sample label may include, but is not limited to, a set of metadata used to describe and identify the content and attributes of the training sample data. The data identifier in the sample label is part of the sample label and is used to uniquely identify each dataset or data file. For example, the sample label may include information such as the dataset's name, source, collection date, and version number, while the data identifier may be a unique serial number or hash value used for quickly retrieving and referencing a specific dataset in a database.

[0061] Optionally, in this embodiment, the "storage path" may include, but is not limited to, a file system path, a URL, a record location in a database, or any identifier that can uniquely identify the data storage location. The storage path indicates the storage address of the training sample data on the target node, enabling the system to accurately access and retrieve the data. For example, the storage path can be an absolute path on a server, such as " / server / data / dataset1", a relative path, such as "data / dataset1", or a URL in a cloud storage service, such as "s3: / / bucket-name / dataset1".

[0062] Optionally, in this embodiment, the "database" may include, but is not limited to, a relational database, a non-relational database, a distributed database, or any system capable of storing and managing data. The database stores information such as data identifiers, data types, data formats, and storage paths corresponding to training sample data. This information helps the system manage and retrieve data, as well as perform subsequent data preprocessing and model training tasks. For example, the database may be MySQL, MongoDB, or a Hadoop Distributed File System, which can store structured or unstructured data and provide query and data management functions.

[0063] It should be noted that the process of storing training sample data to the target node can involve various storage media and environments, and this application does not limit this. These storage media may include, but are not limited to, local hard drives, network attached storage (NAS), cloud storage services (such as Amazon S3, Google Cloud Storage), and distributed file systems (such as Hadoop HDFS). Each storage medium has its specific access speed, capacity, cost, and reliability characteristics, and is suitable for different application scenarios.

[0064] Furthermore, the storage path generated when training sample data is stored on the target node can have various formats and structures, which are not limited in this application. The storage path can be a simple file system path, such as " / data / dataset1", or a complex path containing multiple subdirectories, such as " / data / 2024 / Q1 / dataset1". In addition, the storage path can also be a bucket and object identifier in a cloud storage service, such as "s3: / / my-bucket / dataset1", or a table and record ID in a database, such as "table:dataset1, record:12345".

[0065] On the other hand, the operation of storing the data identifiers, data types, data formats, and storage paths corresponding to the training sample data into a database can employ various database technologies and models, and this application does not limit this. These databases can be relational databases such as MySQL and PostgreSQL, which organize data through tables and relational models; they can also be non-relational databases such as MongoDB and Cassandra, which store data through documents or key-value pairs; or they can be time-series databases such as InfluxDB, which are specifically designed for processing time-series data. Each database technology has its specific query language, optimization strategies, and applicable scenarios, and the most suitable database solution can be selected based on the characteristics of the data and access patterns.

[0066] In one exemplary embodiment, storing training sample data to a target node includes: obtaining the amount of training sample data; searching for a target node on a target server based on the amount of data, wherein the target node represents a free node with sufficient storage space to accommodate the amount of data; and storing the training sample data to the target node in response to finding the target node.

[0067] Optionally, in this embodiment, the aforementioned "data volume" may include, but is not limited to, the total size of the training sample data. This size can be measured in units such as bytes, kilobytes (KB), megabytes (MB), gigabytes (GB), or terabytes (TB). The size of the data directly affects the data processing and storage requirements, therefore it needs to be accurately determined to identify suitable storage nodes. For example, if a dataset contains a large number of high-definition images or video files, its data volume may reach several GB or even TB levels; while a text dataset may only be a few MB to a few hundred MB in size. Understanding the data volume helps the system assess the storage resources that need to be allocated and the expected processing time.

[0068] Optionally, in this embodiment, the "target server" may include, but is not limited to, a single physical server, a virtual server, a server cluster, or a cloud service platform. The target server is the computing resource used by the system to store and process training sample data. For example, the target server can be a physical server configured with a high-performance CPU and a large amount of memory, specifically designed for processing large-scale datasets; or it can be a cloud service platform, such as AWS or Azure, providing elastic computing and storage resources that can dynamically adjust resource allocation based on the amount of data.

[0069] Optionally, in this embodiment, the "target node" may include, but is not limited to, a physical storage device, virtual storage space, or cloud storage node with a specific storage capacity. The target node is the actual location where the training sample data is stored, and it must have sufficient storage space to meet the data volume requirements. For example, if the data volume is 500GB, the system may find a storage node with 1TB of free space to store this data; if the data volume is larger, such as reaching several TB, the system may need to find multiple nodes or a node with sufficient capacity to store this data. The selection of the target node will take into account factors such as storage space, read / write speed, data security, and cost-effectiveness.

[0070] It should be noted that the step of obtaining the amount of training sample data described above can involve various data measurement methods, and this application does not limit this. The amount of data can be automatically obtained by counting the file size, estimated by the number of data record entries, or calculated by the logical storage units occupied by the data. For example, for text data, the amount of data may be calculated based on the number of characters or words; for image data, it may be estimated based on the total number of pixels or file size; for time series data, it may be measured based on the number of time points. These measurement methods can be flexibly selected according to the characteristics of the data and the use case.

[0071] Furthermore, the process of searching for target nodes on the target server based on data volume can involve various storage resource allocation strategies, which this application does not limit. When searching for target nodes, the system may consider multiple dimensions such as node storage capacity, read / write speed, network latency, and energy consumption. For example, the system may prioritize nodes with larger storage capacity and faster read / write speeds to improve data processing efficiency; at the same time, it may also consider the node's energy consumption and cooling capacity to achieve the goal of green energy saving. In addition, the system may also select storage nodes with different performance levels based on the data access frequency and importance to optimize overall storage costs and performance.

[0072] On the other hand, the operation of storing training sample data to the target node in response to its discovery can involve various data transmission and storage technologies, which this application does not limit. Data transmission can be synchronous or asynchronous, and can be transmitted directly over a network or copied via physical media such as hard drives. Storage technologies can include block storage, file storage, or object storage. For example, for data requiring frequent random access, block storage or file storage may be chosen; while for large-scale unstructured data, object storage may be chosen. In addition, data deduplication, compression, and other technologies can be used to optimize storage space utilization during data storage. The selection of these technologies can be determined based on the characteristics of the data and business needs to achieve efficient data storage and management.

[0073] In one exemplary embodiment, in response to a first interactive operation, generating and running a target script to process training sample data includes: in response to the first interactive operation, determining a script template based on a data format; in response to the first interactive operation, determining target key-value pairs based on key fields in the training sample data; replacing target fields in the script template with target key-value pairs to generate the target script; and running the target script to process the training sample data.

[0074] In this embodiment, the aforementioned "script template" may include, but is not limited to, a set of predefined code frameworks that contain the basic logic and structure required to process specific data formats. Script templates are designed for different types of data formats and may include code for common operations such as data parsing, data cleaning, and data transformation. For example, for JSON format data, a script template may contain code for parsing JSON objects and arrays; for CSV format data, a script template may contain code for reading and writing CSV files. Script templates can vary depending on the data format to adapt to different data processing needs.

[0075] Optionally, in this embodiment, the "target key-value pair" may include, but is not limited to, key information pairs extracted from training sample data. These key-value pairs are used to specify data fields that require special attention in the data processing script. The target key-value pairs are specified by the user according to data processing needs. They can be column names and values ​​in a database, keys and values ​​in a JSON object, or any data identifiers and corresponding data that need to be extracted from the raw data. For example, when processing data from an e-commerce platform, target key-value pairs may include key fields such as "product ID" and "price," and "user rating"; when processing medical and health data, target key-value pairs may include key information such as "patient ID" and "diagnosis result," and "drug name."

[0076] Optionally, in this embodiment, the "target field" may include, but is not limited to, placeholder variables or identifiers in the script template, which will be replaced by target key-value pairs during script generation. The target field is predefined in the script template and indicates the specific data content that the script needs to process during execution. For example, in a script template processing text data, the target field might be "text content" or "author name"; in a script template processing image data, the target field might be "image path" or "image tag". Replacing the target field is a crucial step in generating the target script, ensuring that the script can perform the expected data processing operations on a specific dataset.

[0077] Optionally, in this embodiment, "running the target script" may include, but is not limited to, the process of executing a generated script to process training sample data. This process involves applying the logic defined in the script to the actual data to perform operations such as data cleaning, transformation, and extraction. For example, if the target script is to extract specific keywords from text data, running the target script will involve scanning the text, identifying the keywords, and saving them into a new data structure; if the target script is to convert image data into the format required for model training, running the target script will involve reading image files, applying image processing algorithms, and saving the processed image to a specified storage location. Running the target script is a core step in the data processing flow, directly affecting the results of data preprocessing and the effectiveness of subsequent model training.

[0078] It should be noted that the process of determining the script template based on the data format in response to the first interactive operation can select the appropriate script template according to the characteristics of different data formats and processing requirements, and this application does not limit this. Data formats may include structured data such as CSV, JSON, and XML; semi-structured data such as log files; or unstructured data such as text and images. For example, for CSV format data, the script template may contain specific reading and parsing logic; for image data, the script template may contain image recognition and processing algorithms. Furthermore, the selection of the script template may also be based on the data source, purpose, and expected processing results; for example, social media data may require a specific text analysis template.

[0079] Furthermore, the process of determining the target key-value pair based on key fields in the training sample data in response to the first interactive operation can be determined according to the data content and business needs, and this application does not limit it in this regard. Key fields may include, but are not limited to, the primary key, foreign key, index field, or key attributes in the business logic. For example, when processing customer data, key fields may include "customer ID", "purchase history", and "credit score"; when processing sensor data, key fields may include "timestamp", "sensor ID", and "measurement value". The determination of the target key-value pair may also take into account the privacy and security requirements of the data to ensure that sensitive information is properly handled.

[0080] Furthermore, the process of generating the target script by replacing the target field in the target key-value pair replacement script template can be customized according to different data processing logics and business rules, and this application does not impose any limitations on this. The replacement of target fields may involve operations such as data extraction, transformation, and aggregation. For example, when processing financial data, it may be necessary to replace the "amount" and "date" fields to generate a financial report; when processing medical data, it may be necessary to replace the "patient name" and "diagnosis result" fields to generate a medical record summary. The replacement of target fields can also be adjusted according to the timeliness and frequency of change of the data to adapt to the dynamically changing data environment.

[0081] On the other hand, the process of running the target script and processing the training sample data can be executed according to different data processing goals and performance requirements, and this application does not limit this. The execution of the target script may involve batch processing or real-time processing, and may require consideration of data processing parallelization and optimization. For example, when processing large-scale text data, a distributed computing framework may be needed to accelerate the processing; when processing real-time transaction data, it may be necessary to ensure low latency and high throughput. The execution of the target script can also be adjusted according to the data storage location and access mode, such as execution in the cloud, local, or edge computing environments, to optimize performance and cost.

[0082] In one exemplary embodiment, different data formats correspond to different target scripts. Running the target script to process training sample data includes at least one of the following: when it is necessary to perform a sampling operation on the training sample data, running the target script to perform a sampling operation on the training sample data; when it is necessary to perform a cleaning operation on the training sample data, running the target script to perform a cleaning operation on the training sample data; when it is necessary to perform a splicing operation on the training sample data, running the target script to perform a splicing operation on the training sample data.

[0083] Optionally, in the embodiments of this application, the aforementioned "sampling operation" may include, but is not limited to, the process of selecting a portion of data from the training sample dataset as samples. This process may be based on different sampling techniques such as random sampling, stratified sampling, and systematic sampling. The purpose of the sampling operation is to reduce the size of the dataset while maintaining its representativeness as much as possible, which is particularly important for training large-scale machine learning models. For example, when processing large-scale image datasets, the sampling operation may include randomly selecting a certain number of images as training samples; when processing financial transaction data, the sampling operation may include selecting transaction records within a specific time period as samples.

[0084] Optionally, in the embodiments of this application, the aforementioned "cleaning operation" may include, but is not limited to, preprocessing the training sample data to eliminate noise, fill in missing values, remove outliers, etc., to improve data quality. The cleaning operation is a key step in data preprocessing, which helps improve the effectiveness of subsequent model training. For example, when processing text data, the cleaning operation may include removing useless punctuation marks, filtering stop words, and stemming; when processing sensor data, the cleaning operation may include filtering, outlier detection, and filling in missing sensor readings.

[0085] Optionally, in this embodiment, the "stitching operation" described above may include, but is not limited to, the process of merging multiple training sample datasets into a large dataset. This process may be based on simple data merging, complex feature alignment, or data fusion based on specific business logic. The purpose of the stitching operation is to integrate data from different sources to enrich the features and information of the dataset. For example, when processing data from multiple sensors, the stitching operation may include aligning and merging time-series data from different sensors; when processing text data from different sources, the stitching operation may include merging the content of different documents into a large text file for subsequent text mining and analysis.

[0086] It should be noted that when sampling operations are required on the training sample data, the process of running the target script can be adjusted according to different sampling strategies and data characteristics, and this application does not limit this. Sampling operations may involve random sampling, stratified sampling, or sampling based on specific conditions. For example, when dealing with large-scale datasets, random sampling may be used to reduce the amount of data while maintaining data diversity; when data balance needs to be maintained, stratified sampling may be used to ensure the proportion of each category or group in the sample. Furthermore, sampling operations can also be performed based on the time-series characteristics, spatial distribution characteristics, or business logic characteristics of the data. For example, in time-series analysis, sampling may need to be performed at time intervals.

[0087] Furthermore, when it is necessary to perform cleaning operations on training sample data, the process of running the target script can be customized according to different data quality issues and business needs, and this application does not limit this. Cleaning operations may include removing duplicate records, correcting erroneous data, filling in missing values, or smoothing noise. For example, when processing text data, cleaning may include removing irrelevant characters, standardizing text format, or eliminating sensitive information; when processing numerical data, it may be necessary to identify and handle outliers, fill in missing values, or perform data standardization. Cleaning operations can also be performed according to the source, format, and intended use of the data; for example, when processing medical data, special attention may be needed to protect patient privacy.

[0088] On the other hand, in cases where concatenation operations are required on training sample data, the process of running the target script can be designed according to different data integration needs and data structures, and this application does not impose any limitations on this. Concatenation operations may involve merging datasets from different sources, aligning data of different formats, or fusing data of different granularities. For example, when integrating data from different databases, the concatenation operation may need to address the alignment issues of different data models and data structures; when processing multimodal data, it may be necessary to fuse text, image, and audio data together. Concatenation operations can also be optimized based on the temporal sensitivity, spatial relevance, or business logic of the data. For instance, when integrating data from different geographical locations, the proximity and relevance of geographical locations may need to be considered.

[0089] In one exemplary embodiment, in response to a second interactive operation, target sample data is extracted from the training sample data according to a target ratio, and the target sample data is converted into model training data, including: in response to the second interactive operation, determining a dataset identifier and a target ratio; extracting target sample data from the training sample data according to the dataset identifier and the target ratio; and calling a format conversion module to perform format conversion on the target sample data to obtain model training data.

[0090] Optionally, in this embodiment, the aforementioned "dataset identifier" may include, but is not limited to, a set of information used to uniquely identify the training sample dataset. This set of information may be the dataset's name, number, label, or other features that can distinguish different datasets. The purpose of the dataset identifier is to quickly locate and reference a specific dataset in a complex data management environment. For example, in a data warehouse containing multiple sources and types, the dataset identifier may be a naming convention consisting of a date and source channel, such as "2024_06_medical_imaging," indicating medical imaging data collected in June 2024; or it may be an internally generated unique identifier, such as a UUID, used for indexing and retrieving the dataset in the database.

[0091] Optionally, in the embodiments of this application, the aforementioned "target ratio" may include, but is not limited to, the sample extraction ratio specified according to specific needs or algorithms. This ratio can be a fixed value or a dynamically calculated result. The target ratio is used to determine the number of samples extracted from the original dataset to meet the needs of model training or other data processing tasks. For example, when performing model training, the target ratio may be the proportion of the training set determined based on the size of the validation set and the test set, such as 70% of the data used for training, 15% for validation, and 15% for testing; when performing imbalanced data processing, the target ratio may be the oversampling or undersampling ratio of samples of a specific class, such as increasing the proportion of minority class samples to be equal to that of the majority class.

[0092] Optionally, in this embodiment, the "format conversion module" may include, but is not limited to, a set of functions used to convert target sample data into a format suitable for model training. The format conversion module can process different types of data, such as text, images, audio, or video, and convert them into an input format that the model can understand. For example, for text data, the format conversion module may include steps such as word segmentation, part-of-speech tagging, and word embedding construction; for image data, it may include operations such as size adjustment, normalization, and color space conversion; for audio data, it may include processing such as sampling rate conversion and spectrogram extraction. The format conversion module is designed to improve the efficiency and effectiveness of model training, ensuring that data is input into the model in the optimal way.

[0093] It should be noted that the process of determining the dataset identifier and target ratio in response to the second interactive operation can be adjusted according to various user needs and business scenarios, and this application does not limit it. The dataset identifier can be defined based on multiple dimensions such as data source, content theme, or timestamp. For example, it can be a news dataset from a specific news website, a collection of posts on social media within a specific time period, or a sensor dataset collected in a specific region. The target ratio can be determined according to the size of the dataset, the needs of model training, or the need for data balancing. For example, it can be sampling according to class balance, sampling according to the proportion of data distribution, or sampling according to the traditional proportions of 70%, 15%, and 15% for the model training set, validation set, and test set.

[0094] Furthermore, the aforementioned operation of extracting target sample data from the training sample data according to the dataset identifier and target proportion can be customized according to different data characteristics and processing objectives, and this application does not impose any limitations on this. The extraction operation may involve random selection, stratified selection, or selection based on specific rules. For example, it could be stratified sampling based on the statistical characteristics of the data, time-series sampling based on the time-series characteristics of the data, or focused sampling based on the importance of the data. In addition, the extraction operation can also be filtered based on the quality and completeness of the data; for example, excluding data records with many missing values, selecting data samples with rich information, or selecting based on the novelty and diversity of the data.

[0095] On the other hand, the process of calling the format conversion module to convert the target sample data to obtain model training data can be adjusted according to the specific requirements of model training and the original format of the data, and this application does not limit this. Format conversion may involve data standardization, normalization, or encoding conversion. For example, text data can be converted into word vectors, image data into pixel value arrays, and audio data into spectrograms. In addition, format conversion can be customized according to the input requirements of the model. For example, for deep learning models, it may be necessary to convert the data into a specific tensor format; for traditional machine learning models, it may be necessary to convert the data into feature vectors. The format conversion module is designed to ensure that the data can be input in the form most suitable for model training, thereby improving the efficiency and effectiveness of model training.

[0096] In one exemplary embodiment, the method further includes: after acquiring the training sample data uploaded by the user and the data type and data format of the training sample data, acquiring the data volume of the training sample data; searching for a target node on a target server based on the data volume, wherein the target node represents a free node whose storage space meets the data volume; in response to finding the target node, storing the training sample data to the target node; in response to storing the training sample data to the target node, generating a storage path, wherein the storage path is used to indicate the storage address of the training sample data on the target node; storing the data identifier, data type, data format, and storage path corresponding to the training sample data to a database; and in response to a first interactive operation, determining a script template based on the data format and, according to the training sample data... The key fields in this data determine the target key-value pairs; the target key-value pairs are used to replace the target fields in the script template to automatically generate the target script and create a data processing task; the target script is run to execute the data processing task, and the processing results of the data processing task are reported to the database in real time for task status marking; in response to the second interactive operation, the data identifier, data type, data format, and storage path corresponding to the training sample data marked as completed in the database are displayed; the dataset identifier and target ratio are determined, where the dataset identifier indicates the identifier that meets the preset identifier conditions; target sample data is extracted from the training sample data according to the dataset identifier and target ratio; the format conversion module is called to convert the format of the target sample data to obtain the model training data.

[0097] For example, suppose we are dealing with a large-scale e-commerce review dataset that will be used to train a sentiment analysis model to identify the sentiment in customer reviews.

[0098] S1: A user uploaded a dataset containing 10 million comments, stored in JSON format. Each comment includes a user ID, comment text, and rating.

[0099] S2: The system has obtained the data volume of the dataset, which is 500GB in size.

[0100] S3: Based on the data volume, the system searches for target nodes with sufficient free storage space on the target server. The server cluster has 5 nodes, of which 3 nodes have more than 500GB of free space.

[0101] S4: The system selects one of the idle nodes and stores the training sample data in that node. For example, the storage path is ` / server / datasets / ecommerce comments / `.

[0102] S5: The system generates a storage path and stores it in the database along with the data identifier, data type (text), and data format (JSON), recording detailed information about the dataset.

[0103] S6: In response to the first interactive operation, the system determines the script template based on the JSON data format and determines the target key-value pair based on the key fields (user ID, comment text, and rating).

[0104] S7: The system replaces the target field in the script template with the target key-value pair, automatically generates the target script, and creates data processing tasks, such as data cleaning and feature extraction.

[0105] S8: The system runs the target script to perform data processing tasks and reports the processing results to the database in real time to mark the task status, for example, mark it as "processing completed".

[0106] S9: In response to the second interactive operation, the system displays the data identifier, data type, data format, and storage path corresponding to the training sample data whose task status is marked as completed in the database.

[0107] S1O: The system determines the dataset identifier and target proportion. For example, it selects reviews with a rating of 4 stars or higher (including 5 stars) as training data, with a target proportion of 80%.

[0108] S11: The system extracts target sample data from the training sample data according to the dataset identifier and target ratio, that is, extracts 80% of the high-rated reviews.

[0109] S12: The system calls the format conversion module to convert the target sample data into a format, for example, converting JSON format comments into CSV format required for model training, thus obtaining model training data.

[0110] Through the embodiments of this application, by utilizing automated storage space management and data storage, the system can quickly locate and store large-scale datasets, ensuring the efficiency of data storage—the first step in data processing. Secondly, through the generation and execution of automated scripts, the system can perform customized data processing for specific formats and key fields, improving the flexibility and accuracy of data processing. Thirdly, through real-time task status updates and data extraction, the system ensures that only high-quality data is used for model training, thereby improving the effectiveness and efficiency of model training. Finally, through the format conversion module, the system can convert data into a format suitable for model training, further optimizing the input data for model training and providing a solid data foundation for building a high-performance sentiment analysis model.

[0111] The following specific examples will further explain this application:

[0112] This application designs a large model pre-training data processing system to automatically record the data matching process, ensuring the integrity and consistency of the data. It also automatically identifies and generates corresponding processing scripts based on different data formats (such as JSON, TXT, etc.) to reduce manual intervention and improve processing efficiency.

[0113] The core principle of this application lies in structured data management, utilizing a database to store raw data information. Furthermore, it enables distributed computing resource optimization by writing programs to mount information from all current servers, facilitating task distribution. On the other hand, this application achieves automated data processing, reducing code development workload through pre-built processing scripts and user-manual key / value replacement. Through multi-layered table structure management and distributed computing design, combined with automated script generation and real-time task keep-alive mechanisms, it ensures high efficiency in data processing and system maintainability.

[0114] Figure 3 This is a schematic diagram of the hardware environment for a data management method according to an embodiment of this application, such as... Figure 3 As shown, the various modules of this application are described below:

[0115] The data upload / download module is responsible for uploading raw data and associating it with user-specified data tags, and synchronously recording the upload path and tag information in the database. This design facilitates quick identification and management of uploaded data, improving data access efficiency.

[0116] Database Module: Data Storage Table Design. The newly added data storage table includes fields such as dataset name, data source, data organization format, data type, and data storage path. The data organization format field records the format of the original data (e.g., JSON, TXT, etc.), and the data type field is categorized according to encyclopedic standards, including [Arts & Culture, Geography, Humanities & Social Sciences, Natural Sciences, Engineering & Applied Sciences]. This design helps to quickly locate and organize data. Data Task Table: Used to manage the lifecycle of tasks. Fields include task information, task creation time, task end time, task update time, and task status. The task information field records detailed task allocation nodes and their specific requirements; the task status field indicates the current stage of the task, with status values ​​including [Initialized, Running, Completed, Abnormal]. Data Cache Table: Records decompressed high-frequency data. Fields are set to dataset name, data type, original data compressed package path, and decompressed file path. This table improves data access speed and avoids repeated decompression.

[0117] Data Processing Module: Users can manually specify key / value pairs on the interface. The system automatically replaces placeholders in the preset text processing code templates based on these specified key / value pairs, generating processing scripts for cleaning, sampling, and concatenation of the current dataset. This feature simplifies the coding work for data processing. The preset text processing code templates are common multi-process data processing code without special algorithms. The number of code templates corresponds to the number of original data formats; for example, there is a code template for processing JSON data and a code template for processing TXT data.

[0118] Data to BIN module: The management system collects status information from all nodes (task status information indicates completion and readiness for conversion) to determine whether to perform the data conversion to BIN format. Before conversion, the user is required to input the proportions of different data types. Then, all data meeting the criteria are integrated and the process data format conversion is performed. This module is only executed when all tasks are completed and in normal status, ensuring the integrity and accuracy of data processing.

[0119] Figure 4 This is a schematic diagram of the logical environment of the data management method according to an embodiment of this application, such as... Figure 4 As shown, the following logic implementation methods are included, but are not limited to:

[0120] S1: The data upload / download module is responsible for processing the raw datasets uploaded by users. For example, if a user uploads a compressed file containing 1 TB of text data, this module will receive the data and store it on a specified path on the server. This step involves receiving and initially storing the data, laying the foundation for subsequent data processing.

[0121] S2: The data decompression module decompresses the uploaded data archive. For example, if the uploaded archive is in ZIP format, this module will perform the decompression operation, restoring the data to its original format, such as a TXT or JSON file, for easier subsequent cleaning and processing.

[0122] S3: The data cleaning module cleans the decompressed data. For example, for text data, this module may remove irrelevant characters, correct spelling errors, and standardize text formatting to improve data quality and ensure data consistency and accuracy.

[0123] S4: The data processing module automatically replaces placeholders in the preset text processing code template based on the key / value pairs specified by the user on the interface, generating processing scripts for cleaning, sampling, and concatenation of the current dataset. For example, if the user specifies that the "tit1e" and "content" fields need to be extracted from JSON data, the system will generate corresponding scripts to extract this information from the JSON file.

[0124] S5: The data sampling module extracts samples from the cleaned data according to the sampling strategy specified by the user. For example, if the user may need to randomly extract 10% of the samples from the dataset for model training, this module will perform the sampling operation according to this requirement.

[0125] S6: The data concatenation module concatenates the sampled data according to certain rules to form a dataset suitable for model training. For example, if model training requires continuous text sequences, this module will concatenate multiple text fragments into a long sequence.

[0126] S7: The bin-to-bin module converts the processed data into the bin format required for model training after all data processing tasks are completed. For example, if model training requires a fixed-size input, this module will convert text data into a fixed-length binary format for easy reading and processing by the model.

[0127] S8: The database module records information such as data storage path, data type, data format, and task status throughout the process. For example, the database will record the name, source, storage location, creation time, end time, and status of each dataset, facilitating data management and tracking.

[0128] S9: Server-layer nodes are responsible for executing the tasks of the various modules in the application layer mentioned above. For example, node 1 may be responsible for the data upload and download module, node 2 may be responsible for the data decompression module, and so on. Each node executes the corresponding task according to its computing resources and load, realizing distributed processing and resource optimization.

[0129] Figure 5 This is a schematic diagram of the data upload process according to the data management method of the embodiment of this application, such as... Figure 5As shown, during data upload, the user specifies the name of the current dataset and the type of the dataset. The management system automatically searches for available storage space based on the size of the dataset. If there is insufficient storage space, the task is terminated directly. If available storage space is found, task information is generated and recorded in the task table of the database. Based on the current status of the child node, the task is distributed to any available child node to complete the data upload task. Finally, the data storage path and the user-specified dataset name, type, and other information are recorded in the data storage table of the database.

[0130] Figure 6 This is a schematic diagram of the data processing flow of the data management method according to an embodiment of this application, such as... Figure 6 As shown, the uploaded data is processed. Taking JSON data as an example, the user specifies the required key / value pairs, and then the user's input string is replaced in the JSON data processing template script. The path of the uploaded data is passed to the script, and the required data text can be obtained after the script is executed.

[0131] Figure 7 This is a schematic diagram of the data extraction process according to the data management method of this application embodiment, such as... Figure 7 As shown, the system performs matching and binning operations on the processed data. The user selects the dataset to use and specifies the proportion of each type of data. Based on the data and matching information selected by the user, the system extracts the corresponding amount of data from each dataset according to the proportion, merges them into a result file, and then calls the binning module to process the data into the input file format required for large model pre-training.

[0132] The embodiments of this application can improve the efficiency and accuracy of data management. Through structured data storage (such as data storage tables, task tables, and cache tables), the system can efficiently manage large amounts of data and their related metadata. This approach ensures clear and standardized data management, facilitates rapid querying and retrieval, and avoids data redundancy and chaos.

[0133] Furthermore, resource utilization can be maximized. Through distributed data processing and rational optimization of computing resources (such as reusing GPU servers for efficient computation), the system avoids wasting CPU resources and achieves efficient utilization of computing resources. Tasks are counted and allocated according to the original compressed packages, avoiding idle computing nodes and improving processing efficiency.

[0134] On the other hand, it can reduce labor costs and improve work efficiency. Due to the high degree of automation in the data processing process (such as automated script generation, task status monitoring, and automated scheduling), the system can greatly reduce human intervention and decrease reliance on human resources. This not only improves work efficiency but also reduces labor costs.

[0135] On the other hand, it can achieve standardization and accuracy of data recording, transforming the recording of information such as data matching from manual management to standardized system processing, effectively reducing the probability of human error and ensuring the accuracy and consistency of records.

[0136] In other words, this application involves the management and use of large model pre-training data processing, including:

[0137] Efficient management of large-scale datasets: This application achieves efficient management of large-scale data and its related metadata through a structured data storage method, ensuring that the data processing process is orderly and standardized, and optimizing the efficiency of data retrieval and access.

[0138] Rapid data matching and automatic recording: This application supports rapid data matching and automatically records each matching result to the database, avoiding omissions and errors that may occur during manual recording and ensuring the accuracy and traceability of the matching process.

[0139] Automated data format processing: This application pre-classifies data formats to suit different needs. For data in formats such as JSON, the system quickly generates processing scripts by specifying key / value pairs, avoiding the time and effort required for manual script writing.

[0140] Maximizes CPU resource utilization; this application can utilize only CPU resources and can be deployed in a GPU server cluster. When the GPU server is under high load, the system can make full use of idle CPU resources (e.g., for decompressing raw data compressed packages), avoiding waste of CPU resources.

[0141] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0142] This embodiment also provides a data management device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0143] Figure 8 This is a structural block diagram of a data management device according to an embodiment of this application, such as... Figure 8 As shown, the device includes:

[0144] The acquisition module 802 is used to acquire the training sample data uploaded by the user, as well as the data type and data format of the training sample data;

[0145] The first execution module 804 is used to generate and run a target script in response to the first interactive operation, and process training sample data through the target script. The target script is determined by the target key-value pairs and script template specified by the first interactive operation, and the script template is a script template corresponding to the data format.

[0146] The second execution module 806 is used to respond to the second interactive operation, extract target sample data from the training sample data according to the target ratio, and convert the target sample data into model training data. The target data type and target ratio of the target sample data are specified by the second interactive operation. The data type includes the target data type. The model training data represents the training set used by the training target large model.

[0147] In an exemplary embodiment, the above-described apparatus is further configured to: after acquiring training sample data uploaded by a user and the data type and data format of the training sample data, store the training sample data to a target node; in response to the storage of the training sample data to the target node, generate a storage path, wherein the storage path is used to indicate the storage address of the training sample data on the target node; and store the data identifier, data type, data format and storage path corresponding to the training sample data to a database.

[0148] In an exemplary embodiment, the above-described apparatus is used to store training sample data to a target node in the following manner: obtaining the amount of training sample data; searching for a target node on a target server based on the amount of data, wherein the target node represents a free node whose storage space meets the amount of data; and storing the training sample data to the target node in response to finding the target node.

[0149] In an exemplary embodiment, the above-described apparatus is configured to generate and run a target script in response to a first interactive operation, and process training sample data through the target script by: determining a script template based on a data format in response to the first interactive operation; determining target key-value pairs based on key fields in the training sample data in response to the first interactive operation; replacing target fields in the script template with target key-value pairs to generate a target script; and running the target script to process the training sample data.

[0150] In one exemplary embodiment, different data formats correspond to different target scripts. The apparatus is used to run the target scripts in at least one of the following ways to process training sample data: when a sampling operation needs to be performed on the training sample data, the target script is run to perform the sampling operation on the training sample data; when a cleaning operation needs to be performed on the training sample data, the target script is run to perform the cleaning operation on the training sample data; when a splicing operation needs to be performed on the training sample data, the target script is run to perform the splicing operation on the training sample data.

[0151] In an exemplary embodiment, the above-described apparatus is configured to respond to a second interactive operation by extracting target sample data from training sample data according to a target ratio and converting the target sample data into model training data in the following manner: in response to the second interactive operation, determining a dataset identifier and a target ratio; extracting target sample data from training sample data according to the dataset identifier and the target ratio; and calling a format conversion module to perform format conversion on the target sample data to obtain model training data.

[0152] In an exemplary embodiment, the apparatus is further configured to: after acquiring training sample data uploaded by a user and the data type and data format of the training sample data, acquire the data volume of the training sample data; search for a target node on a target server based on the data volume, wherein the target node represents a free node whose storage space meets the data volume; in response to finding the target node, store the training sample data to the target node; in response to storing the training sample data to the target node, generate a storage path, wherein the storage path is used to indicate the storage address of the training sample data on the target node; store the data identifier, data type, data format, and storage path corresponding to the training sample data in a database; and in response to a first interactive operation, determine a script template based on the data format and, according to the training sample data... The key fields in this data determine the target key-value pairs; the target key-value pairs are used to replace the target fields in the script template to automatically generate the target script and create a data processing task; the target script is run to execute the data processing task, and the processing results of the data processing task are reported to the database in real time for task status marking; in response to the second interactive operation, the data identifier, data type, data format, and storage path corresponding to the training sample data marked as completed in the database are displayed; the dataset identifier and target ratio are determined, where the dataset identifier indicates the identifier that meets the preset identifier conditions; target sample data is extracted from the training sample data according to the dataset identifier and target ratio; the format conversion module is called to convert the format of the target sample data to obtain the model training data.

[0153] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0154] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when run.

[0155] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0156] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0157] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0158] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0159] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0160] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A data management method, characterized in that, include: Obtain the training sample data uploaded by the user, as well as the data type and data format of the training sample data; In response to the first interactive operation, a target script is generated and run to process the training sample data. The target script is determined by the target key-value pairs and script template specified by the first interactive operation, and the script template is a script template corresponding to the data format. In response to the second interactive operation, target sample data is extracted from the training sample data according to the target ratio, and the target sample data is converted into model training data. The target data type and the target ratio of the target sample data are specified by the second interactive operation. The data type includes the target data type. The model training data represents the training set used to train the target large model. The step of generating and running a target script in response to a first interactive operation, and processing the training sample data through the target script, includes: determining a script template based on the data format in response to the first interactive operation; determining a target key-value pair based on key fields in the training sample data in response to the first interactive operation; replacing the target fields in the script template with the target key-value pair to generate the target script; and running the target script to process the training sample data.

2. The method according to claim 1, characterized in that, After obtaining the user-uploaded training sample data and the data type and format of the training sample data, the method further includes: The training sample data is stored in the target node; In response to the training sample data being stored in the target node, a storage path is generated, wherein the storage path is used to indicate the storage address of the training sample data on the target node; The data identifier, data type, data format, and storage path corresponding to the training sample data are stored in the database.

3. The method according to claim 2, characterized in that, The step of storing the training sample data to the target node includes: Obtain the amount of data from the training sample data; Based on the data volume, the target node is located on the target server, wherein the target node represents a free node whose storage space meets the data volume; In response to finding the target node, the training sample data is stored in the target node.

4. The method according to claim 1, characterized in that, Different data formats correspond to different target scripts. Running the target script to process the training sample data includes at least one of the following: When it is necessary to perform a sampling operation on the training sample data, the target script is run, and the sampling operation on the training sample data is performed through the target script. When it is necessary to perform cleaning operations on the training sample data, the target script is run to perform cleaning operations on the training sample data. When it is necessary to perform a splicing operation on the training sample data, the target script is run to perform the splicing operation on the training sample data.

5. The method according to claim 1, characterized in that, The step of responding to the second interactive operation by extracting target sample data from the training sample data according to a target ratio and converting the target sample data into model training data includes: In response to the second interactive operation, the dataset identifier and the target ratio are determined; Target sample data is extracted from the training sample data according to the dataset identifier and the target ratio; The format conversion module is invoked to convert the target sample data to obtain the model training data.

6. The method according to claim 1, characterized in that, The method further includes: After obtaining the training sample data uploaded by the user, as well as the data type and data format of the training sample data, the data volume of the training sample data is obtained. Based on the data volume, a target node is located on the target server, wherein the target node represents a free node whose storage space meets the data volume. In response to finding the target node, the training sample data is stored in the target node; In response to the training sample data being stored in the target node, a storage path is generated, wherein the storage path is used to indicate the storage address of the training sample data on the target node; The data identifier, data type, data format, and storage path corresponding to the training sample data are stored in the database; In response to the first interactive operation, the script template is determined based on the data format, and the target key-value pair is determined based on the key fields in the training sample data; Replace the target field in the script template with the target key-value pair, automatically generate the target script, and create a data processing task; The target script is run to execute the data processing task, and the processing results of the data processing task are reported to the database in real time for task status marking. In response to the second interactive operation, the data identifier, data type, data format, and storage path corresponding to the training sample data whose task status is marked as completed are displayed in the database; Determine the dataset identifier and the target ratio, wherein the dataset identifier represents an identifier that satisfies a preset identifier condition; Target sample data is extracted from the training sample data according to the dataset identifier and the target ratio; The format conversion module is invoked to convert the target sample data to obtain the model training data.

7. A data management device, characterized in that, include: The acquisition module is used to acquire the training sample data uploaded by the user, as well as the data type and data format of the training sample data; A first execution module is configured to generate and run a target script in response to a first interactive operation, and process the training sample data through the target script, wherein the target script is determined by a target key-value pair and a script template specified by the first interactive operation, and the script template is a script template corresponding to the data format; The second execution module is configured to, in response to the second interactive operation, extract target sample data from the training sample data according to a target ratio, and convert the target sample data into model training data, wherein the target data type and the target ratio of the target sample data are specified by the second interactive operation, the data type includes the target data type, and the model training data represents the training set used to train the target large model; The apparatus is configured to generate and run a target script in response to a first interactive operation, and process the training sample data through the target script, by: determining the script template based on the data format in response to the first interactive operation; determining the target key-value pair based on key fields in the training sample data in response to the first interactive operation; replacing the target fields in the script template with the target key-value pair to generate the target script; and running the target script to process the training sample data.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of claims 1 to 6.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 6.

Images