A data set management method, system, device, equipment, medium and product

Abstract

The present invention relates to the field of artificial intelligence technology, and specifically discloses a data set management method, system, device, equipment, medium and product. Sample data in a file of a target data set is read by an artificial intelligence platform device according to a file directory of a locally stored target data set; a preview file of the target data set is created according to the sample data, and the preview file is sent to a user device so that the user can determine the sample data to be modified through the preview file, and a sample data modification command is sent to the artificial intelligence platform device through the user device for it to modify the target data set, thereby achieving an improvement in the visualization level of the data set of the artificial intelligence platform and the ease of modifying massive file data sets.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a data set management method, system, device, equipment, medium and product. Background Art

[0002] With the development of artificial intelligence (AI) technology, training AI models can achieve a variety of functions. To facilitate model training tasks or model fine-tuning tasks, the AI ​​platform provides data set management functions, including data set addition, deletion, and modification, tracing processing flow, and data set multi-version release, so that users can publish the data set after making changes to the AI ​​platform data set to submit it to model training tasks or model fine-tuning tasks.

[0003] However, the dataset management function provided by the current artificial intelligence platform has poor dataset visualization effect. Users need to search for files that need to be modified from massive file lists, which is time-consuming and labor-intensive, and the modification efficiency is low.

[0004] How to facilitate users to view and modify the data sets of artificial intelligence platforms is a technical problem that technicians in this field need to solve. Summary of the invention

[0005] The purpose of the present invention is to provide a data set management method, system, device, equipment, medium and product for improving the visualization level of the data set of an artificial intelligence platform to facilitate users to view and modify the data set of the artificial intelligence platform.

[0006] In order to solve the above technical problems, the present invention provides a data set management method, which is applied to an artificial intelligence platform device, comprising:

[0007] Reading sample data from a file of the target data set according to a file directory of the target data set stored locally;

[0008] Creating a preview file of the target data set according to the sample data;

[0009] Sending the preview file to a user device;

[0010] Modify the target data set according to the sample data modification command sent by the user equipment;

[0011] Wherein, the sample data modification command is determined according to the preview file.

[0012] On the one hand, modifying the target data set according to the sample data modification command sent by the user equipment includes:

[0013] Determine the slice file to be modified according to the sample data modification command;

[0014] Reading the slice file to be modified into the memory, modifying the slice file to be modified according to the sample data modification command, and obtaining a modified slice file;

[0015] Writing the modified slice file back to the local storage;

[0016] Among them, one file of the target data set corresponds to one or more slice files.

[0017] On the other hand, the step of importing the target data set into the local storage includes:

[0018] Slicing the file of the target data set according to a preset slice size to obtain the slice file;

[0019] The slice file is stored in the local storage.

[0020] On the other hand, slicing the file of the target data set according to a preset slice size to obtain the slice file includes:

[0021] According to the order of the file directory, the sample data in the file is read one by one and written into the slice file until the data amount of the sample data in the slice file reaches the preset slice size, thereby obtaining a slice file.

[0022] On the other hand, determining the slice file to be modified according to the sample data modification command includes:

[0023] Determine the information of the sample data to be modified and the sample data modification type according to the sample data modification command;

[0024] If the sample data modification type is to edit sample data or to delete sample data, determining that the slice file where the sample data to be modified is located is the slice file to be modified;

[0025] If the sample data modification type is adding sample data, then the last slice file of the file to be modified is determined as the slice file to be modified.

[0026] On the other hand, it also includes:

[0027] After receiving a data set publishing command for the target data set, merging the slice files corresponding to the same file to obtain a final version of the target data set;

[0028] Release the final version of the target dataset.

[0029] On the other hand, it also includes:

[0030] After modifying the target data set according to the file modification command sent by the user equipment, updating the preview file;

[0031] The updated preview file is sent to the user device.

[0032] On the other hand, after the target data set is modified according to the file modification command sent by the user equipment, updating the preview file includes:

[0033] Creating a file modification tag according to the file modification type of the file modification command;

[0034] The preview file is updated according to the file directory and the file modification tag.

[0035] On the other hand, the file modification tag includes a new file tag and a deleted file tag;

[0036] Creating a file modification tag according to the file modification type of the file modification command includes:

[0037] If the file modification type is a new file addition operation, after the newly added file is stored in the new version directory corresponding to the target data set, the newly added file tag corresponding to the newly added file is created;

[0038] If the file modification type is a file modification operation, after the modified file is stored in the new version directory, the newly added file tag corresponding to the modified file is created, and the deleted file tag corresponding to the file before the modification is created;

[0039] If the file modification type is a file deletion operation, the file deletion tag corresponding to the deleted file is created.

[0040] On the other hand, it also includes:

[0041] After receiving a data set publishing command for the target data set, obtaining the file modification tag of the latest version corresponding to the current target data set;

[0042] Determine the final version of the target data set according to the file directory and the file modification tag of the latest version;

[0043] Release the final version of the target dataset.

[0044] On the other hand, the target data set is stored in the local storage in the form of a slice file;

[0045] Dataset management methods also include:

[0046] After receiving a data set publishing command for the target data set, determining the latest version of the file directory according to the file directory and the file modification tag;

[0047] According to the latest version of the file directory, the slice files corresponding to the same file are merged to obtain the final version of the target data set;

[0048] Release the final version of the target dataset.

[0049] On the other hand, according to the file directory of the target data set stored locally, reading the sample data in the file of the target data set includes:

[0050] According to the file directory, the sample data in each of the files of the target data set are traversed, and a first preset number of the sample data is read into the memory each time.

[0051] On the other hand, creating a preview file of the target data set according to the sample data includes:

[0052] Generate preview information corresponding to each sample data in sequence;

[0053] The preview information is written into the preview file so that the preview information of each page does not exceed the preset number of single-page data items.

[0054] On the other hand, the preview information is key-value data corresponding to the sample data;

[0055] Writing the preview information into the preview file so that the preview information on each page does not exceed the preset number of single-page data entries includes:

[0056] The key value data corresponding to the sample data is filled into the table of the preview file with the number of data rows per page not exceeding the preset number of data items on a single page. The header field of the table is the sample data type corresponding to the key value data of this page.

[0057] In order to solve the above technical problems, the present invention further provides a data set management method, which is applied to a user device and includes:

[0058] Receive a preview file of the target data set sent by the artificial intelligence platform device;

[0059] Generate a sample data modification command according to the preview file;

[0060] Sending the sample data modification command to the artificial intelligence platform device, so that the artificial intelligence platform device modifies the target data set according to the sample data modification command;

[0061] The preview file is created based on sample data in each file of the target data set.

[0062] In order to solve the above technical problems, the present invention also provides a data set management system, including an artificial intelligence platform device and a user device;

[0063] The artificial intelligence platform device is used to read sample data in a file of the target data set according to a file directory of the target data set stored locally; create a preview file of the target data set according to the sample data; send the preview file to the user device; modify the target data set according to a sample data modification command sent by the user device;

[0064] Wherein, the sample data modification command is determined according to the preview file.

[0065] In order to solve the above technical problems, the present invention also provides a data set management device, which is applied to an artificial intelligence platform device, comprising:

[0066] A reading unit, configured to read sample data from a file of the target data set according to a file directory of the target data set stored locally;

[0067] A creating unit, used for creating a preview file of the target data set according to the sample data;

[0068] A first sending unit, configured to send the preview file to a user device;

[0069] A first modification unit, configured to modify the target data set according to a sample data modification command sent by the user equipment;

[0070] Wherein, the sample data modification command is determined according to the preview file.

[0071] In order to solve the above technical problems, the present invention further provides a data set management device, comprising:

[0072] Memory for storing computer programs;

[0073] The processor is used to execute the computer program, and the computer program, when executed by the processor, implements the steps of any one of the data set management methods described above.

[0074] In order to solve the above technical problem, the present invention further provides a non-volatile storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the data set management method as described in any one of the above items are implemented.

[0075] In order to solve the above technical problem, the present invention further provides a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of any one of the above data set management methods are implemented.

[0076] The data set management method provided by the present invention has the beneficial effect that the artificial intelligence platform device reads the sample data in the file of the target data set according to the file directory of the target data set stored locally; creates a preview file of the target data set according to the sample data, and sends the preview file to the user device so that the user can determine the sample data that needs to be modified through the preview file, and sends the sample data modification command to the artificial intelligence platform device through the user device for it to modify the target data set, thereby achieving an improvement in the visualization level of the data set of the artificial intelligence platform and the ease of modifying massive file data sets.

[0077] The data set management system, device, equipment, medium and product provided by the present invention have the above-mentioned beneficial effects, which will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0079] Figure 1 A flowchart of a data set management method provided by an embodiment of the present invention;

[0080] Figure 2 An architectural diagram of a data set management system provided by an embodiment of the present invention;

[0081] Figure 3 A flowchart of another data set management method provided by an embodiment of the present invention;

[0082] Figure 4 A schematic diagram of the structure of a data set management device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0083] The core of the present invention is to provide a data set management method, system, device, equipment, medium and product for improving the visualization level of the data set of the artificial intelligence platform to facilitate users to view and modify the data set of the artificial intelligence platform.

[0084] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0085] To facilitate understanding of the technical solution provided by the embodiment of the present invention, some key terms used in the embodiment of the present invention are first explained here.

[0086] The Artificial Intelligence (AI) platform is a system or framework that supports the development, deployment and management of AI applications and services, and provides functions such as data management, model training, deployment and monitoring. Dataset management is an important function of the AI ​​platform, which includes adding, deleting and modifying datasets, tracing processing procedures, and publishing multiple versions of datasets. Through this function, users can import datasets into the AI ​​platform, modify and publish datasets on the AI ​​platform, and provide them for model training tasks or model fine-tuning tasks. It also supports data iteration and dataset version management.

[0087] The current AI platform provides a dataset modification function that displays a dataset file list on the platform. Users log in to the AI ​​platform's browser interface to view the file list and determine the file to be modified. However, for the datasets required for training and fine-tuning large-scale models, especially for language models, the amount of data contained in each file is increasing. If the platform still uses a file list to display the preview information of the target dataset, the following problems exist:

[0088] (1) The training data is relatively large, and it is difficult to directly open the file for preview;

[0089] (2) The fine-tuning data is mostly in object formats such as json and jsonl. Directly opening the file for preview is not intuitive enough, and it is impossible to turn pages. It is impossible to quickly browse and judge the data quality and eliminate low-quality data;

[0090] (3) Even if the file can be opened directly for preview, the process of editing sample data, adding sample data, or deleting sample data is inefficient.

[0091] It can be seen that the data set display scheme of the artificial intelligence platform in the related technology makes it very difficult for users to open files for viewing, and the raw data of the data set is usually structured data, and it is not convenient to directly open the file for viewing to judge the data quality. Therefore, the visualization level of the data set management function provided by the current artificial intelligence platform is low, which is specifically reflected in the inability to allow users to overview the specific content of the files in the data set. Even if the files in the data set can be opened for viewing, the readability is poor. This is not conducive to the needs of users to view and modify data sets through artificial intelligence platforms.

[0092] In order to improve the visualization level of the data set of the artificial intelligence platform so as to facilitate users to view and modify the data set of the artificial intelligence platform, the data set management solution provided in the embodiment of the present invention reads the sample data in the file of the target data set according to the file directory of the target data set stored locally by the artificial intelligence platform device; creates a preview file of the target data set according to the sample data, and sends the preview file to the user device so that the user can determine the sample data to be modified through the preview file, and sends the sample data modification command to the artificial intelligence platform device through the user device for it to modify the target data set, thereby achieving the improvement of the visualization level of the data set of the artificial intelligence platform and the ease of modifying massive file data sets.

[0093] The data set management method provided by the embodiment of the present invention is described below with reference to the accompanying drawings.

[0094] Figure 1 A flowchart of a data set management method provided by an embodiment of the present invention; Figure 2 An architectural diagram of a data set management system provided in an embodiment of the present invention.

[0095] like Figure 1 As shown, the data set management method provided by the embodiment of the present invention, applied to an artificial intelligence platform device, may include:

[0096] S101: Reading sample data from a file of a target data set according to a file directory of a target data set stored locally;

[0097] S102: Creating a preview file of the target data set according to the sample data;

[0098] S103: Send the preview file to the user device;

[0099] S104: modifying the target data set according to the sample data modification command sent by the user equipment;

[0100] Among them, the sample data modification command is determined according to the preview file.

[0101] In an embodiment of the present invention, the artificial intelligence platform device is a hardware device for running the artificial intelligence platform, which can be a single-node device or an artificial intelligence platform cluster composed of multiple server nodes.

[0102] like Figure 2 As shown, the functional modules of the artificial intelligence platform may include a file management module, a data set management module, and a data set usage module. Among them, the file management module is used to receive the original data (which can be called raw data) of the externally imported data set and store it in the local storage of the artificial intelligence platform device. The data set management module is used to implement data visualization, data modification, version management, state management and other functions of the data set in the local storage of the artificial intelligence platform. The data set usage module is used to call the published data set to perform model training tasks or model fine-tuning tasks. The functional modules of the artificial intelligence platform can be located on the same artificial intelligence platform device or different devices in the artificial intelligence platform cluster.

[0103] The user device is a device used by the user to log in to the artificial intelligence platform, and the user device and the artificial intelligence platform device can communicate through the Hypertext Transfer Protocol (HTTP). In an embodiment of the present invention, the user can log in to the browser interface provided by the artificial intelligence platform through the user device to access the application of the artificial intelligence platform, so as to view the data set of the artificial intelligence platform and modify and publish it.

[0104] In the embodiment of the present invention, the target data set is a data set stored in the local storage of the artificial intelligence platform device, and is also the object that the user needs to modify. Local storage refers to non-volatile storage devices local to the artificial intelligence platform device, such as hard disks (HDDs), solid-state drives (SSDs), etc. In the artificial intelligence platform cluster, some server nodes can be configured as computing nodes, and other server nodes can be configured as storage nodes. Then, file management modules, data set management modules, and data set use modules can be deployed on the computing nodes, and storage nodes can be configured as shared storage for each computing node. The data set is stored in the shared storage for easy calling by each functional module.

[0105] The data sets in the local storage of artificial intelligence platform devices can come from three sources: data sets uploaded by user devices calling the upload function of the file management module of the artificial intelligence platform; using the data download function of the artificial intelligence platform to download remote data (such as open source website links, file transfer protocol (FTP) server data) to the artificial intelligence platform device and manage it by the file management module; using the platform data processing tools of the artificial intelligence platform to process native data into the data required for model training or model fine-tuning and then save it to the file management module.

[0106] In the embodiment of the present invention, the data set managed by the file management module can be called raw data, that is, the original data that is not managed by the data set management module, and exists in the shared storage of the artificial intelligence platform in the form of files. Then, the data set file of the file management module is imported into the data set management module, and the artificial intelligence platform manages the data.

[0107] The dataset management module can support functions such as data visualization, data modification, version management, and status management, and ultimately cache the dataset to the running node of the model training task or model fine-tuning task for use.

[0108] By applying the data set management method provided in the embodiment of the present invention, a user can access the file management module of an artificial intelligence platform through a user device, select data from the files under the file management module as the import data path of the data set, and the artificial intelligence platform imports the selected data into the data set management module for management.

[0109] To achieve version management when users modify datasets, version numbers can be assigned to each version of the dataset starting from the initial version of the dataset imported from the file management module to the dataset management module (e.g., the initial version is recorded as V001). To reduce the local storage space occupied by the dataset, the dataset can be stored in the shared storage of the file management module and the dataset management module. When the original dataset (raw data) is imported from the file management module to the dataset management module, only one copy of the original dataset is saved in the shared storage without modification, and directories are divided in the shared storage to create version directories for each version of the dataset to store the modified data of the version. The artificial intelligence platform uses business control methods to manage the import path of the dataset, and displays the shared icon imported into the dataset management module in the file management module for distinction.

[0110] In an embodiment of the present invention, a data set that is managed in a data set management module and has not been published can be used as a modifiable target data set. At this time, the artificial intelligence platform device can read sample data in the file of the target data set from the local storage into the memory, and generate preview information of the sample data to create a preview file of the target data set. The preview file is sent to the user device, that is, the preview file is loaded on the browser interface of the artificial intelligence platform of the user device for the user to view.

[0111] The data set preview solution provided by the embodiment of the present invention allows users to view the specific content of the sample data contained in the target data set file through the preview file, rather than opening the files one by one on the artificial intelligence platform. Users can quickly determine the quality of the sample data based on the preview file, and can perform data modification operations such as removing sensitive words, deleting selected lines, and adding lines to the end of the file on the previewed data.

[0112] The user can determine the sample data that needs to be modified through the preview information in the preview file, and can select the sample data to be modified or edit the sample data modification command in the browser interface. The user device sends the sample data modification command to the artificial intelligence platform device, so that the artificial intelligence platform device can modify the target data set according to the sample data modification command.

[0113] The dataset management method provided in the embodiment of the present invention reads sample data in the files of the target dataset according to the file directory of the target dataset stored locally through an artificial intelligence platform device; creates a preview file of the target dataset according to the sample data, and sends the preview file to a user device so that the user can determine the sample data that needs to be modified through the preview file, and sends a sample data modification command to the artificial intelligence platform device through the user device for it to modify the target dataset, thereby improving the visualization level of the dataset of the artificial intelligence platform and facilitating the modification of massive file datasets.

[0114] Based on the above embodiment, the embodiment of the present invention further illustrates a preview method of a target data set.

[0115] In the embodiment of the present invention, the preview file may be generated after the data set is imported from the file management module into the data set management module and the target data set is managed by the data set management module, that is, the preview file of the target data set is generated. Alternatively, the preview file of the target data set may be generated when the artificial intelligence platform device receives the data set modification command sent by the user device according to the information of the target data set recorded in the command.

[0116] In some optional implementations of the embodiments of the present invention, S101 reads sample data in files of the target data set according to the file directory of the target data set stored locally, which may include: traversing the sample data in each file of the target data set according to the file directory, and reading a first preset number of sample data into the memory each time.

[0117] When the target dataset is imported into the dataset management module and has not been modified, the file directory is the file directory of the original dataset. When the target dataset is modified at the file level (adding files, deleting files, etc.), the file directory can be updated with the modified version of the target dataset.

[0118] Determine the file structure of the target dataset based on the file directory, such as whether to include subsets or multi-level directories. If the target dataset includes subsets, it can be divided according to the subsets, and corresponding preview files can be generated for each subset. It is also possible not to divide the subsets and directly store the target dataset files flatly. In this case, only the sample data of the first-level directory of the target dataset can be previewed, and the sample data of the nested subdirectories will not be displayed.

[0119] When generating a preview file of a target data set for the first time, it may be necessary to traverse all sample data of the target data set. If all sample data of the target data set is loaded into the memory at one time, it may cause great memory pressure and disk IO pressure on the microservice, resulting in excessive microservice resource usage and abnormality. For this reason, in an embodiment of the present invention, only the first preset number of sample data is read into the memory each time, and the sample data of the target data set is traversed by multiple readings.

[0120] In an embodiment of the present invention, the size of the first preset number can be determined according to the size of the memory space allocated to the data set management module by the artificial intelligence platform device. In order to improve the efficiency of generating preview files, in an embodiment of the present invention, a multi-threaded parallel reading method can be used to read sample data from multiple file starting points of the target data set into the memory, and multiple threads are used to respectively count the sample data of each part of the target data set and generate preview information, so as to reduce the waiting time of the computing thread caused by the reading process.

[0121] In the related art, the data set is visualized in the form of a file list. When users open the files in the data set, they are usually in notepad format and cannot turn pages. In addition, since the sample data in the language model data set is usually in object formats such as json and jsonl, directly opening the file for preview is not intuitive enough.

[0122] Therefore, in the embodiment of the present invention, creating a preview file of the target data set according to the sample data in S102 may include: generating preview information corresponding to each sample data in sequence; writing the preview information into the preview file so that the preview information of each page does not exceed the preset number of single-page data. That is, a preview file with pages is generated by presetting the number of single-page data, so that the user can flip pages to browse conveniently and can choose which pages to preview and which pages not to preview.

[0123] In addition, in response to the problem that the original sample data is not easy to read, in an embodiment of the present invention, the key-value information contained in the sample data can be obtained and parsed into a table form for display, so as to achieve the effect that the preview information is easier to read than the sample data. In an embodiment of the present invention, the preview information can be the key-value data corresponding to the sample data; the preview information is written into the preview file so that the number of preview information per page does not exceed the preset number of single-page data items, which can include: the key-value data corresponding to the sample data is filled into the table of the preview file so that the number of data rows per page does not exceed the preset number of single-page data items, and the header field of the table is the sample data type corresponding to the key-value data of this page. Usually, the data types of sample data belonging to the same data set are the same, such as a name data set, an age data set, a height data set, etc., so the header fields of each page of the preview file of the target data set can be dynamically displayed according to the data type of the corresponding sample data. The preset number of single-page data items can be the number of single-page data items configured by the user, or determined according to the page settings of the user page of the artificial intelligence platform.

[0124] In an embodiment of the present invention, since the preview file displayed to the user needs to be transmitted from the artificial intelligence platform device to the user device, the more sample data there is, the larger the preview file corresponding to the entire target data set, and some of the sample data may not be needed by the user. In some optional implementations of the embodiment of the present invention, sending the preview file to the user device in S103 may include: sending the file directory of the target data set to the user device; receiving the area of ​​files to be previewed determined according to the file directory sent by the user device; and sending the preview file of the area of ​​files to be previewed to the user device. That is to say, the file directory of the target data set can be loaded first in the browser interface of the user device, and the first-level directory can be loaded first, and then the sub-directory can be loaded under the user's trigger, so that the user can determine the area of ​​files to be previewed that actually needs to be modified layer by layer and then load the preview file of the corresponding area, instead of loading all the preview files of the target data set at one time.

[0125] In addition, sending the preview file to the user device in S103 may also include: sending the corresponding page in the preview file to the user device according to the number of pages to be previewed sent by the user device. That is, the user can choose to browse the preview file page by page in the order of the file directory until a certain page in the middle ends, or jump to browse, and in this case, there is no need to load other pages of the preview file on the user device.

[0126] In addition, creating a preview file of the target data set based on the sample data in S102 may also include: dividing the target data set into multiple file areas, and generating overview information corresponding to the sample data contained in the file areas. Sending the preview file to the user device in S103 may also include: sending the overview information corresponding to the multiple file areas of the target data set to the user device; receiving the file area to be previewed determined according to the overview information sent by the user device; and sending the preview file of the file area to be previewed to the user device. Different from the preview information that carries the same amount of information as the sample data, the overview information only reflects part of the information of the sample data corresponding to the file area, and its data volume is much smaller than all the preview information corresponding to the sample data in the file area, and can be used by the user to make a preliminary judgment on whether it is necessary to further preview the sample data in this file area, thereby further reducing the data volume of the transmitted preview file.

[0127] In summary of the above schemes, creating a preview file of the target data set based on the sample data in S102 may include: dividing the target data set into multiple file areas, and generating overview information corresponding to the sample data contained in the file areas. Sending the preview file to the user device in S103 may include: sending the overview information corresponding to the multiple file areas of the target data set to the user device; receiving the first file area to be previewed determined according to the overview information sent by the user device; sending the file directory corresponding to the first file area to be previewed to the user device; receiving the second file area to be previewed determined in the first file area to be previewed according to the file directory sent by the user device; and sending the preview file of the second file area to be previewed to the user device.

[0128] Based on the above embodiment, the data set management method provided by the embodiment of the present invention may further include: updating the preview file after modifying the target data set according to the file modification command sent by the user device; and sending the updated preview file to the user device.

[0129] In an embodiment of the present invention, the artificial intelligence platform can also provide users with file-level modification functions for the target data set, that is, files can be added, deleted, and edited.

[0130] As described in the above embodiment of the present invention, the data set management module can maintain a new version directory after the target data set is modified at the file level. For ease of management, in the embodiment of the present invention, after the target data set is modified according to the file modification command sent by the user device, updating the preview file can include: creating a file modification tag according to the file modification type of the file modification command; updating the preview file according to the file directory and the file modification tag. That is, after the file level modification occurs, the preview file corresponding to the target data set also needs to be modified to ensure that the preview file corresponds to the latest version of the target data set.

[0131] In some optional implementations of the embodiments of the present invention, the file modification tag may include a new file tag and a deleted file tag. Creating a file modification tag according to the file modification type of the file modification command may include: if the file modification type is a new file operation, after storing the newly added file in the new version directory corresponding to the target data set, creating a new file tag corresponding to the newly added file; if the file modification type is a modify file operation, after storing the modified file in the new version directory, creating a new file tag corresponding to the modified file, and creating a deleted file tag corresponding to the file before modification; if the file modification type is a delete file operation, creating a deleted file tag corresponding to the deleted file.

[0132] An example of adding a new file tag (ADD tag) is as follows:

[0133] {

[0134] "data":[ / user1 / data1 / dataset.json]

[0135] "operationTime":"Operation time"

[0136] }

[0137] The "data" field is a valid field of the newly added file tag, which is used to indicate the storage location of the newly added file. The "operationTime" is the operation time, which records the operation time of the newly added file tag and plays the role of log recording.

[0138] The following is an example of deleting a file tag (DELETE tag):

[0139] {

[0140] "data":[ / dataset / V001 / dataset.json] / / Data location: the newly added label is the location of the newly added data set; the deleted label is the original data location

[0141] "operationTime":"Operation time"

[0142] }

[0143] The "data" field is a valid field for deleting file tags, which is used to indicate the storage location of the deleted file. The "operationTime" is the operation time, which records the operation time of deleting the file tag and plays the role of log record.

[0144] In other implementations of the embodiment of the present invention, other types of file modification tags may also be set.

[0145] By integrating the file directory of the initial target data set and the file modification tags of each version, the file directory of the current latest version can be obtained, and the file directory corresponds to the final version of the target data set. The data set management method provided by the embodiment of the present invention may also include: after receiving a data set publishing command for the target data set, obtaining the latest version of the file modification tag corresponding to the current target data set; determining the final version of the target data set according to the file directory and the latest version of the file modification tag; and publishing the final version of the target data set.

[0146] Based on the above embodiment, the embodiment of the present invention further illustrates the step of modifying the target data set according to the sample data modification command.

[0147] Figure 3 A flowchart of another data set management method provided by an embodiment of the present invention.

[0148] Since the target data set is stored in the local storage of the artificial intelligence platform device in files, in order to ensure data integrity, the file system restricts the artificial intelligence platform device to read the entire file into the memory for modification when modifying the target data set, even if only one sample data in a file needs to be modified, and the entire file needs to be written back to the local storage after modification. After testing, when the local storage uses a solid-state drive, it takes about 5 seconds to modify 1GB of data. It can be seen that the efficiency of artificial intelligence platform devices in modifying data sets is low, and the problem becomes more prominent as the file size increases.

[0149] In this regard, in an embodiment of the present invention, the target data set is stored in the local storage in the form of a slice file, that is, the storage in the form of a file is improved to the storage in the form of a slice file. In this embodiment of the present invention, modifying the target data set according to the sample data modification command sent by the user device in S104 may include: determining the slice file to be modified according to the sample data modification command; reading the slice file to be modified into the memory, modifying the slice file to be modified according to the sample data modification command, and obtaining the modified slice file; writing the modified slice file back to the local storage; wherein one file of the target data set corresponds to one or more slice files.

[0150] In a specific implementation, the original data set of the target data set can be stored in the shared storage of the file management module and the data set management module, and when the data set management module manages the target data set, a sliced ​​target data set is stored. This is the underlying storage design and does not need to be presented in the user interface.

[0151] In an embodiment of the present invention, the step of importing the target data set into local storage may include: slicing the file of the target data set according to a preset slice size to obtain a slice file; and storing the slice file into local storage.

[0152] Among them, the preset slice size can be determined according to the memory space size allocated to the dataset management module by the artificial intelligence platform device, for example, it can be 100MB.

[0153] In a specific implementation, slicing the target data set file according to the preset slice size to obtain a slice file may include: reading sample data from the file one by one and writing them into the slice file according to the order of the file directory until the amount of sample data in the slice file reaches the preset slice size, thereby obtaining a slice file. That is, the integrity of the sample data in the slice file must be ensured, and it must be ensured that the end of each slice file must end with a complete sample data. When writing the sample data into the slice file, the sample data in the file may be read one by one and written into the slice file according to the order of the file directory, until the size of the slice file exceeds the preset slice size if one sample data is added, and does not exceed it otherwise, thereby obtaining a slice file.

[0154] For each slice file, the artificial intelligence platform can maintain a description file to describe the number of rows of sample data corresponding to each slice file. The slice files corresponding to the initial version of the target dataset can be stored in the / dataset / V001 directory, which is invisible in the platform. They can be named part_000.json, part_001.json, part_002.json... in the underlying storage. These slice files will not be displayed in the file directories of each version of the target dataset, that is, the file directory presented to the user only needs to be generated based on the initial file directory and file modification tags, and the preview file can be generated based on the order of the file directory. This slice management method is imperceptible to users.

[0155] The description file may include:

[0156] {

[0157] "chunk_lengths":[32127,32295,32138,32286,32200,32239,32216,32167,32338,32147,2842]

[0158] }

[0159] Among them, "chunk_lengths" indicates the number of rows of sample data corresponding to the slice file.

[0160] Then, in this slice storage mode, determining the slice file to be modified according to the sample data modification command may include: determining the information of the sample data to be modified and the sample data modification type according to the sample data modification command; if the sample data modification type is to edit the sample data or delete the sample data, then determining the slice file where the sample data to be modified is located as the slice file to be modified; if the sample data modification type is to add sample data, then determining the last slice file of the file to be modified as the slice file to be modified. That is to say, when it is necessary to edit or delete a row of the target data set, it is only necessary to find the corresponding slice file to modify and save the sample data. When it is necessary to add a row to the target data set, for the convenience of management, the sample data can be appended at the end of the last slice file corresponding to the file. When the number of added rows causes the size of the last slice file to exceed the preset slice size, the new slice file is written to the excess part. After executing the sample data modification command, modify the number of rows of the corresponding slice file in the description file.

[0161] To improve the efficiency of modifying slice files, in an embodiment of the present invention, reading the slice file to be modified into the memory may include: traversing the sample data in the slice file to be modified according to the description file, and reading a second preset number of sample data into the memory each time. The size of the second preset number can be determined according to the size of the memory space allocated to the data set management module by the artificial intelligence platform device. To improve the efficiency of modifying slice files, in an embodiment of the present invention, an asynchronous read and write method can be used to read the slice file to be modified into the memory, which may include: using a read thread to read the slice file to be modified from the local storage into the memory; writing the modified slice file back to the local storage may include: using a write thread to write the modified slice file back to the local storage. To further improve the efficiency of modifying slice files, multiple read threads and multiple write threads can be set. The read thread is used to read the slice file to be modified from the local storage to the memory, which may include: using multiple read threads to read sample data from multiple data starting points of the slice file to be modified into the memory; after traversing each slice file to be modified in the memory to determine the actual modified sample data and modifying it, the write thread is used to write the modified slice file back to the local storage, which may include: using multiple write threads to write each part of the modified slice file back to the local storage.

[0162] The dataset management method provided by the embodiment of the present invention may further include: after receiving a dataset publishing command for a target dataset, merging slice files corresponding to the same file to obtain a final version of the target dataset; and publishing the final version of the target dataset.

[0163] In combination with the file modification tags introduced in the above-mentioned embodiments of the present invention, when the target dataset is stored in the form of slice files in the local storage, the dataset management method provided by the embodiments of the present invention may further include: after receiving a dataset publishing command for the target dataset, determining the latest version of the file directory according to the file directory and the file modification tags; according to the latest version of the file directory, merging the slice files corresponding to the same file to obtain the final version of the target dataset; and publishing the final version of the target dataset.

[0164] like Figure 2 As shown, when the target dataset is imported into the dataset management module, the preview mode can be determined by subset or by directly storing the file in a flat manner. When the target dataset is imported, the initial version (V001 version) is automatically generated. If the target dataset has a subset, each subset is traversed and the size of the file in the subset is obtained. If it is larger than the preset slice size, the sample data is read one by one according to the preset slice size for slicing. The slicing process can be tracked and recorded through the transfer list. When an exception occurs, the failure reason is displayed in the transfer list, and the failed slice file is deleted. If the target dataset does not have a subset, the files in the first-level directory of the target dataset are traversed and the file size is obtained. If it is larger than the preset slice size, the sample data is read one by one according to the preset slice size for slicing. After the slicing is completed, the number of samples of each slice file in the description file needs to be synchronously maintained.

[0165] After the initial version of the target dataset is created, the target dataset version is in the unpublished state by default.

[0166] The target dataset in the unpublished state can be edited. To edit or delete a row, you need to find the corresponding slice file and modify and save the data.

[0167] After a file-level modification occurs, the AI ​​platform device needs to perform the following operations:

[0168] (1) Synchronously write file modification tags to be used for reassembling data when displaying the file directory of the target dataset.

[0169] (2) Update the total number of samples in the cached data when the number of samples in the target data set is first counted.

[0170] If the target dataset version has been modified and is to be used for model training or model fine-tuning, the target dataset version needs to be published. The published target dataset version no longer supports data modification.

[0171] At the same time, the artificial intelligence platform will merge the slice files when publishing the target data set.

[0172] The process of publishing the target dataset version may include: if the file is sliced, merging the slice files and renaming them to the corresponding file names in the initial version. The process of merging slice files can be tracked and recorded through the transfer list. When an exception occurs, the failure reason is displayed in the transfer list, and the failed slice files are deleted.

[0173] Based on the above embodiment, the embodiment of the present invention also provides a data set management method applied to a user device, which may include:

[0174] Receive a preview file of the target data set sent by the artificial intelligence platform device;

[0175] Generate sample data modification commands based on preview files;

[0176] Sending a sample data modification command to the artificial intelligence platform device so that the artificial intelligence platform device modifies the target data set according to the sample data modification command;

[0177] The preview file is created based on the sample data in each file of the target data set.

[0178] The specific implementation of the data set management method provided in the embodiment of the present invention can refer to the description of the above embodiment.

[0179] The dataset management method provided in the embodiment of the present invention reads sample data in the files of the target dataset according to the file directory of the target dataset stored locally through an artificial intelligence platform device; creates a preview file of the target dataset according to the sample data, and sends the preview file to a user device so that the user can determine the sample data that needs to be modified through the preview file, and sends a sample data modification command to the artificial intelligence platform device through the user device for it to modify the target dataset, thereby improving the visualization level of the dataset of the artificial intelligence platform and facilitating the modification of massive file datasets.

[0180] It should be noted that in the embodiments of the data set management method of the present invention, some of the steps or features may be ignored or not executed. The hardware or software functional modules divided for the convenience of description are not the only implementation form of the data set management method provided in the embodiments of the present invention.

[0181] The above describes in detail various embodiments corresponding to the data set management method. On this basis, the present invention also discloses a data set management system, apparatus, device, non-volatile storage medium and computer program product corresponding to the above method.

[0182] Applied to an artificial intelligence platform device, the data set management device provided by the embodiment of the present invention may include:

[0183] A reading unit, used for reading sample data from a file of the target data set according to a file directory of the target data set stored locally;

[0184] A creation unit, used for creating a preview file of a target data set according to sample data;

[0185] A first sending unit, configured to send the preview file to a user device;

[0186] A first modification unit, configured to modify a target data set according to a sample data modification command sent by a user device;

[0187] Among them, the sample data modification command is determined according to the preview file.

[0188] The data set management device provided in the embodiment of the present invention reads sample data in the files of the target data set according to the file directory of the target data set stored locally through an artificial intelligence platform device; creates a preview file of the target data set according to the sample data, and sends the preview file to a user device so that the user can determine the sample data that needs to be modified through the preview file, and sends a sample data modification command to the artificial intelligence platform device through the user device for it to modify the target data set, thereby achieving an improvement in the visualization level of the data set of the artificial intelligence platform and the ease of modifying massive file data sets.

[0189] In some optional implementations of the embodiments of the present invention, the first modification unit modifies the target data set according to the sample data modification command sent by the user device, which may include: determining the slice file to be modified according to the sample data modification command; reading the slice file to be modified into the memory, modifying the slice file to be modified according to the sample data modification command to obtain the modified slice file; writing the modified slice file back to the local storage; wherein, one file of the target data set corresponds to one or more slice files.

[0190] The step of importing the target data set into local storage may include: slicing the file of the target data set according to a preset slice size to obtain a slice file; and storing the slice file into local storage.

[0191] Slicing the target data set file according to a preset slice size to obtain a slice file may include: reading sample data from the file one by one and writing them into the slice file according to the order of the file directory until the amount of sample data in the slice file reaches the preset slice size, thereby obtaining a slice file.

[0192] Determining the slice file to be modified according to the sample data modification command may include: determining the information of the sample data to be modified and the sample data modification type according to the sample data modification command; if the sample data modification type is to edit the sample data or to delete the sample data, determining that the slice file where the sample data to be modified is located is the slice file to be modified; if the sample data modification type is to add sample data, determining that the last slice file of the file to be modified is the slice file to be modified.

[0193] The dataset management device provided by the embodiment of the present invention may also include: a first publishing unit, which is used to merge slice files corresponding to the same file after receiving a dataset publishing command for the target dataset to obtain a final version of the target dataset; and publish the merged target dataset.

[0194] The data set management device provided by the embodiment of the present invention may further include: a second modification unit, configured to update the preview file after modifying the target data set according to the file modification command sent by the user device; and send the updated preview file to the user device.

[0195] Among them, after the second modification unit modifies the target data set according to the file modification command sent by the user device, updating the preview file may include: creating a file modification tag according to the file modification type of the file modification command; and updating the preview file according to the file directory and the file modification tag.

[0196] In some optional implementations of the embodiments of the present invention, the file modification tag may include a new file tag and a deleted file tag; the second modification unit creates a file modification tag according to the file modification type of the file modification command, which may include: if the file modification type is a new file operation, then after storing the newly added file in the new version directory corresponding to the target data set, a new file tag corresponding to the newly added file is created; if the file modification type is a file modification operation, then after storing the modified file in the new version directory, a new file tag corresponding to the modified file is created, and a deleted file tag corresponding to the file before the modification is created; if the file modification type is a file deletion operation, then a deleted file tag corresponding to the deleted file is created.

[0197] The dataset management device provided by the embodiment of the present invention may also include: a second publishing unit, which is used to obtain the latest version of the file modification tag corresponding to the current target dataset after receiving the dataset publishing command for the target dataset; determine the final version of the target dataset according to the file directory and the latest version of the file modification tag; and publish the final version of the target dataset.

[0198] In some optional implementations of the embodiments of the present invention, the target dataset can be stored in the form of slice files in the local storage. The dataset management device provided by the embodiments of the present invention can also include: a third publishing unit, which is used to determine the latest version of the file directory according to the file directory and the file modification tag after receiving the dataset publishing command for the target dataset; merge the slice files corresponding to the same file according to the latest version of the file directory to obtain the final version of the target dataset; and publish the final version of the target dataset.

[0199] In some optional implementations of the embodiments of the present invention, the reading unit reads the sample data in the files of the target data set according to the file directory of the target data set stored locally, which may include: traversing the sample data in each file of the target data set according to the file directory, and reading a first preset number of sample data into the memory each time.

[0200] In some optional implementations of the embodiments of the present invention, the creation unit creates a preview file of the target data set based on the sample data, which may include: generating preview information corresponding to each sample data in sequence; writing the preview information into the preview file so that the preview information per page does not exceed the preset number of single-page data items.

[0201] In some optional implementations of the embodiments of the present invention, the preview information may be key-value data corresponding to the sample data; the creation unit writes the preview information into the preview file so that the number of preview information per page does not exceed the preset number of data items on a single page, which may include: filling the key-value data corresponding to the sample data into a table in the preview file so that the number of data rows per page does not exceed the preset number of data items on a single page, and the header field of the table is the sample data type corresponding to the key-value data of this page.

[0202] Applied to user equipment, the data set management device provided by the embodiment of the present invention may include:

[0203] A receiving unit, used to receive a preview file of a target data set sent by an artificial intelligence platform device;

[0204] A generating unit, used for generating a sample data modification command according to a preview file;

[0205] A second sending unit is used to send a sample data modification command to the artificial intelligence platform device, so that the artificial intelligence platform device modifies the target data set according to the sample data modification command;

[0206] The preview file is created based on the sample data in each file of the target data set.

[0207] The data set management device provided in the embodiment of the present invention reads sample data in the files of the target data set according to the file directory of the target data set stored locally through an artificial intelligence platform device; creates a preview file of the target data set according to the sample data, and sends the preview file to a user device so that the user can determine the sample data that needs to be modified through the preview file, and sends a sample data modification command to the artificial intelligence platform device through the user device for it to modify the target data set, thereby achieving an improvement in the visualization level of the data set of the artificial intelligence platform and the ease of modifying massive file data sets.

[0208] It should be noted that in each implementation of the data set management device provided in the embodiment of the present invention, the division of units is only a logical functional division, and other division methods can be used. The connection method between different units can be electrical, mechanical or other connection methods. Separated units can be located in the same physical location or distributed on multiple network nodes. Each unit can be implemented in the form of hardware or in the form of a software functional unit. That is, part or all of the units provided in the embodiment of the present invention can be selected according to actual needs and the corresponding connection method or integration method can be adopted to achieve the purpose of the embodiment of the present invention.

[0209] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.

[0210] The data set management system provided by the embodiment of the present invention may include an artificial intelligence platform device and a user device.

[0211] The artificial intelligence platform device is used to read sample data from files of the target data set according to the file directory of the target data set stored locally; create a preview file of the target data set according to the sample data; send the preview file to the user device; and modify the target data set according to the sample data modification command sent by the user device.

[0212] Among them, the sample data modification command is determined according to the preview file.

[0213] The dataset management system provided by the embodiment of the present invention reads sample data in the files of the target dataset according to the file directory of the locally stored target dataset through an artificial intelligence platform device; creates a preview file of the target dataset according to the sample data, and sends the preview file to a user device so that the user can determine the sample data that needs to be modified through the preview file, and sends a sample data modification command to the artificial intelligence platform device through the user device for it to modify the target dataset, thereby improving the visualization level of the dataset of the artificial intelligence platform and facilitating the modification of massive file datasets.

[0214] Since the embodiments of the system part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the system part, which will not be repeated here.

[0215] Figure 4 A schematic diagram of the structure of a data set management device provided by an embodiment of the present invention.

[0216] like Figure 4 As shown, the data set management device provided in the embodiment of the present invention includes: a memory 410, used to store a computer program 411; a processor 420, used to execute the computer program 411, and when the computer program 411 is executed by the processor 420, the steps of the data set management method provided in any of the above embodiments are implemented.

[0217] Among them, the processor 420 may include one or more processing cores, such as a 3-core processor, an 8-core processor, etc. The processor 420 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 420 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 420 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 420 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0218] The memory 410 may include one or more non-volatile storage media, which may be non-transitory. The memory 410 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 410 is at least used to store the following computer program 411, wherein the computer program 411, after being loaded and executed by the processor 420, can implement the relevant steps in the data set management method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 410 may also include an operating system 412 and data 413, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 412 may be Windows or other types of operating systems. Data 413 may include, but is not limited to, the data involved in the above method.

[0219] In some embodiments, the data set management device may further include a display screen 430 , a power supply 440 , a communication interface 450 , an input / output interface 460 , a sensor 470 , and a communication bus 480 .

[0220] Those skilled in the art will understand that Figure 4 The structure shown in the figure does not constitute a limitation of the data set management device, and may include more or fewer components than shown in the figure.

[0221] The data set management device provided in the embodiment of the present invention includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the steps of the data set management method provided in the above embodiment, and the effect is the same as above.

[0222] An embodiment of the present invention provides a non-volatile storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the data set management method provided in any one of the above embodiments can be implemented.

[0223] The non-volatile storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0224] For the introduction of the non-volatile storage medium provided in the embodiment of the present invention, please refer to the above method embodiment, and the effect thereof is the same as the data set management method provided in the embodiment of the present invention, and the present invention will not elaborate on it here.

[0225] An embodiment of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the data set management method provided in any one of the above embodiments.

[0226] For an introduction to the computer program product provided by the embodiment of the present invention, please refer to the above method embodiment, and the effect thereof is the same as the data set management method provided by the embodiment of the present invention, and the present invention will not elaborate on it here.

[0227] The above is a detailed introduction to a data set management method, system, device, equipment, medium and product provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can refer to each other. For the systems, devices, equipment, non-volatile storage media and computer program products disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can refer to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the present invention.

[0228] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A data set management method, characterized in that: Applied to artificial intelligence platform equipment, including: Reading sample data from a file of the target data set according to a file directory of the target data set stored locally; Creating a preview file of the target data set according to the sample data; Sending the preview file to a user device; Modify the target data set according to the sample data modification command sent by the user equipment; Wherein, the sample data modification command is determined according to the preview file; Modifying the target data set according to the sample data modification command sent by the user equipment includes: Determine the slice file to be modified according to the sample data modification command; Reading the slice file to be modified into the memory, modifying the slice file to be modified according to the sample data modification command, and obtaining a modified slice file; Writing the modified slice file back to the local storage; Among them, one file of the target data set corresponds to one or more slice files.

2. The data set management method according to claim 1, characterized in that: The step of importing the target data set into the local storage comprises: Slicing the file of the target data set according to a preset slice size to obtain the slice file; The slice file is stored in the local storage.

3. The data set management method according to claim 2, characterized in that: Slicing the file of the target data set according to a preset slice size to obtain the slice file includes: According to the order of the file directory, the sample data in the file is read one by one and written into the slice file until the data amount of the sample data in the slice file reaches the preset slice size, thereby obtaining a slice file.

4. The data set management method according to claim 1, characterized in that: Determining the slice file to be modified according to the sample data modification command includes: Determine the information of the sample data to be modified and the sample data modification type according to the sample data modification command; If the sample data modification type is to edit sample data or to delete sample data, determining that the slice file where the sample data to be modified is located is the slice file to be modified; If the sample data modification type is adding sample data, then the last slice file of the file to be modified is determined as the slice file to be modified.

5. The data set management method according to claim 1, characterized in that: Also includes: After receiving a data set publishing command for the target data set, merging the slice files corresponding to the same file to obtain a final version of the target data set; Release the final version of the target dataset.

6. The data set management method according to claim 1, characterized in that: Also includes: After modifying the target data set according to the file modification command sent by the user equipment, updating the preview file; The updated preview file is sent to the user device.

7. The data set management method according to claim 6, characterized in that: After modifying the target data set according to the file modification command sent by the user equipment, updating the preview file includes: Creating a file modification tag according to the file modification type of the file modification command; The preview file is updated according to the file directory and the file modification tag.

8. The data set management method according to claim 7, characterized in that: The file modification tags include adding a new file tag and deleting a file tag; Creating a file modification tag according to the file modification type of the file modification command includes: If the file modification type is a new file addition operation, after the newly added file is stored in the new version directory corresponding to the target data set, the newly added file tag corresponding to the newly added file is created; If the file modification type is a file modification operation, after the modified file is stored in the new version directory, the newly added file tag corresponding to the modified file is created, and the deleted file tag corresponding to the file before the modification is created; If the file modification type is a file deletion operation, the file deletion tag corresponding to the deleted file is created.

9. The data set management method according to claim 7, characterized in that: Also includes: After receiving a data set publishing command for the target data set, obtaining the file modification tag of the latest version corresponding to the current target data set; Determine the final version of the target data set according to the file directory and the file modification tag of the latest version; Release the final version of the target dataset.

10. The data set management method according to claim 7, characterized in that: The target data set is stored in the local storage in the form of a slice file; Dataset management methods also include: After receiving a data set publishing command for the target data set, determining the latest version of the file directory according to the file directory and the file modification tag; According to the latest version of the file directory, the slice files corresponding to the same file are merged to obtain the final version of the target data set; Release the final version of the target dataset.

11. The data set management method according to claim 1, characterized in that: Reading sample data from a file of the target data set according to a file directory of the locally stored target data set includes: According to the file directory, the sample data in each of the files of the target data set are traversed, and a first preset number of the sample data is read into the memory each time.

12. The data set management method according to claim 1, characterized in that: Creating a preview file of the target data set according to the sample data includes: Generate preview information corresponding to each sample data in sequence; The preview information is written into the preview file so that the preview information of each page does not exceed the preset number of single-page data items.

13. The data set management method according to claim 12, characterized in that: The preview information is the key value data corresponding to the sample data; Writing the preview information into the preview file so that the preview information on each page does not exceed the preset number of single-page data entries includes: The key value data corresponding to the sample data is filled into the table of the preview file with the number of data rows per page not exceeding the preset number of data items on a single page. The header field of the table is the sample data type corresponding to the key value data of this page.

14. A data set management method, characterized in that: Applied to user equipment, including: Receive a preview file of the target data set sent by the artificial intelligence platform device; Generate a sample data modification command according to the preview file; Sending the sample data modification command to the artificial intelligence platform device, so that the artificial intelligence platform device modifies the target data set according to the sample data modification command; Wherein, the preview file is created according to the sample data in each file of the target data set; The artificial intelligence platform device modifies the target data set according to the sample data modification command, including: The artificial intelligence platform device determines the slice file to be modified according to the sample data modification command; reads the slice file to be modified into the memory of the artificial intelligence platform device, modifies the slice file to be modified according to the sample data modification command, and obtains a modified slice file; writes the modified slice file back to the local storage of the artificial intelligence platform device; Among them, one file of the target data set corresponds to one or more slice files.

15. A data set management system, characterized in that: Including AI platform devices and user devices; The artificial intelligence platform device is used to read sample data from a file of the target data set according to a file directory of the target data set stored locally; Creating a preview file of the target data set according to the sample data; Sending the preview file to the user device; Modify the target data set according to the sample data modification command sent by the user equipment; Wherein, the sample data modification command is determined according to the preview file; Modifying the target data set according to the sample data modification command sent by the user equipment includes: Determine the slice file to be modified according to the sample data modification command; Reading the slice file to be modified into the memory, modifying the slice file to be modified according to the sample data modification command, and obtaining a modified slice file; Writing the modified slice file back to the local storage; Among them, one file of the target data set corresponds to one or more slice files.

16. A data set management device, characterized in that: Applied to artificial intelligence platform equipment, including: A reading unit, configured to read sample data from a file of the target data set according to a file directory of the target data set stored locally; A creating unit, used for creating a preview file of the target data set according to the sample data; A first sending unit, configured to send the preview file to a user device; A first modification unit, configured to modify the target data set according to a sample data modification command sent by the user equipment; Wherein, the sample data modification command is determined according to the preview file; Modifying the target data set according to the sample data modification command sent by the user equipment includes: Determine the slice file to be modified according to the sample data modification command; Reading the slice file to be modified into the memory, modifying the slice file to be modified according to the sample data modification command, and obtaining a modified slice file; Writing the modified slice file back to the local storage; Among them, one file of the target data set corresponds to one or more slice files.

17. A data set management device, characterized in that: include: Memory for storing computer programs; A processor is used to execute the computer program, and when the computer program is executed by the processor, the steps of the data set management method according to any one of claims 1 to 14 are implemented.

18. A non-volatile storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data set management method according to any one of claims 1 to 14 are implemented.

19. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the data set management method according to any one of claims 1 to 14 are implemented.

Images