Data processing method and device based on data set, equipment, medium and product

Abstract

The invention provides a data processing method and device based on a data set, equipment, a medium and a product. The method comprises the steps of obtaining a target data set task; wherein the target data set task is used for indicating to import and store a data set, the target data set task comprises a plurality of sub-tasks, and each sub-task corresponds to one storage object; analyzing each subtask in the target data set task to obtain source data and data records of the subtasks; wherein the source data represents a storage object in the subtask, and the data record represents attribute information of the source data; storing the source data of the subtasks in a preset first database to obtain address information of the source data of the subtasks in the preset first database; and storing the address information of the source data of the subtasks in a preset first database and the data records of the subtasks in a preset second database. According to the method, the data processing efficiency based on the data set is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a data processing method, device, equipment, medium and product based on a data set. Background Art

[0002] With the development of big model technology, massive data sets need to be stored, viewed or processed.

[0003] When the size of a data set grows rapidly, using a relational database to directly store a large amount of data sets will lead to poor database system performance and high storage costs. If the data set is directly stored on the disk in the form of a file, it will also face the disk throughput bottleneck.

[0004] Therefore, when storing, viewing or processing massive data sets, we face the problem of low data processing efficiency. Summary of the invention

[0005] The present application provides a data processing method, device, equipment, medium and product based on a data set, so as to solve the technical problem of low efficiency of data processing based on a data set.

[0006] In a first aspect, the present application provides a data processing method based on a data set, comprising:

[0007] Obtain a target data set task; wherein the target data set task is used to instruct to import and store the data set, and the target data set task includes multiple subtasks, each of which corresponds to a storage object;

[0008] Each subtask in the target data set task is parsed and processed to obtain the source data and data records of the subtask; wherein the source data represents the storage object in the subtask, and the data record represents the attribute information of the source data;

[0009] Storing the source data of the subtask in a preset first database, and obtaining the address information of the source data of the subtask in the preset first database;

[0010] The address information of the source data of the subtask in the preset first database and the data record of the subtask are stored in the preset second database.

[0011] Optionally, as in the above method, one row in the target data set task corresponds to one subtask; each subtask in the target data set task is parsed to obtain source data and data records of the subtask, including:

[0012] Split the target dataset task into multiple subtasks according to the row arrangement order in the target dataset task;

[0013] Each subtask in the target data set task is parsed and processed, the data content of the subtask is determined as the source data of the subtask, and the attribute information of the source data is determined as the data record of the subtask; wherein the data content of the subtask represents the storage object.

[0014] Optionally, the method as described above further includes:

[0015] Identify a preset row identifier from the target data set task; wherein the preset row identifier is used to distinguish subtasks;

[0016] According to the preset row identifier, the target data set task is split and processed to obtain multiple subtasks.

[0017] Optionally, the method as described above further includes:

[0018] For each subtask, determine the data format of the subtask;

[0019] If the data format of the subtask is the preset format, continue to parse each subtask in the target data set task, determine the data content of the subtask as the source data of the subtask, and determine the attribute information of the source data as the data record of the subtask.

[0020] Optionally, as in the above method, a thread pool is pre-set, and the thread pool includes multiple threads; each subtask in the target data set task is parsed and processed to obtain the source data and data records of the subtask, including:

[0021] Through multiple threads, multiple subtasks in the target data set task are parsed and processed to obtain the source data and data records of the subtasks; wherein each thread processes one subtask at a time.

[0022] Optionally, the method as described above further includes:

[0023] Determine the average speed of the thread pool; wherein the average speed represents the parsing speed of all threads in the thread pool;

[0024] If the average speed exceeds a preset speed threshold, the source data of the subtask is obtained from the thread pool, the data source of the subtask is stored in a preset first database, and the address information of the source data of the subtask in the preset first database is obtained;

[0025] The address information of the source data of the subtask in the preset first database and the data record of the subtask are stored in the preset second database.

[0026] Optionally, the method as described above further includes:

[0027] If the address information of the source data of the subtask in the preset first database and the data record of the subtask are stored in the preset second database, the processing status of the subtask is determined to be imported successfully; wherein the processing status represents the import progress of the subtask.

[0028] Optionally, the method as described above further includes:

[0029] In response to the export instruction for the subtask, acquiring address information of the source data of the subtask in the preset first database from the preset second database;

[0030] The source data of the subtask is acquired from the preset first database according to the address information of the source data of the subtask in the preset first database.

[0031] In a second aspect, the present application provides a data processing device based on a data set, comprising:

[0032] An acquisition unit is used to acquire a target data set task; wherein the target data set task is used to instruct to import and store the data set, and the target data set task includes multiple subtasks, each of which corresponds to a storage object;

[0033] A parsing unit, used to parse each subtask in the target data set task to obtain the source data and data records of the subtask; wherein the source data represents the storage object in the subtask, and the data record represents the attribute information of the source data;

[0034] A first storage unit, used to store the source data of the subtask in a preset first database, and obtain address information of the source data of the subtask in the preset first database;

[0035] The second storage unit is used to store the address information of the source data of the subtask in the preset first database and the data record of the subtask in the preset second database.

[0036] In a third aspect, the present application provides an electronic device, including: a memory, a processor;

[0037] Memory stores computer-executable instructions;

[0038] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0039] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the above first aspect and / or various possible implementations of the first aspect.

[0040] In a fifth aspect, the present application provides a computer program product, comprising: a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementations of the first aspect.

[0041] The data processing method, device, equipment, medium and product based on data set provided by the present application obtains the target data set task, parses and processes each subtask in the target data set task, obtains the source data and data record of the subtask, and then stores the source data of the subtask in a preset first database, obtains the address information of the source data of the subtask in the preset first database, and stores the address information of the source data of the subtask in the preset first database and the data record of the subtask in the preset second database; wherein, the target data set task is used to indicate the import and storage of the data set, the target data set task includes multiple subtasks, each subtask corresponds to a storage object, the source data represents the storage object in the subtask, and the data record represents the attribute information of the source data. By splitting the target data set task into subtasks, each subtask is parsed and processed, thereby improving the data processing efficiency based on the data set. At the same time, the source data corresponding to the subtask is stored in the preset first database, and the address information of the source data corresponding to the subtask in the preset first database and the data record of the subtask are stored in the preset second database. When the source data corresponding to the subtask needs to be obtained, the address information of the source data of the subtask in the preset first database can be quickly obtained from the preset second database, and then the source data of the subtask can be quickly found and obtained from the preset first database. The data processing method based on the data set provided by the present application improves the data processing efficiency based on the data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0043] Figure 1 A flowchart of a data processing method based on a data set provided in this application;

[0044] Figure 2 A flowchart of another data processing method based on a data set provided in this application;

[0045] Figure 3 A flowchart of another data processing method based on a data set provided in this application;

[0046] Figure 4 A schematic diagram of the architecture of a data processing method based on a data set provided in this application;

[0047] Figure 5 A schematic diagram of the structure of a data processing device based on a data set provided in this application;

[0048] Figure 6 A schematic diagram of the structure of another data processing device based on a data set provided by the present application;

[0049] Figure 7 A schematic diagram of the structure of the electronic device provided in this application.

[0050] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0051] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0052] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0053] With the development of big model technology, massive data sets need to be stored, viewed or processed.

[0054] Based on this background, the format of massive data sets is generally JSONL (JavaScript Object Notation Lines) format. It can be understood that each line of a text file in JSONL format can contain an independent JSON object.

[0055] When the size of a data set grows rapidly, using a relational database to directly store a large amount of data sets will lead to poor database system performance and high storage costs. If the data set is directly stored on the disk in the form of a file, it will also face a disk throughput bottleneck.

[0056] Therefore, when storing, viewing or processing massive data sets, we face the problem of low data processing efficiency.

[0057] The data processing method, device, equipment, medium and product based on data set provided by the present application obtains the target data set task, parses and processes each subtask in the target data set task, obtains the source data and data record of the subtask, and then stores the source data of the subtask in a preset first database, obtains the address information of the source data of the subtask in the preset first database, and stores the address information of the source data of the subtask in the preset first database and the data record of the subtask in the preset second database; wherein, the target data set task is used to indicate the import and storage of the data set, the target data set task includes multiple subtasks, each subtask corresponds to a storage object, the source data represents the storage object in the subtask, and the data record represents the attribute information of the source data. By splitting the target data set task into subtasks, each subtask is parsed and processed, thereby improving the data processing efficiency based on the data set. At the same time, the source data corresponding to the subtask is stored in the preset first database, and the address information of the source data corresponding to the subtask in the preset first database and the data record of the subtask are stored in the preset second database. When the source data corresponding to the subtask needs to be obtained, the address information of the source data of the subtask in the preset first database can be quickly obtained from the preset second database, and then the source data of the subtask can be quickly found and obtained from the preset first database. The data processing method based on the data set provided by the present application improves the data processing efficiency based on the data set.

[0058] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0059] Figure 1 This is a flow chart of a data processing method based on a data set provided in this application. The execution subject of this method can be a server, a host or other device, such as Figure 1 As shown, the method may include:

[0060] S101, obtaining a target data set task; wherein the target data set task is used to instruct to import and store a data set, and the target data set task includes a plurality of subtasks, and each subtask corresponds to a storage object.

[0061] The target data set task may refer to a task for processing a data set. In this embodiment, the target data set task may be used to indicate importing and storing a data set.

[0062] In a possible implementation, a queue is provided in the execution body of the present application, and the queue is used to store at least one data set task.

[0063] Exemplarily, the execution subject of the present application is provided with an interactive interface, which is used to receive a task instruction for processing a data set from a user, and store the data set task corresponding to the task instruction in a queue.

[0064] In a possible implementation, the queue follows a first-in-first-out rule, that is, the data set task that enters the queue first is processed first.

[0065] It can be understood that the target data set task can be obtained by obtaining the data set task from the tail of the queue as the target data set task.

[0066] For a data set, the data set may be composed of multiple storage objects. Correspondingly, the target data set task may include multiple subtasks, and each subtask corresponds to a storage object.

[0067] In a possible implementation, the data set may be text data in JSONL format, and each row of the data set may contain an independent JSON object, that is, a storage object.

[0068] For example, the data set can be represented as follows:

[0069] {"Name":"Zhang San","Age":25,"City":"Beijing"}

[0070] {"Name":"Li Si","Age":30,"City":"Shanghai"}

[0071] {"Name":"Wang Wu","Age":35,"City":"Guangzhou"};

[0072] Among them, each curly bracket {} and the content in the curly brackets occupy one line of the data set. Each line contains an independent JSON object. The format of the JSON object is a key-value pair, that is, the key is used to identify the name of the data, which can be called the field name, and the value is the data corresponding to the field name. Taking "Name":"Zhang San" as an example, Name is the field name, and Zhang San is the data corresponding to the Name field name.

[0073] In this example, there are three JSON objects, which can represent the basic information of the user, namely name, age, and city, which can be called keys. Each key corresponds to data, which can be called a value. For example, the value corresponding to the name is Zhang San, the value corresponding to the age is Li Si, and the value corresponding to the city is Beijing.

[0074] For one of the JSON objects, such as {"name":"Zhang San","age":25,"city":"Beijing"}, the storage object corresponds to a subtask, and the purpose of the subtask is to store the storage object.

[0075] Similarly, for other JSON objects, each JSON object corresponds to a subtask.

[0076] It should be noted that the content of the above data set is an example and is only used to understand the solution. In actual operation scenarios, the number of storage objects in the data set can be tens of thousands, and the content of the data set can be any other content. In order to comply with relevant laws, regulations and standards, there is no restriction on the number of storage objects in the data set and the content of the data set.

[0077] S102, parse and process each subtask in the target data set task to obtain source data and data records of the subtask; wherein the source data represents the storage object in the subtask, and the data record represents the attribute information of the source data.

[0078] Among them, the purpose of the subtask can be to store a storage object in the data set. Each subtask corresponds to a storage object. It can be understood that when each subtask realizes the storage of the storage object corresponding to the subtask, it can mark the completion of the target data set task.

[0079] For each subtask in the target data set task, the source data and data records of the subtask can be obtained by parsing the subtask.

[0080] The source data may represent the storage object in the subtask, and the data record may represent the attribute information of the source data.

[0081] It can be understood that the source data can be the actual data of the storage object, and the actual data can be used for modification, analysis and other processing. The data record is the attribute information of the source data, such as the memory size occupied by the source data, the modification time of the source data, etc.

[0082] In a possible implementation, the source data may be in a file format, such as a JSON file format.

[0083] In an optional implementation, one row in the target data set task corresponds to one subtask, and step S102 may include:

[0084] According to the row arrangement order in the target data set task, the target data set task is split into multiple subtasks; each subtask in the target data set task is parsed and processed, the data content of the subtask is determined as the source data of the subtask, and the attribute information of the source data is determined as the data record of the subtask; wherein the data content of the subtask represents the storage object.

[0085] In the target data set task, the data set may include multiple storage objects, each storage object corresponds to a subtask, and all subtasks may constitute the target data set task.

[0086] The row arrangement order may refer to the arrangement order of storage objects corresponding to the subtasks. It can be understood that one row of the data set in the target data set task is a storage object, and the arrangement order of the storage objects is the arrangement order corresponding to the subtasks.

[0087] For example, if the data set is as above:

[0088] {"Name":"Zhang San","Age":25,"City":"Beijing"}

[0089] {"Name":"Li Si","Age":30,"City":"Shanghai"}

[0090] {"Name":"Wang Wu","Age":35,"City":"Guangzhou"};

[0091] Using the value corresponding to the name as the identifier of each storage object and the identifier of the subtask, the subtasks corresponding to the row arrangement order may be: subtask Zhang San, subtask Li Si, subtask Wang Wu.

[0092] By splitting in sequence according to the row arrangement order, multiple subtasks can be obtained, so that each subtask can be parsed and processed, the data content of the subtask is determined as the source data of the subtask, and the attribute information of the source data is determined as the data record of the subtask; wherein the data content of the subtask represents the storage object.

[0093] It should be noted that, in addition to the row arrangement order, the target dataset task can also be split into multiple subtasks in other orders. The order in the target dataset task is not limited here, as long as the split subtasks have a certain logical order.

[0094] The beneficial effect of this setting is that by splitting the target data set task into multiple subtasks in a certain logical order, the splitting of the subtasks has a certain logical order and ensures the consistency of the logical order and the subtasks, avoiding data loss caused by confusion in the subtask order and improving the manageability of the subtasks.

[0095] In an optional implementation, step S102 may further include:

[0096] A preset row identifier is identified from the target data set task; wherein the preset row identifier is used to distinguish subtasks; and the target data set task is split according to the preset row identifier to obtain a plurality of subtasks.

[0097] The preset row identifier may refer to a pre-set row identifier, and the row identifier may be used to distinguish subtasks in the target data set task.

[0098] Exemplarily, the preset row identifier can be the} symbol in the curly brackets {}, or a line break character, which can be represented by \n. Of course, the preset row identifier can also be other symbols as long as the subtasks can be distinguished, and the form of the row identifier is not limited here.

[0099] Furthermore, according to the identified row identifiers, the target data set task can be split and processed to obtain multiple subtasks, each of which corresponds to a storage object.

[0100] The beneficial effect of this setting is that by setting row identifiers to distinguish subtasks in the target dataset task, the efficiency and accuracy of splitting subtasks are improved, making the splitting of dataset tasks inclusive and adaptable for datasets of different sizes.

[0101] In an optional implementation, step S102 may further include:

[0102] For each subtask, determine the data format of the subtask; if the data format of the subtask is a preset format, continue to parse and process each subtask in the target data set task, determine the data content of the subtask as the source data of the subtask, and determine the attribute information of the source data as the data record of the subtask.

[0103] The data format of the subtask may be the data format of the source data of the subtask.

[0104] It can be understood that the source data represents a storage object of the subtask, and the storage object can be a JSON object.

[0105] Exemplarily, the data format of the JSON object may satisfy the format of a key-value pair, and the data format of the JSON object may be exemplified as {"X":"A","Y":"B"}.

[0106] Among them, "X":"A" is a key-value pair, X is the key of the key-value pair, and A is the value of the key-value pair. If the key or value in the key-value pair is a string, you need to use quotation marks "" to encapsulate the string to ensure that the computer or other device can recognize the content of the key-value pair. At the same time, the left side of the colon : represents the key of the key-value pair, and the right side represents the value of the key-value pair. The colon can be used to characterize the correspondence between the key and the value. In addition, two adjacent key-value pairs can be separated by a comma, and the entire JSON object can be wrapped in a brace {}, that is, the content in the braces is the data content of the JSON object, that is, multiple key-value pairs.

[0107] The preset format can be the data format of the source data of the pre-set subtask. Exemplarily, the data format of the source data can be the key-value pair format of the above-mentioned JSON object, such as curly brackets {}, quotation marks "", colons :, commas, and other symbols, and there is a corresponding positional relationship between the text content of the JSON object and each symbol, for example, the left side of the colon: represents the key of the key-value pair, and the right side represents the value of the key-value pair, etc.

[0108] It should be noted that the preset format may also be expressed in other ways, and the preset format is not limited here.

[0109] If the data format of the subtask is the preset format, the steps of parsing each subtask in the target data set task, determining the data content of the subtask as the source data of the subtask, and determining the attribute information of the source data as the data record of the subtask can be continued.

[0110] If the data format of the subtask is not the preset format, the processing of the subsequent steps of the subtask can be suspended and a prompt message can be output, which can be used to prompt the user to check the data format of the subtask so that the user can manually correct the data format of the subtask.

[0111] The beneficial effect of such a setting is that before the source data of the subtask is stored in the preset first database, that is, in the early stage of the data processing method based on the data set provided in the present application, the data format of the subtask is timely identified and verified, which can avoid data damage caused by the source data of the subtask not being in the preset format, thereby saving computing resources for storing damaged data.

[0112] S103: Store the source data of the subtask in a preset first database, and obtain address information of the source data of the subtask in the preset first database.

[0113] The preset first database may refer to a pre-set database, and the first database may refer to a database or a database system for storing source data of the subtask.

[0114] Exemplarily, the source data of the subtask may be stored in the first database in the form of a file, and the file of the source data represents a storage object.

[0115] In a possible implementation, the first database may be an object storage system, such as a minio object storage system.

[0116] By storing the source data of the subtask in the preset first database, address information of the source data of the subtask in the preset first database can be obtained.

[0117] Exemplarily, if the source data of the subtask is the source data of the above-mentioned subtask Zhang San, the data content of the source data can be {"Name":"Zhang San","Age":25,"City":"Beijing"}, and the file format type of the source data is a JSON file type. For example, the file name of the source data can be zhangsa.json, and the address information of the source data in the preset first database can be data / zhangsa.json. The address information can represent the data content of the zhangsa.json file of the data source in the data folder. The data content of the JSON file under the address information can be obtained through the address information of the source data of the subtask in the preset first database.

[0118] S104: Store the address information of the source data of the subtask in the preset first database and the data record of the subtask in the preset second database.

[0119] The preset second database may also refer to a pre-set database, and the second database may be used to store address information of source data of subtasks in the preset first database and data records of subtasks, that is, the second database is different from the first database.

[0120] Exemplarily, the second database may store a target data record of a subtask, where the target data record of the subtask includes address information of source data of the subtask in the preset first database and the data record of the subtask.

[0121] In a possible implementation, the second database may be a relational database, such as a MySQL database.

[0122] Exemplarily, in a relational database, target data records of a plurality of subtasks are stored, and the unique identifier of each subtask may be address information of source data of the subtask in a preset first database.

[0123] The present application provides a data processing method based on a data set, which not only ensures the convenience of obtaining data records, but also improves the efficiency of storing the source data corresponding to the subtask, by storing the source data corresponding to the subtask in a first database and storing the data records corresponding to the subtask in a second database, so as to conveniently and quickly obtain the source data of the subtask from the second database according to the address information of the source data of the subtask in the preset first database.

[0124] Figure 2 This is a flow chart of another data processing method based on a data set provided by the present application. The execution subject of the method can be a server, a host or other device, such as Figure 2 As shown, the method may include:

[0125] S201, obtaining a target data set task; wherein the target data set task is used to instruct to import and store a data set, and the target data set task includes a plurality of subtasks, and each subtask corresponds to a storage object.

[0126] Exemplarily, this step may refer to the above-mentioned step S101 and will not be described in detail.

[0127] S202. Analyze and process multiple subtasks in the target data set task through multiple threads of a pre-set thread pool to obtain source data and data records of the subtasks; wherein each thread processes one subtask at a time.

[0128] Among them, the pre-set thread pool includes multiple threads. It can be understood that at the same time, each thread can process a subtask. The purpose of the subtask is to store the data source of the subtask in the preset first database and obtain the address information of the source data of the subtask in the preset first database; store the address information of the source data of the subtask in the preset first database and the data record of the subtask in the preset second database.

[0129] It can be understood that multiple threads in the thread pool can be processed in parallel, so that the thread pool can process multiple subtasks concurrently to improve the data processing efficiency of the data processing method based on the data set of the present application. For example, the data set includes 1 million storage objects. If the subtasks of the target data set task corresponding to the data set are processed one by one in a serial manner, it may take 1 to 2 hours. If a thread pool including multiple threads is used to process the subtasks of the target data set task in parallel, it may take less than 30 minutes.

[0130] In an optional implementation, step S202 may further include:

[0131] Determine the average speed of the thread pool; wherein the average speed represents the parsing speed of all threads in the thread pool; if the average speed exceeds a preset speed threshold, obtain the source data of the subtask from the thread pool, store the data source of the subtask in a preset first database, and obtain the address information of the source data of the subtask in the preset first database; store the address information of the source data of the subtask in the preset first database and the data record of the subtask in a preset second database.

[0132] Among them, the average speed of the thread pool may refer to the average parsing speed of all threads in the thread pool, and the parsing speed of the thread may refer to the speed at which the thread processes subtasks, which can be understood as the speed at which the thread parses and processes the subtask, obtains the source data and data records of the subtask, stores the data source of the subtask in a preset first database, obtains the address information of the source data of the subtask in the preset first database, stores the address information of the source data of the subtask in the preset first database and the data records of the subtask in the preset second database, and processes this series of steps.

[0133] The preset speed threshold may refer to a pre-set speed threshold. For example, the preset speed threshold may be 200 kilobytes per second (kb / s). For example, if the average speed of the thread pool is 300kb / s, the average speed exceeds the preset speed threshold. At this time, it can be considered that the load pressure in the thread pool is relatively small, and there are sufficient thread resources for processing subtasks.

[0134] For example, if the average speed of the thread pool is 100kb / s, the average speed does not exceed the preset speed threshold. At this time, it can be considered that the load pressure of the thread pool is relatively large, and there are fewer subtasks used to process thread resources. In this case, the processing of subtasks that have not been started can be suspended, or the number of threads in the thread pool can be reduced so that the average speed of the thread pool exceeds the preset speed threshold, and then the steps of obtaining the source data of the subtask from the thread pool, storing the data source of the subtask in the preset first database, and obtaining the address information of the source data of the subtask in the preset first database; storing the address information of the source data of the subtask in the preset first database and the data record of the subtask in the preset second database are performed.

[0135] In a possible implementation, a monitor is provided in the execution body of the present application, and the monitor is used to monitor parameter information in the thread pool in real time, and the parameter information may include the speed at which the thread pool processes subtasks, the number of threads in the thread pool, etc. Exemplarily, the average speed of the thread pool can be obtained by dividing the speed at which the thread pool processes subtasks by the number of threads in the thread pool.

[0136] The beneficial effect of such a setting is that, by dynamically adjusting whether to obtain the source data of the subtask from the thread pool according to the average speed of the thread pool, the data source of the subtask is stored in a preset first database, and the address information of the source data of the subtask in the preset first database is obtained; the address information of the source data of the subtask in the preset first database and the data record of the subtask are stored in the preset second database, thereby ensuring the efficiency of the thread pool in processing multiple subtasks while ensuring the procedurality of the thread pool in processing multiple subtasks, and avoiding the problem of throughput bottleneck of the thread pool.

[0137] S203: Store the source data of the subtask in a preset first database, and obtain address information of the source data of the subtask in the preset first database.

[0138] Exemplarily, this step may refer to the above-mentioned step S103 and will not be described in detail.

[0139] S204: Store the address information of the source data of the subtask in the preset first database and the data record of the subtask in the preset second database.

[0140] Exemplarily, this step may refer to the above-mentioned step S104 and will not be described in detail.

[0141] In an optional implementation, another data processing method based on a data set provided by the present application further includes:

[0142] If the address information of the source data of the subtask in the preset first database and the data record of the subtask are stored in the preset second database, the processing status of the subtask is determined to be imported successfully; wherein the processing status represents the import progress of the subtask.

[0143] Among them, the processing status of the subtask can refer to the processing status of the entire step of the thread corresponding to the subtask obtaining the source data of the subtask, storing the data source of the subtask in a preset first database, and obtaining the address information of the source data of the subtask in the preset first database; storing the address information of the source data of the subtask in the preset first database and the data record of the subtask in the preset second database.

[0144] Exemplarily, the processing status of the subtask may represent the import progress of the subtask, and the status of the subtask may include states such as importing, import successful, and import failed.

[0145] If it is determined that the address information of the source data of the subtask in the preset first database and the data record of the subtask are stored in the preset second database, then the processing status of the subtask can be determined as successful import;

[0146] If the source data and data records of the subtask are obtained, it can be determined that the processing status of the subtask is importing;

[0147] If after obtaining the source data and data records of the subtask, it exceeds a certain time threshold, for example 2 minutes, and the address information of the source data of the subtask in the preset first database and the data records of the subtask cannot be determined to be stored in the preset second database, then the processing status of the subtask can be determined as import failure.

[0148] In a possible implementation, a display interface is provided in the execution body of the present application, and the display interface is used to display the processing status of the subtask processed by each thread in the thread pool.

[0149] The beneficial effect of such a setting is that the processing status of the subtask processed by each thread in the thread pool can be intuitively understood, thereby facilitating the user to judge and analyze the processing capacity of the thread pool.

[0150] Another data processing method based on a data set provided by the present application provides setting up a thread pool, and multiple threads in the thread pool can be processed in parallel, so that the thread pool can process multiple subtasks concurrently, so as to improve the data processing efficiency of the data processing method based on a data set of the present application. At the same time, by dynamically adjusting the thread resources used for processing subtasks in the thread pool according to the average speed of the thread pool, the throughput bottleneck problem of the thread pool is avoided.

[0151] Figure 3 A flowchart of another data processing method based on a data set provided by the present application is shown below. The execution subject of the method may be a server, a host or other device, such as Figure 3 As shown, the method may include:

[0152] S301, obtaining a target data set task; wherein the target data set task is used to instruct to import and store a data set, and the target data set task includes multiple subtasks, and each subtask corresponds to a storage object.

[0153] Exemplarily, this step may refer to the above-mentioned step S101 and will not be described in detail.

[0154] S302, parse and process each subtask in the target data set task to obtain source data and data records of the subtask; wherein the source data represents the storage object in the subtask, and the data record represents the attribute information of the source data.

[0155] Exemplarily, this step may refer to the above-mentioned step S102 and will not be described in detail.

[0156] S303: Store the source data of the subtask in a preset first database, and obtain address information of the source data of the subtask in the preset first database.

[0157] Exemplarily, this step may refer to the above-mentioned step S103 and will not be described in detail.

[0158] S304: Store the address information of the source data of the subtask in the preset first database and the data record of the subtask in the preset second database.

[0159] Exemplarily, this step may refer to the above-mentioned step S104 and will not be described in detail.

[0160] S305 . In response to the export instruction for the subtask, obtain address information of the source data of the subtask in the preset first database from the preset second database.

[0161] The export instruction of the subtask may represent that the user exports, annotates or views the storage object corresponding to the subtask.

[0162] In a possible implementation, the interactive interface set by the execution subject of the present application can also be used to receive an export instruction for a subtask from a user, and generate a corresponding data set task from the export instruction. The types of data set tasks can include export task types, annotation task types, and viewing task types. Furthermore, the data set task corresponding to the export instruction is stored in a queue.

[0163] In a possible implementation, the queue follows a first-in-first-out rule, that is, the data set task that enters the queue first is processed first.

[0164] It can be understood that the method of responding to the export instruction for the subtask may be to obtain the data set task from the tail of the queue, where the data set task represents the export instruction for the subtask.

[0165] In a possible implementation, the export instruction for the subtask may be understood as an export instruction for the storage object corresponding to at least one subtask.

[0166] In response to the export instruction for the subtask, address information of the source data of the subtask in the preset first database may be obtained from the preset second database.

[0167] It can be understood that the preset second database can be a relational database, and the relational database has a paging query function. For example, the target data records of all subtasks are queried, and the target data records of the subtasks include the address information of the source data of the subtasks in the preset first database and the data records of the subtasks. For example, the target data records of all subtasks are 1 million. If a relational database is not used, it may take 30 seconds to load the target data records of 1 million subtasks. If a relational database is used, the paging function of the relational database can enable each page to display the target data records of some subtasks in parallel. The target data records of the subtasks displayed on all pages are added together, which is the target data records of 1 million subtasks. It may only take 2 seconds to load the target data records of 1 million subtasks.

[0168] It can be seen that by storing the target data records of the subtasks separately in the preset second database, the target data records of all subtasks can be quickly and efficiently acquired, thereby enhancing the stability and efficiency of data processing based on the data set.

[0169] By loading the target data record of the subtask completed by the relational database, the address information of the source data of the subtask in the preset first database can be obtained from the preset second database, so that further operations can be performed on the source data of the subtask.

[0170] S306: Acquire the source data of the subtask from the preset first database according to the address information of the source data of the subtask in the preset first database.

[0171] Wherein, by acquiring the address information of the source data of the subtask in the preset first database from the preset second database, the source data of the subtask can be acquired from the preset first database.

[0172] In a possible implementation, the source data of the subtask may be acquired from a preset first database through a plurality of threads of a preset thread pool.

[0173] Each subtask corresponds to a storage object, and each thread processes a subtask at a time.

[0174] The processing logic of the multiple threads of the preset thread pool is consistent with the processing logic of the multiple threads of the preset thread pool in the above step S202 for parsing and processing the multiple subtasks in the target data set task, and will not be repeated here.

[0175] If the type of the data set task corresponding to the export instruction for the subtask is an export task type, the acquired source data of the subtask may be exported to a destination storage address.

[0176] If the type of the data set task corresponding to the above-mentioned export instruction for the subtask is a labeling task type, the acquired source data of the subtask can be labeled, and then the data content of the source data of the subtask in the preset first database can be synchronously updated.

[0177] If the type of the data set task corresponding to the export instruction for the subtask is a viewing task type, the acquired source data of the subtask can be displayed. In a possible implementation, the preset first database can also achieve the effect of displaying the source data of the subtask.

[0178] It can be understood that the address information of the source data of the subtask in the preset second database in the preset first database and the source data of the subtask in the preset first database are in a one-to-one correspondence, that is, for a subtask, the source data corresponding to the subtask is stored in the preset first database, and the data records and address information corresponding to the subtask are stored in the preset second database. As long as the address information of the source data of the subtask in the preset first database is used, the source data of the subtask can be obtained from the preset first database, and subsequent operations can be performed on the source data of the subtask.

[0179] Another data processing method based on a data set provided by the present application can, after the data set is imported and stored, export, annotate, view and other operations on the source data of the subtask corresponding to the data set, thereby realizing the scalability of the data processing method based on the data set and improving the processing efficiency of the data processing based on the data set.

[0180] Figure 4 A schematic diagram of the architecture of a data processing method based on a data set provided in this application, such as Figure 4 As shown, the architecture of the data processing method based on the data set may include a task support layer, a file processing layer, and a storage layer.

[0181] Among them, the task support layer can be used to receive instructions from users and generate data set tasks; the types of data set tasks can include import storage task type, export task type, labeling task type, and viewing task type.

[0182] The dataset task of the import storage task type can represent the import and storage of the dataset, the dataset task of the export task type can represent the export processing of exporting the source data of the subtask obtained to the destination storage address, the dataset task of the annotation task type can represent the annotation processing of the source data of the subtask obtained, and the viewing task type can represent the display of the source data of the subtask obtained.

[0183] The file processing layer can represent the processing of dataset tasks according to different types of dataset tasks.

[0184] Among them, the file processing layer may include a task queue, a subtask splitter, a file verifier, a concurrent thread pool, and a monitor.

[0185] The task queue is used to store the dataset tasks from the task support layer, so that the file processing layer can obtain the dataset tasks from the task queue and process the dataset tasks.

[0186] The subtask splitter can be used to parse and process each subtask in the target data set task to obtain the source data and data records of the subtask; wherein the source data represents the storage object in the subtask, and the data record represents the attribute information of the source data.

[0187] The file verifier can be used to determine the data format of each subtask and judge whether the data format of the subtask is a preset format. If yes, continue to execute the next step; if not, skip the subtask.

[0188] The concurrent thread pool can be used to obtain the source data of the subtask, store the data source of the subtask in a preset first database, and obtain the address information of the source data of the subtask in the preset first database; store the address information of the source data of the subtask in the preset first database and the data record of the subtask in the preset second database; wherein the concurrent thread pool includes multiple threads, and each thread processes one subtask at a time.

[0189] The monitor can be used to monitor parameter information in the thread pool in real time. The parameter information may include the speed at which the thread pool processes subtasks, the number of threads in the thread pool, etc.

[0190] The storage layer includes a preset first database and a preset second database, wherein the preset first database may be a relational database, and the preset second database may be an object storage system.

[0191] The address information of the source data of the subtask in the preset second database in the preset first database is in a one-to-one correspondence with the source data of the subtask in the preset first database.

[0192] The architecture of a data processing method based on a data set provided in the present application is divided into a task support layer, a file processing layer, and a storage layer, so that each layer has a clear division of labor, thereby improving the efficiency of data processing based on the data set.

[0193] Figure 5 A schematic diagram of a data processing device based on a data set provided in this application, such as Figure 5As shown, the data processing device 50 based on the data set includes: an acquisition unit 501, a parsing unit 502, a first storage unit 503, and a second storage unit 504.

[0194] The acquisition unit 501 is used to acquire a target data set task; wherein the target data set task is used to instruct to import and store a data set, and the target data set task includes multiple subtasks, each of which corresponds to a storage object;

[0195] The parsing unit 502 is used to parse each subtask in the target data set task to obtain the source data and data records of the subtask; wherein the source data represents the storage object in the subtask, and the data record represents the attribute information of the source data;

[0196] The first storage unit 503 is used to store the source data of the subtask in a preset first database, and obtain the address information of the source data of the subtask in the preset first database;

[0197] The second storage unit 504 is used to store the address information of the source data of the subtask in the preset first database and the data record of the subtask in the preset second database.

[0198] Figure 6 A structural diagram of another data processing device based on a data set provided in this application, such as Figure 6 As shown, the data processing device 60 based on the data set includes: an acquisition unit 601 , a parsing unit 602 , a first storage unit 603 , and a second storage unit 604 , wherein the parsing unit 602 also includes a first parsing module 6021 and a second parsing module 6022 .

[0199] In an optional example, a row in the target dataset task corresponds to one subtask;

[0200] A first parsing module 6021 is used to split the target data set task into multiple subtasks according to the row arrangement order in the target data set task;

[0201] The second parsing module 6022 is used to parse each subtask in the target data set task, determine the data content of the subtask as the source data of the subtask, and determine the attribute information of the source data as the data record of the subtask; wherein the data content of the subtask represents the storage object.

[0202] In an optional example, the parsing unit 602 is further configured to identify a preset row identifier from the target data set task; wherein the preset row identifier is used to distinguish subtasks;

[0203] According to the preset row identifier, the target data set task is split and processed to obtain multiple subtasks.

[0204] In an optional example, the parsing unit 602 is further configured to determine the data format of each subtask;

[0205] If the data format of the subtask is the preset format, continue to parse each subtask in the target data set task, determine the data content of the subtask as the source data of the subtask, and determine the attribute information of the source data as the data record of the subtask.

[0206] In an optional example, a thread pool is pre-set, and the thread pool includes multiple threads;

[0207] The parsing unit 602 is further used to parse and process multiple subtasks in the target data set task through multiple threads to obtain source data and data records of the subtasks; wherein each thread processes one subtask at a time.

[0208] In an optional example, the parsing unit 602 further includes a third parsing module 603, and the third parsing module 603 is used to determine an average speed of the thread pool; wherein the average speed represents the parsing speed of all threads in the thread pool;

[0209] If the average speed exceeds a preset speed threshold, the source data of the subtask is obtained from the thread pool, the data source of the subtask is stored in a preset first database, and the address information of the source data of the subtask in the preset first database is obtained;

[0210] The address information of the source data of the subtask in the preset first database and the data record of the subtask are stored in the preset second database.

[0211] In an optional example, a status processing unit 605 is also included, which is used to determine that the processing status of the subtask is successfully imported if the address information of the source data of the subtask is determined in the preset first database and the data record of the subtask is stored in the preset second database; wherein the processing status represents the import progress of the subtask.

[0212] In an optional example, it further includes an instruction processing unit 606, which is used to obtain address information of the source data of the subtask in the preset first database from the preset second database in response to the export instruction of the subtask;

[0213] The source data of the subtask is acquired from the preset first database according to the address information of the source data of the subtask in the preset first database.

[0214] Figure 7 A schematic diagram of the structure of the electronic device provided in this application, such as Figure 7As shown, the electronic device 70 provided in this embodiment includes: at least one processor 701 and a memory 702. Optionally, the device 70 further includes a communication component 703. The processor 701, the memory 702 and the communication component 703 are connected via a bus 704.

[0215] In a specific implementation process, at least one processor 701 executes the computer-executable instructions stored in the memory 702, so that at least one processor 701 executes the above method.

[0216] The specific implementation process of the processor 701 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0217] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0218] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk storage.

[0219] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0220] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0221] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0222] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0223] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0224] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0225] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0226] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. If not specifically stated, the processor may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. If not specifically stated, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc.

[0227] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0228] In the above embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0229] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0230] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A data processing method based on a data set, characterized in that: include: Obtain a target data set task; wherein the target data set task is used to instruct to import and store the data set, and the target data set task includes multiple subtasks, each subtask corresponding to a storage object; Parsing each subtask in the target data set task to obtain source data and data records of the subtask; wherein the source data represents the storage object in the subtask, and the data record represents the attribute information of the source data; Storing the source data of the subtask in a preset first database, and obtaining address information of the source data of the subtask in the preset first database; The address information of the source data of the subtask in the preset first database and the data record of the subtask are stored in a preset second database.

2. The method according to claim 1, characterized in that A row in the target data set task corresponds to a subtask; Each subtask in the target data set task is parsed to obtain source data and data records of the subtask, including: Splitting the target data set task into a plurality of subtasks according to the row arrangement order in the target data set task; Each subtask in the target data set task is parsed and processed, the data content of the subtask is determined as the source data of the subtask, and the attribute information of the source data is determined as the data record of the subtask; wherein the data content of the subtask represents a storage object.

3. The method according to claim 2, characterized in that Also includes: Identify a preset row identifier from the target data set task; wherein the preset row identifier is used to distinguish subtasks; According to the preset row identifier, the target data set task is split and processed to obtain a plurality of subtasks.

4. The method according to claim 2, characterized in that: Also includes: For each subtask, determining a data format of the subtask; If the data format of the subtask is a preset format, continue to execute the parsing process for each subtask in the target data set task, determine the data content of the subtask as the source data of the subtask, and determine the attribute information of the source data as the data record of the subtask.

5. The method according to claim 1, characterized in that A thread pool is pre-set, and the thread pool includes multiple threads; each subtask in the target data set task is parsed and processed to obtain source data and data records of the subtask, including: Through multiple threads, multiple subtasks in the target data set task are parsed and processed to obtain source data and data records of the subtasks; wherein each thread processes one subtask at a time.

6. The method according to claim 5, characterized in that Also includes: Determine an average speed of the thread pool; wherein the average speed represents the parsing speed of all threads in the thread pool; If the average speed exceeds a preset speed threshold, the source data of the subtask is obtained from the thread pool, the data source of the subtask is stored in a preset first database, and the address information of the source data of the subtask in the preset first database is obtained; The address information of the source data of the subtask in the preset first database and the data record of the subtask are stored in a preset second database.

7. The method according to claim 1, characterized in that Also includes: If it is determined that the address information of the source data of the subtask in the preset first database and the data record of the subtask are stored in the preset second database, then the processing status of the subtask is determined to be imported successfully; wherein the processing status represents the import progress of the subtask.

8. The method according to any one of claims 1 to 7, characterized in that Also includes: In response to the export instruction for the subtask, acquiring address information of the source data of the subtask in the preset first database from the preset second database; The source data of the subtask is acquired from the preset first database according to the address information of the source data of the subtask in the preset first database.

9. A data processing device based on a data set, characterized in that: include: An acquisition unit, used for acquiring a target data set task; wherein the target data set task is used for instructing to import and store a data set, and the target data set task includes a plurality of subtasks, each of which corresponds to a storage object; A parsing unit, used to parse each subtask in the target data set task to obtain source data and data records of the subtask; wherein the source data represents a storage object in the subtask, and the data record represents attribute information of the source data; A first storage unit, used to store the source data of the subtask in a preset first database, and obtain address information of the source data of the subtask in the preset first database; The second storage unit is used to store the address information of the source data of the subtask in the preset first database and the data record of the subtask in the preset second database.

10. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 8 when executed by a processor.

12. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 8 when being executed by a processor.

Images