Data processing method, device and equipment

Abstract

The present invention provides a data processing method, device and equipment, which can be applied to the field of artificial intelligence technology. The method includes: in response to a backup task having been received, the backup task is divided to obtain multiple backup subtasks, wherein the backup subtasks include at least one table acquisition task and at least one table's respective acquisition path, and the multiple backup subtasks are parallel processing tasks; for each backup subtask, based on at least one table's respective acquisition path, multiple initial metadata of at least one table are acquired from a database to be backed up; index information corresponding to each of the multiple initial metadata is spliced ​​with the multiple initial metadata to determine multiple target metadata; based on the data volume of multiple data buffers and the multiple target metadata, multiple target files with a preset data volume are generated to store data; and multiple target files are uploaded to the backup space in parallel.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a data processing method, device and equipment. Background Art

[0002] Backup technology is an important measure in the field of computer systems to prevent data loss and damage caused by emergencies. Therefore, it is very important for enterprises to back up data. However, with the advent of the big data era, the amount of data in each database in the enterprise is increasing. This requires that a large amount of data be backed up to the backup database when performing data backup. In related technologies, all data is usually backed up by directly transferring a large amount of data to the backup database.

[0003] In the process of implementing the concept of the present invention, it is found that there are at least the following problems in the related art: the method of directly transmitting a large amount of backup data to the backup database in the related art has the problem of low backup efficiency. Summary of the invention

[0004] In view of the above problems, the present invention provides a data processing method, apparatus, device, medium and program product.

[0005] According to one aspect of the present invention, a data processing method is provided, comprising: in response to having received a backup task, dividing the backup task into a plurality of backup subtasks, wherein the backup subtasks include at least one table acquisition task and at least one table respective acquisition path, and the plurality of backup subtasks are parallel processing tasks; for each backup subtask, based on at least one table respective acquisition path, acquiring a plurality of initial metadata of at least one table from a database to be backed up; splicing index information corresponding to each of the plurality of initial metadata with the plurality of initial metadata to determine a plurality of target metadata; generating a plurality of target files having a storage data volume of a preset data volume based on a plurality of data buffers and a plurality of target metadata respective data volumes; and uploading the plurality of target files to a backup space in parallel.

[0006] According to an embodiment of the present invention, based on the data volumes of multiple data buffers and multiple target metadata, multiple target files with a preset data volume are generated, including: inputting multiple target metadata into multiple data buffers in parallel; for each data buffer, when it is determined that the storage data volume of the target metadata in the data buffer is equal to the preset data volume, storing the target metadata in the data buffer as a target file by column; and obtaining multiple target files with a storage data volume of the preset data volume based on the target files generated by each of the multiple data buffers.

[0007] According to an embodiment of the present invention, the data processing method further includes: determining a compression ratio of the target file; and multiplying the data volume of the target metadata in the data buffer by the compression ratio to obtain the storage data volume.

[0008] According to an embodiment of the present invention, the index information includes a table identifier of a table to which the initial metadata belongs, data type information to which the initial metadata belongs, and a data group identifier to which the initial metadata belongs. The method further includes: for each initial metadata, based on an acquisition path corresponding to the initial metadata, determining the table identifier of the initial metadata; based on the location information of the initial metadata, determining the data type information and the data group identifier corresponding to the initial metadata in a location and index information correspondence table, wherein the location information represents the location of the initial metadata in a table for storing the initial metadata, and the location and index information correspondence table includes a correspondence between the location information and the data type information of the initial metadata and a correspondence between the location information and the data group identifier; generating index information based on the data type information, the data group identifier, and the table identifier of the initial metadata.

[0009] According to an embodiment of the present invention, the above-mentioned data processing method also includes: determining the correspondence between each target file and the table identifier based on the index information of the target metadata in each target file; generating an index file based on the correspondence between each target file and the table identifier; and uploading the index file to the backup space.

[0010] According to an embodiment of the present invention, the above-mentioned data processing method also includes: in response to receiving a data recovery task, dividing the data recovery task to obtain multiple data recovery subtasks, wherein the data recovery subtask includes a recovery task of at least one table, the multiple data recovery subtasks are parallel processing tasks, and the data recovery subtask includes a table identifier of at least one table; for each data recovery subtask, obtaining multiple target metadata carrying table identifiers from multiple target files included in the backup space; splitting the multiple target metadata to obtain multiple initial metadata and multiple index information; based on data writing tools corresponding to each of the multiple index information, writing the multiple initial metadata into the database to be recovered.

[0011] According to an embodiment of the present invention, multiple target metadata carrying a table identifier are obtained from multiple target files included in a backup space, including: obtaining an index file from the backup space, and determining a to-be-processed file from the multiple target files based on the index file and the table identifier, wherein the index file includes a correspondence between the target file and the table identifier, and the to-be-processed file stores multiple target metadata carrying the table identifier; based on the table identifier, obtaining multiple target metadata carrying the table identifier of the index information from the to-be-processed file.

[0012] According to an embodiment of the present invention, based on data writing tools corresponding to each of the multiple index information, multiple initial metadata are written into the database to be recovered, including: based on the data group identifier and data type information carried by each of the multiple target metadata, the initial metadata corresponding to each of the multiple target metadata are grouped to obtain multiple metadata groups; based on the data type information corresponding to each of the multiple metadata groups, the data writing tools corresponding to each of the multiple metadata groups are determined; based on the data writing tools corresponding to each of the multiple metadata groups, the multiple metadata groups are respectively written into the database to be recovered.

[0013] Another aspect of the present invention provides a data processing device, characterized in that the device includes: a task splitting module, which is used to split the backup task in response to having received a backup task, so as to obtain multiple backup subtasks, wherein the backup subtasks include at least one table acquisition task and at least one table's respective acquisition path, and the multiple backup subtasks are parallel processing tasks; a data acquisition module, which is used to obtain, for each backup subtask, multiple initial metadata of at least one table from the database to be backed up based on the respective acquisition path of at least one table; a splicing module, which is used to splice the index information corresponding to each of the multiple initial metadata with the multiple initial metadata to determine multiple target metadata; a file generation module, which is used to generate multiple target files with a preset data volume for storage based on the data volume of multiple data buffers and the multiple target metadata; and a file upload module, which is used to upload the multiple target files to the backup space in parallel.

[0014] Another aspect of the present invention provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0015] Another aspect of the present invention further provides a computer-readable storage medium having a computer program or instructions stored thereon, wherein the computer program or instructions implement the steps of the above method when executed by a processor.

[0016] Another aspect of the present invention further provides a computer program product, including a computer program or instructions, which implement the steps of the above method when executed by a processor.

[0017] According to the data processing method of the present invention, by dividing the data backup task into backup subtasks and executing each backup subtask in parallel, the data backup rate is improved in the first aspect, and for each backup subtask, multiple target files with a preset data volume are generated in parallel based on the data volume of multiple data buffers and multiple target metadata, thereby improving the data backup rate in the second aspect, and because the sizes of the target files are consistent with the preset data volume, the upload delay during uploading can be reduced to a certain extent when multiple target files are uploaded in parallel, and because the parallel process of the above three aspects is adopted, the problem of low backup efficiency is at least partially solved, and the beneficial effect of improving backup efficiency is achieved, and because the index information of each of the multiple initial metadata is spliced ​​with the initial metadata to generate the target metadata, faster data recovery can be achieved through the index information during subsequent data recovery. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0019] Figure 1 The application scenario diagram of the data processing method, apparatus, device, medium and program product according to the embodiments of the present invention is schematically shown.

[0020] Figure 2 The flowchart of the data processing method according to the embodiment of the present invention is schematically shown.

[0021] Figure 3 A schematic diagram of splicing index information and initial metadata according to an embodiment of the present invention is schematically shown.

[0022] Figure 4 The structure block diagram of the data processing device according to the embodiment of the present invention is schematically shown.

[0023] Figure 5 A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0024] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0025] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0026] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0027] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0028] It should be noted that the data processing method and device of the present invention can be used in the field of artificial intelligence technology, and can also be used in any field other than the field of artificial intelligence technology, such as the field of computer technology, and the application field of the data processing method and device of the present invention is not limited.

[0029] During the research process, it was found that in order to ensure data security, mainstream databases will provide relevant off-site disaster recovery backup solutions to prevent data loss caused by regional disasters. Generally, backup data is stored in different data centers or cloud service providers in different geographical locations to effectively reduce the risks caused by regional disasters and ensure data security and reliability. Traditional disaster recovery backup generally chooses to save the metadata information of the database (such as table definitions, indexes, triggers, etc.) in the form of SQL (Structured Query Language) in related files, and executes SQL commands when recovery is required. In this way, for situations where the metadata information is large, the storage of SQL commands will occupy more resources, and if the SQL command fails, it may involve metadata errors, and the metadata can only be obtained after the SQL command is executed, which has the problem of high computing pressure.

[0030] Another solution is to directly export all metadata information in the database to be backed up, and then write all the exported metadata information back to the service when restoring. Under this solution, the metadata information will have the problem of low backup and recovery efficiency.

[0031] In view of this, an embodiment of the present invention provides a data processing method, including: in response to having received a backup task, splitting the backup task to obtain multiple backup subtasks; for each backup subtask, based on the respective acquisition path of at least one table, obtaining multiple initial metadata of at least one table from the database to be backed up; splicing the index information corresponding to each of the multiple initial metadata with the multiple initial metadata to determine multiple target metadata; based on the data volume of multiple data buffers and the multiple target metadata, generating multiple target files with a preset data volume for storage, wherein the preset data volume is related to the network bandwidth used to transmit the target files and the hardware resources of the hardware used to generate the target files; uploading the multiple target files to the backup space in parallel.

[0032] Figure 1 The application scenario diagram of the data processing method, apparatus, device, medium and program product according to the embodiments of the present invention is schematically shown.

[0033] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a terminal device 101, a terminal device 102, a terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the terminal devices 101, 102, 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0034] The user can use the terminal device 101, the terminal device 102, and the terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, the terminal device 102, and the terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0035] Terminal device 101, terminal device 102, and terminal device 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0036] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0037] It should be noted that the data processing method provided in the embodiment of the present invention can generally be executed by the server 105. Accordingly, the data processing device provided in the embodiment of the present invention can generally be set in the server 105. The data processing method provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the data processing device provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0038] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0039] The following will be based on Figure 1 The scene described by Figure 2~Figure 3 The data processing method of the embodiment of the invention is described in detail.

[0040] Figure 2 The flowchart of the data processing method according to the embodiment of the present invention is schematically shown.

[0041] like Figure 2 As shown, the method 200 includes operations S210 to S250.

[0042] In operation S210, in response to receiving a backup task, the backup task is split into multiple backup subtasks, wherein the backup subtasks include at least one table acquisition task and at least one table respective acquisition path, and the multiple backup subtasks are parallel processing tasks.

[0043] In operation S220 , for each backup subtask, a plurality of initial metadata of at least one table is acquired from the database to be backed up based on the acquisition path of at least one table.

[0044] In operation S230, the index information corresponding to each of the plurality of initial metadata is concatenated with the plurality of initial metadata to determine a plurality of target metadata.

[0045] In operation S240, a plurality of target files storing a preset data amount are generated based on the respective data amounts of the plurality of data buffers and the plurality of target metadata.

[0046] In operation S250, a plurality of target files are uploaded to the backup space in parallel.

[0047] According to an embodiment of the present invention, the backup task includes a task of backing up all data in the database to be backed up to the backup space, for example, backing up data of all tables in the database to be backed up to the backup space.

[0048] According to the embodiment of the present invention, there is no limitation on the backup space, and the backup space may be a cloud storage server.

[0049] According to an embodiment of the present invention, the database to be backed up may be a metadata database, and the database to be backed up stores metadata of other databases, that is, the initial metadata may be attribute data such as table definitions and indexes stored in other databases.

[0050] According to an embodiment of the present invention, a backup subtask is a task obtained by splitting a backup task. The splitting method is not limited and may be equal splitting, or may be splitting according to the computing power of the task node used to execute the backup subtask, such as allocating a backup subtask with a larger task volume to a task node with strong computing power.

[0051] According to an embodiment of the present invention, the initial metadata may be key-value pair data, and each initial metadata is a key-value pair.

[0052] According to an embodiment of the present invention, after the backup subtask is divided, the backup subtask can be sent to the corresponding task node for execution, each backup subtask corresponds to a task node, and the task node can be a server for performing data backup or recovery.

[0053] According to an embodiment of the present invention, the task node can continue to divide the backup subtasks according to the dimension of the table, so as to obtain the table acquisition task of at least one table, and put the table acquisition task of each table into the task queue, so that the task node obtains the table acquisition task from the task queue and performs multiple initial metadata acquisitions for each table, and stores the initial metadata of a table as an initial metadata set. After completing the table acquisition task of at least one table, multiple initial metadata of at least one table are obtained.

[0054] According to an embodiment of the present invention, obtaining the path may include data representing the path such as the name of the database to be backed up, the storage folder of the table, and the table identifier.

[0055] According to an embodiment of the present invention, the target metadata may include initial metadata and index information.

[0056] According to the embodiment of the present invention, the data buffer is not limited and may be a container for temporarily storing data during metadata transmission or processing, so that data can be read more quickly when a target file is generated.

[0057] According to an embodiment of the present invention, there is no limitation on the file format of the target file, which may be an optimized row columnar storage (ORC) or other file formats, such as Parquet and other formats.

[0058] According to an embodiment of the present invention, when the file format of the target file is ORC, the data buffer may be an ORC buffer, namely, an ORC Buffer.

[0059] According to an embodiment of the present invention, the preset data volume is not limited. In one embodiment, it may be a preset volume determined based on experience. In another embodiment, it may be related to the network bandwidth used to transmit the target file or the hardware resources of the hardware used to generate the target file or the data volume of multiple target metadata as a whole and the historical preset data volume in the historical backup process. For example, the preset data volume is determined based on the historical preset data volume of the target file under different network bandwidths or different hardware resources or different target metadata data volumes during the historical backup process.

[0060] According to an embodiment of the present invention, in another embodiment, it may also be related to the network bandwidth used to transmit the target file, the hardware resources of the hardware used to generate the target file, and the historical preset data volume in the historical backup process, such as: the preset data volume is determined based on the historical network bandwidth, historical hardware resources, current network bandwidth, current hardware resources, and the historical preset data volume of the target file in the historical backup process.

[0061] According to the embodiment of the present invention, after the preset data volume is obtained, it can be used directly in the subsequent process without the need to recalculate each time.

[0062] According to an embodiment of the present invention, in other embodiments, the preset data volume of the target file may also be determined according to the following formulas (1) and (2) by using an optimization algorithm or by exhaustive enumeration within a certain range.

[0063] ; (1)

[0064] ; (2)

[0065] Among them, n is the generation parallelism when generating target files in parallel, The rate factor generated for each target file, m is the upload parallelism of the target file being uploaded in parallel, The network bandwidth for uploading the target file. is an adjustment coefficient, which is related to the hardware resources of the hardware used to generate the target file, p is the preset data volume of the target file, and w is the repetition of the data type of multiple target metadata.

[0066] According to an embodiment of the present invention, the generation parallelism can be characterized by the number of target files generated in parallel, specifically, it can be characterized by the number of data buffers, such as: if the number of data buffers is 3, then the number of target files generated at one time is 3, that is, the generation parallelism is 3. The upload parallelism can be characterized by the number of threads that can be called in the process of transferring the target file to the backup space, for example: one target file is uploaded by one thread, if three target files need to be uploaded, then three threads are required.

[0067] According to an embodiment of the present invention, when determining the preset data volume of the target file by using an optimization algorithm or an exhaustive method within a certain range according to formulas (1) and (2), the generation parallelism and the upload parallelism can also be determined at the same time, or the generation parallelism and the upload parallelism can be fixed according to actual conditions so as to only determine the preset data volume of the target file.

[0068] According to an embodiment of the present invention, the adjustment coefficient may be obtained by evaluating the input and output performance of a central processing unit (CPU) or a disk.

[0069] According to the embodiment of the present invention, there is no limitation on the optimization algorithm, and it may be a genetic algorithm (GA), Bayesian optimization (Bayesian Optimization), etc.

[0070] According to an embodiment of the present invention, if the upload parallelism has been determined, multiple target files can be uploaded in parallel using the same number of threads as the upload parallelism when the target files are uploaded.

[0071] According to an embodiment of the present invention, if the upload parallelism is not determined, multiple target files may be uploaded in parallel using the same number of threads as the number of target files.

[0072] According to an embodiment of the present invention, the preset data amount of the target file can also be subdivided into the preset initial data amount a of the target file multiplied by the compression ratio b of the target file. When the compression ratio of the target file has been determined, the preset data amount is obtained by determining the preset initial data amount a.

[0073] According to an embodiment of the present invention, the generation parallelism multiplied by the rate factor can represent the generation rate of the target file, and the upload parallelism multiplied by the network bandwidth can represent the upload rate of the target file. Since the higher the data type repetition of multiple target metadata is, the faster the generation rate of the target file is, the repetition of the data type of the target metadata is taken into account in the rate factor.

[0074] According to an embodiment of the present invention, since the network bandwidth used to transmit the target file is related to the upload rate of the target file, and the hardware resources of the hardware used to generate the target file are related to the generation rate of the target file, when the upload rate of the target file and the generation rate of the target file are approximately equal to 1, the time of target file generation and target file upload can be folded to a certain extent, so that multiple target files can be uploaded while the previous group of generated multiple target files is being generated, thereby speeding up the data backup of the database to be backed up.

[0075] According to an embodiment of the present invention, the index information and the initial metadata may be concatenated into one piece of data, thereby obtaining the target metadata.

[0076] According to the embodiments of the present invention, since the sizes of the target files are all the same, the tasks can usually be completed in the same period when the target files are transmitted in parallel, thereby avoiding the problem of long transmission time for larger target files and short transmission time for smaller target files due to different sizes of the target files, but the data backup time must be based on the longer transmission time, resulting in a long overall data backup time, and since the thread used to transmit the smaller target file completes the upload task faster, while the thread used to transmit the larger target file takes a longer time to complete the upload task, there is an uneven use of thread resources and a waste of resources.

[0077] According to the data processing method of the present invention, by dividing the data backup task into backup subtasks and executing each backup subtask in parallel, the data backup rate is improved in the first aspect, and for each backup subtask, multiple target files with a preset data volume are generated in parallel based on the data volume of multiple data buffers and multiple target metadata, thereby improving the data backup rate in the second aspect, and because the sizes of the target files are consistent with the preset data volume, the upload delay during uploading can be reduced to a certain extent when multiple target files are uploaded in parallel, and because the parallel process of the above three aspects is adopted, the problem of low backup efficiency is at least partially solved, and the beneficial effect of improving backup efficiency is achieved, and because the index information of each of the multiple initial metadata is spliced ​​with the initial metadata to generate the target metadata, faster data recovery can be achieved through the index information during subsequent data recovery.

[0078] According to an embodiment of the present invention, generating a plurality of target files having a preset data volume based on the data volume of a plurality of data buffers and a plurality of target metadata may include the following operations.

[0079] Input multiple target metadata into multiple data buffers in parallel; for each data buffer, when it is determined that the storage data amount of the target metadata in the data buffer is equal to the preset data amount, store the target metadata in the data buffer as a target file by column; based on the target files generated by each of the multiple data buffers, obtain multiple target files with a storage data amount of the preset data amount.

[0080] According to an embodiment of the present invention, in the process of inputting multiple target metadata into multiple data buffers in parallel, it is necessary to continuously determine the target metadata to be input into the data buffer from the multiple target metadata, so as to input the target metadata to be input into the data buffer in parallel until the storage data amount of the target metadata in the data buffer is equal to the preset data amount.

[0081] According to an embodiment of the present invention, the number of data buffers is not limited and can be determined based on actual conditions or the historical number of data buffers during historical backup processes or in other embodiments based on the above-mentioned generation parallelism.

[0082] According to an embodiment of the present invention, in some embodiments, for example: if the number of data buffers is 3, then 3 target files can be generated at one time. If the number of target files that can be generated by multiple target metadata is greater than 3, then after the data buffer generates 3 target files for the first time, the data in the data buffer is cleared, and then the remaining target metadata are respectively input into the data buffer after the data is cleared.

[0083] According to an embodiment of the present invention, when the storage data amount of the last remaining target metadata cannot be equal to the preset data amount, all the last remaining target metadata are input into a data buffer and a target file is generated.

[0084] According to an embodiment of the present invention, a target file is generated through a data buffer. Due to the advantages of the data buffer such as improving the data reading rate and reducing input and output operations, generating the target file through the data buffer can speed up the generation of the target file. Moreover, the target file generated by column storage also has the advantages of improving the query rate when the target metadata is subsequently read from the target file.

[0085] According to an embodiment of the present invention, the above data processing method may further include the following operations.

[0086] Determine the compression ratio of the target file; multiply the data volume of the target metadata in the data buffer by the compression ratio to obtain the storage data volume.

[0087] According to the embodiment of the present invention, there is no limitation on the compression ratio of the target file, and different compression ratios can be set according to actual conditions. Specifically, the compression ratio can be set between 1:1.5 and 1:20.

[0088] According to an embodiment of the present invention, the storage data volume of the target metadata can be obtained by multiplying the data volume of the target metadata by the compression ratio. The storage data volume of the target metadata in a data buffer is accumulated to determine the storage data volume of the target file.

[0089] According to an embodiment of the present invention, by storing compressed target metadata in a target file in a column-based manner, more target metadata can be stored in one target file, thereby improving file resource occupancy.

[0090] According to an embodiment of the present invention, the index information includes a table identifier of a table to which the initial metadata belongs, data type information to which the initial metadata belongs, and a data group identifier to which the initial metadata belongs. The above data processing method may further include the following operations.

[0091] For each initial metadata, based on the acquisition path corresponding to the initial metadata, determine the table identifier of the initial metadata; based on the location information of the initial metadata, determine the data type information and data group identifier corresponding to the initial metadata in the location and index information correspondence table, wherein the location information represents the location of the initial metadata in the table used to store the initial metadata, and the location and index information correspondence table includes the correspondence between the location information and the data type information of the initial metadata and the correspondence between the location information and the data group identifier; based on the data type information, the data group identifier and the table identifier of the initial metadata, generate index information.

[0092] According to an embodiment of the present invention, the position and index information correspondence table may be stored in the database to be backed up, and the position and index information correspondence table may be obtained from the database to be backed up before the index information of the initial metadata is determined.

[0093] According to an embodiment of the present invention, when obtaining index information corresponding to each initial metadata, the table identifier can be first determined through the acquisition path of the initial metadata, and then the data type information and data group identifier of the initial metadata can be determined through the storage position of the initial metadata in the table used to store the initial metadata.

[0094] According to an embodiment of the present invention, after obtaining the index information, each index information can be arranged according to a preset priority by the index type of each index information and then spliced ​​with the initial metadata. For example, the index type of the data type information is the data type, the index type of the data group identifier is the data group type, and the index type of the table identifier is the table type. The index information is arranged in the order of table identifier-data type information-data group identifier by the preset priority and then spliced ​​with the initial metadata.

[0095] According to an embodiment of the present invention, by arranging the index information according to a preset priority and then splicing it with the initial metadata, it is possible to conveniently determine the table where the initial metadata is located, the data type of the initial metadata, the data group where it is located, and other information when performing subsequent data recovery, thereby speeding up data recovery.

[0096] According to an embodiment of the present invention, the above data processing method may further include the following operations.

[0097] Based on the index information of the target metadata in each target file, the correspondence between each target file and the table identifier is determined; based on the correspondence between each target file and the table identifier, an index file is generated; and the index file is uploaded to the backup space.

[0098] According to an embodiment of the present invention, after a plurality of target metadata are generated as target files, respective table identifiers of the target metadata stored in each target file are determined respectively, thereby determining the corresponding relationship between the target files and the table identifiers.

[0099] According to an embodiment of the present invention, a first relational index representing the correspondence between each target file and the table identifier can be determined based on the correspondence between the target file and the table identifier, so that metadata in those tables can be determined to be stored in each target file based on the first relational index. A second relational index representing the correspondence between each table and the target file can be determined, so that all metadata in a table can be determined to be stored in which target files, and an index file is generated based on the first relational index and the second relational index.

[0100] According to an embodiment of the present invention, after all target files are uploaded, the index file can be uploaded to the backup space.

[0101] According to the embodiment of the present invention, the corresponding relationship between the table identifier and the target file can be directly determined through the index file, so that the related table can be quickly restored without traversing each target file.

[0102] According to an embodiment of the present invention, the above data processing method may further include the following operations.

[0103] In response to receiving a data recovery task, the data recovery task is divided into multiple data recovery subtasks, wherein the data recovery subtask includes a recovery task for at least one table, the multiple data recovery subtasks are parallel processing tasks, and the data recovery subtask includes a table identifier of at least one table; for each data recovery subtask, multiple target metadata carrying table identifiers are obtained from multiple target files included in the backup space; the multiple target metadata are split to obtain multiple initial metadata and multiple index information; based on data writing tools corresponding to each of the multiple index information, the multiple initial metadata are written into the database to be recovered.

[0104] According to an embodiment of the present invention, the data recovery task includes restoring all metadata related to the database to be recovered in the backup space to the database to be recovered, and the data recovery task includes recovery tasks for multiple tables.

[0105] According to an embodiment of the present invention, when the data recovery task is to recover the initial metadata in the database to be backed up, the database to be recovered and the database to be backed up are the same database.

[0106] According to an embodiment of the present invention, a data recovery subtask is a task obtained by splitting a data recovery task. The splitting method is not limited and may be evenly split, or may be split according to the computing power of the task node used to execute the data recovery subtask, such as assigning a data recovery subtask with a larger task volume to a task node with strong computing power.

[0107] According to an embodiment of the present invention, after the data recovery subtask is divided, the data recovery subtask can be sent to the corresponding task node for execution, each data recovery subtask corresponds to a task node, and the task node can be a server for performing data backup or recovery.

[0108] According to an embodiment of the present invention, each task node can further divide the data recovery subtask based on the table identifier of at least one table included in the data recovery subtask, divide the data recovery subtask into tasks with the table as the dimension, and store the divided tasks in the task queue.

[0109] According to an embodiment of the present invention, the task node obtains the recovery task of the corresponding table through the task queue, and obtains multiple target metadata in the table from multiple target files included in the backup space.

[0110] According to an embodiment of the present invention, the target metadata may be split by a collection container to obtain initial metadata and index information.

[0111] According to the embodiment of the present invention, there is no limitation on the set container, and it may be an unordered set that does not allow duplicate elements, or other set containers such as a hash table.

[0112] According to an embodiment of the present invention, since the collection container does not allow duplicate elements, even if there are duplicated tables during the backup process, they will not be restored repeatedly during restoration, thereby reducing resource waste and data anomalies caused by data duplication.

[0113] According to the embodiment of the present invention, there is no limitation on the data writing tool, and it may be a pre-maintained tool that can be quickly called through a corresponding tool interface.

[0114] According to an embodiment of the present invention, the corresponding initial metadata is written by a corresponding data writing tool, thereby avoiding the problem of data being prone to errors that may exist if the initial metadata is written directly due to different writing rules for different data types.

[0115] According to an embodiment of the present invention, obtaining a plurality of target metadata carrying table identifiers from a plurality of target files included in a backup space may further include the following operations.

[0116] An index file is obtained from a backup space, and based on the index file and the table identifier, a file to be processed is determined from multiple target files, wherein the index file includes a corresponding relationship between the target file and the table identifier, and multiple target metadata carrying the table identifier are stored in the file to be processed; based on the table identifier, multiple target metadata carrying the table identifier of the index information are obtained from the file to be processed.

[0117] According to an embodiment of the present invention, each task node can obtain an index file from the backup space, determine the target file where the table identifier in the subtask to be restored is located through the first relationship index and the second relationship index in the index file, and use the target file as the file to be processed, and download it from the backup space to the local for processing.

[0118] According to an embodiment of the present invention, by traversing the table identifiers in the index information carried by the target metadata in the to-be-processed file, the target metadata having the same table identifier as the table identifier in the data recovery subtask is taken out from the to-be-processed file.

[0119] According to an embodiment of the present invention, when the file format of the target file is the ORC file format, the target metadata having the same table identifier as the table identifier of the data recovery subtask can be determined through the predicate pre-filtering function of the target file.

[0120] According to the embodiment of the present invention, the target file where the table identifier in the subtask to be restored is located can be quickly determined through the index file, thereby improving the data recovery speed to a certain extent.

[0121] According to an embodiment of the present invention, based on data writing tools corresponding to the multiple index information respectively, writing multiple initial metadata into the database to be restored may include the following operations.

[0122] Based on the data group identifier and data type information carried by each of the multiple target metadata, the initial metadata corresponding to each of the multiple target metadata are grouped to obtain multiple metadata groups; based on the data type information corresponding to each of the multiple metadata groups, the data writing tools corresponding to each of the multiple metadata groups are determined; based on the data writing tools corresponding to each of the multiple metadata groups, the multiple metadata groups are respectively written into the database to be recovered.

[0123] According to an embodiment of the present invention, based on the data group identifier and data type information in the index information of each initial metadata, multiple initial metadata can be grouped, so that initial metadata with the same data type and the same data identifier are grouped into one metadata group.

[0124] According to an embodiment of the present invention, a plurality of initial metadata may be grouped based on a collection container.

[0125] According to an embodiment of the present invention, since the data type of the initial metadata in each metadata group is the same, the data of the initial metadata group can be written together into the database to be recovered through a data writing tool corresponding to the data type information of the initial metadata group.

[0126] According to an embodiment of the present invention, the location information of each initial metadata may be recorded when data is written, thereby generating a new location and index information correspondence table.

[0127] According to an embodiment of the present invention, since the initial metadata is first grouped and then a data writing tool is determined for each metadata group, the number of times the data writing tool is determined and the number of times the data writing tool writes the initial metadata can be reduced, thereby improving the recovery speed of the data recovery process.

[0128] Figure 3 A schematic diagram of splicing index information and initial metadata according to an embodiment of the present invention is schematically shown.

[0129] like Figure 3 As shown, Figure 3The content in the tag 300 in shows the index information of initial metadata 311, initial metadata 312, initial metadata 313, initial metadata 321, initial metadata 322, initial metadata 323, initial metadata 331, initial metadata 332, and initial metadata 333. It can be seen that the data group identifiers of initial metadata 311, initial metadata 312, and initial metadata 313 are all data group identifiers A1 and the data type information is data type information A, the data group identifiers of initial metadata 321, initial metadata 322, and initial metadata 323 are all data group identifiers B1 and the data type information is data type information B, and the data group identifiers of initial metadata 311, initial metadata 312, and initial metadata 313 are all data group identifiers B1 and the data type information is data type information B. The data group identifiers of initial metadata 331, initial metadata 332, and initial metadata 333 are all data group identifier B2, and the data type information is all data type information B, and initial metadata 321, initial metadata 322, initial metadata 323 and initial metadata 331, initial metadata 332, and initial metadata 333 have the same data type information B, and the relationship between initial metadata 311, initial metadata 312, initial metadata 313, initial metadata 321, initial metadata 322, initial metadata 323, initial metadata 331, initial metadata 332, and initial metadata 333 are all metadata in a table, and its table identifier is table identifier 1.

[0130] According to an embodiment of the present invention, when the index information and the initial metadata are spliced, the table identifier 1-data type information B-data group identifier B2-initial metadata can be spliced ​​in sequence as shown in the target metadata 340 corresponding to the initial metadata 333. Since the initial metadata can be key-value pair data, the target metadata 340 can be spliced ​​into table identifier 1-data type information B-data group identifier B2-initial metadata key-initial metadata value. The same is true for other initial metadata. By splicing each initial metadata, the target metadata corresponding to multiple initial metadata can be obtained.

[0131] Based on the above data method, the present invention also provides a data processing device. Figure 4 The device is described in detail.

[0132] Figure 4 The structure block diagram of the data processing device according to the embodiment of the present invention is schematically shown.

[0133] like Figure 4 As shown, the data processing device 400 of this embodiment includes a task segmentation module 410 , a data acquisition module 420 , a splicing module 430 , a file generation module 440 and a file upload module 450 .

[0134] The task splitting module 410 is used to split the backup task in response to the received backup task to obtain multiple backup subtasks, wherein the backup subtasks include at least one table acquisition task and at least one table respective acquisition path, and the multiple backup subtasks are parallel processing tasks.

[0135] The data acquisition module 420 is configured to acquire, for each backup subtask, a plurality of initial metadata of at least one table from the database to be backed up based on the acquisition path of at least one table.

[0136] The splicing module 430 is used to splice the index information corresponding to each of the multiple initial metadata with the multiple initial metadata to determine multiple target metadata.

[0137] The file generation module 440 is used to generate a plurality of target files with a storage data volume of a preset data volume based on the data volumes of the plurality of data buffers and the plurality of target metadata.

[0138] The file upload module 450 is used to upload multiple target files to the backup space in parallel.

[0139] According to an embodiment of the present invention, the file generating module 440 includes: an input submodule, a generating submodule and a first file determining submodule.

[0140] The input submodule is used to input multiple target metadata into multiple data buffers in parallel. The generation submodule is used to store the target metadata in the data buffer as a target file by column when it is determined that the storage data amount of the target metadata in the data buffer is equal to the preset data amount for each data buffer. The first file determination submodule is used to obtain multiple target files with a preset data amount based on the target files generated by each of the multiple data buffers.

[0141] According to an embodiment of the present invention, the data processing device 400 includes: a compression ratio determination module and a calculation module.

[0142] The compression ratio determination module is used to determine the compression ratio of the target file. The calculation module is used to multiply the data volume of the target metadata in the data buffer by the compression ratio to obtain the storage data volume.

[0143] According to an embodiment of the present invention, the index information includes the table identifier of the table to which the initial metadata belongs, the data type information to which the initial metadata belongs, and the data group identifier to which the initial metadata belongs; the data processing device 400 includes: a first information determination module, a second information determination module, and an information generation module.

[0144] The first information determination module is used to determine the table identifier of the initial metadata for each initial metadata based on the acquisition path corresponding to the initial metadata. The second information determination module is used to determine the data type information and data group identifier corresponding to the initial metadata in the position and index information correspondence table based on the position information of the initial metadata, wherein the position information represents the position of the initial metadata in the table used to store the initial metadata, and the position and index information correspondence table includes the correspondence relationship between the position information and the data type information of the initial metadata and the correspondence relationship between the position information and the data group identifier. The information generation module is used to generate index information based on the data type information, data group identifier and table identifier of the initial metadata.

[0145] According to an embodiment of the present invention, the data processing device 400 further includes: a relationship determination module, an index file generation module, an index file generation module, and an index file upload module.

[0146] The relationship determination module is used to determine the corresponding relationship between each target file and the table identifier based on the index information of the target metadata in each target file.

[0147] The index file generation module is used to generate an index file based on the corresponding relationship between each target file and the table identifier.

[0148] The index file upload module is used to upload the index file to the backup space.

[0149] According to an embodiment of the present invention, the data processing device 400 further includes: a recovery subtask determination module, a recovery data determination module, a data splitting module and a data writing module.

[0150] A recovery subtask determination module is used to, in response to receiving a data recovery task, split the data recovery task into multiple data recovery subtasks, wherein the data recovery subtask includes a recovery task for at least one table, the multiple data recovery subtasks are parallel processing tasks, and the data recovery subtask includes a table identifier of at least one table.

[0151] The recovery data determination module is used to obtain, for each data recovery subtask, a plurality of target metadata carrying table identifiers from a plurality of target files included in the backup space.

[0152] The data splitting module is used to split the multiple target metadata to obtain multiple initial metadata and multiple index information.

[0153] The data writing module is used to write a plurality of initial metadata into the database to be restored based on data writing tools corresponding to the plurality of index information respectively.

[0154] According to an embodiment of the present invention, a restored data determination module includes: a second file determination submodule and a restored data determination submodule.

[0155] The second file determination submodule is used to obtain an index file from the backup space, and determine a file to be processed from multiple target files based on the index file and the table identifier, wherein the index file includes a correspondence between the target file and the table identifier, and the file to be processed stores multiple target metadata carrying the table identifier.

[0156] The recovery data determination submodule is used to obtain multiple target metadata of the index information carrying the table identifier from the to-be-processed file based on the table identifier.

[0157] According to an embodiment of the present invention, the data writing module includes: a grouping submodule, a tool determination submodule and a writing submodule.

[0158] The grouping submodule is used to group the initial metadata corresponding to each of the multiple target metadata based on the data group identifier and data type information carried by each of the multiple target metadata to obtain multiple metadata groups.

[0159] The tool determination submodule is used to determine the data writing tools corresponding to each of the plurality of metadata groups based on the data type information corresponding to each of the plurality of metadata groups.

[0160] The writing submodule is used to write the multiple metadata groups into the database to be recovered respectively based on the data writing tools corresponding to the multiple metadata groups respectively.

[0161] According to an embodiment of the present invention, any multiple modules among the task segmentation module 410, the data acquisition module 420, the splicing module 430, the file generation module 440 and the file upload module 450 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the task segmentation module 410, the data acquisition module 420, the splicing module 430, the file generation module 440 and the file upload module 450 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware and firmware or by a suitable combination of any of them. Alternatively, at least one of the task segmentation module 410, the data acquisition module 420, the splicing module 430, the file generation module 440 and the file upload module 450 may be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function may be executed.

[0162] Figure 5 A block diagram of an electronic device suitable for implementing a data method according to an embodiment of the present invention is schematically shown.

[0163] like Figure 5 As shown, the electronic device 500 according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 to a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include an onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0164] In RAM 503, various programs and data required for the operation of electronic device 500 are stored. Processor 501, ROM 502 and RAM 503 are connected to each other via bus 504. Processor 501 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 502 and / or RAM 503. It should be noted that the program can also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in the one or more memories.

[0165] According to an embodiment of the present invention, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device 500 may further include one or more of the following components connected to the input / output (I / O) interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 508 including a hard disk, etc.; and a communication portion 509 including a network interface card such as a LAN card, a modem, etc. The communication portion 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed so that a computer program read therefrom is installed into the storage portion 508 as needed.

[0166] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0167] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 502 and / or RAM 503 described above and / or one or more memories other than ROM 502 and RAM 503.

[0168] The embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the data processing method provided by the embodiment of the present invention.

[0169] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when it is executed by the processor 501. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0170] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 509, and / or installed from the removable medium 511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0171] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the processor 501, the above functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0172] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0173] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0174] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present invention may be combined and / or coupled in a variety of ways, even if such combinations and / or couplings are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention may be combined and / or coupled in a variety of ways without departing from the spirit and teachings of the present invention. All of these combinations and / or couplings fall within the scope of the present invention.

[0175] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: In response to receiving a backup task, splitting the backup task into a plurality of backup subtasks, wherein the backup subtasks include at least one table acquisition task and respective acquisition paths of the at least one table, and the plurality of backup subtasks are parallel processing tasks; For each of the backup subtasks, based on the acquisition path of the at least one table, obtaining a plurality of initial metadata of the at least one table from the database to be backed up; splicing the index information corresponding to each of the plurality of initial metadata with the plurality of initial metadata to determine a plurality of target metadata; Based on the data amounts of the plurality of data buffers and the plurality of target metadata, a plurality of target files having a storage data amount of a preset data amount are generated; Uploading a plurality of the target files to the backup space in parallel; The step of generating a plurality of target files having a preset data volume based on the data volumes of the plurality of data buffers and the plurality of target metadata comprises: Inputting a plurality of the target metadata into a plurality of the data buffers in parallel; For each of the data buffers, when it is determined that the storage data amount of the target metadata in the data buffer is equal to the preset data amount, storing the target metadata in the data buffer as a target file by column; Based on the target files generated by the plurality of data buffers respectively, a plurality of target files having a storage data volume of a preset data volume are obtained; Wherein, the method further comprises: Determining a compression ratio of the target file; The data volume of the target metadata in the data buffer is multiplied by the compression ratio to obtain the storage data volume.

2. The method according to claim 1, characterized in that The index information includes a table identifier of a table to which the initial metadata belongs, data type information to which the initial metadata belongs, and a data group identifier to which the initial metadata belongs. The method further includes: For each of the initial metadata, determining a table identifier of the initial metadata based on an acquisition path corresponding to the initial metadata; Based on the position information of the initial metadata, determine the data type information and the data group identifier corresponding to the initial metadata in a position and index information correspondence table, wherein the position information represents the position of the initial metadata in a table used to store the initial metadata, and the position and index information correspondence table includes a correspondence relationship between the position information of the initial metadata and the data type information and a correspondence relationship between the position information and the data group identifier; The index information is generated based on the data type information of the initial metadata, the data group identifier and the table identifier.

3. The method according to claim 2, characterized in that The method further comprises: Determine, based on the index information of the target metadata in each of the target files, a corresponding relationship between each of the target files and the table identifier; Generate an index file based on the correspondence between each of the target files and the table identifier; The index file is uploaded to the backup space.

4. The method according to claim 3, characterized in that The method further comprises: In response to receiving a data recovery task, the data recovery task is split into a plurality of data recovery subtasks, wherein the data recovery subtasks include at least one table recovery task, the plurality of data recovery subtasks are parallel processing tasks, and the data recovery subtasks include a table identifier of the at least one table; For each of the data recovery subtasks, acquiring a plurality of target metadata carrying the table identifier from a plurality of target files included in the backup space; Splitting the plurality of target metadata to obtain a plurality of initial metadata and a plurality of index information; Based on data writing tools corresponding to each of the plurality of index information, the plurality of initial metadata are written into the database to be restored.

5. The method according to claim 4, characterized in that The step of acquiring a plurality of target metadata carrying the table identifier from a plurality of target files included in the backup space includes: Acquire an index file from the backup space, and determine a to-be-processed file from a plurality of target files based on the index file and the table identifier, wherein the index file includes a correspondence between the target file and the table identifier, and the to-be-processed file stores a plurality of target metadata carrying the table identifier; Based on the table identifier, a plurality of target metadata whose index information carries the table identifier is obtained from the to-be-processed file.

6. The method according to claim 4, characterized in that The step of writing the plurality of initial metadata into the database to be restored based on the data writing tools corresponding to the plurality of index information respectively includes: Based on the data group identifiers and data type information respectively carried by the plurality of target metadata, the initial metadata respectively corresponding to the plurality of target metadata are grouped to obtain a plurality of metadata groups; Determining a data writing tool corresponding to each of the plurality of metadata groups based on data type information corresponding to each of the plurality of metadata groups; Based on the data writing tools corresponding to the plurality of metadata groups respectively, the plurality of metadata groups are written into the database to be recovered respectively.

7. A data processing device, characterized in that: The device comprises: A task splitting module, configured to split the backup task into a plurality of backup subtasks in response to receiving the backup task, wherein the backup subtasks include at least one table acquisition task and respective acquisition paths of the at least one table, and the plurality of backup subtasks are parallel processing tasks; A data acquisition module, configured to acquire, for each of the backup subtasks, a plurality of initial metadata of the at least one table from the database to be backed up based on the acquisition path of the at least one table; A splicing module, used for splicing the index information corresponding to each of the plurality of initial metadata with the plurality of initial metadata to determine a plurality of target metadata; A file generation module, used for generating a plurality of target files with a storage data volume of a preset data volume based on the data volume of each of the plurality of data buffers and the plurality of target metadata; A file uploading module, used for uploading a plurality of target files to the backup space in parallel; Wherein, the file generation module includes: an input submodule, a generation submodule and a first file determination submodule; The input submodule is used to input the plurality of target metadata into the plurality of data buffers in parallel; The generating submodule is configured to, for each of the data buffers, store the target metadata in the data buffer as a target file by column when it is determined that the storage data amount of the target metadata in the data buffer is equal to the preset data amount; The first file determination submodule is used to obtain a plurality of target files with a storage data volume of a preset data volume based on the target files generated by the plurality of data buffers respectively; The data processing device further includes: a compression ratio determination module and a calculation module; The compression ratio determination module is used to determine the compression ratio of the target file; The calculation module is used to multiply the data volume of the target metadata in the data buffer by the compression ratio to obtain the storage data volume.

8. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 6.

Images