Data processing method and related device

Abstract

The invention relates to the field of data processing. The embodiment of the invention discloses a data processing method and a related device. The method comprises the steps that when a deletion event for first data is monitored, basic information of the first data is acquired; determining k pieces of second data based on the basic information, and storing address information and authority information of the first data and the k pieces of second data into a recycle bin; when a recovery event for the first data is monitored, determining third data in the first data and the k pieces of second data; and recovering the third data based on the address information of the third data in the recycle bin, and recovering the authority of the third data based on the authority information of the third data. By adopting the embodiment of the invention, the data of the specified version can be recovered, the data recovery function is more flexible, and meanwhile, the permission information of the data can be quickly recovered, so that the purpose of quickly recovering the production business is achieved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method and related devices. Background Art

[0002] Hadoop Distributed File System (HDFS) is a distributed file system designed to address the challenges faced by traditional file systems in big data environments. It can run on general-purpose hardware and provide high-throughput data access, making it suitable for application scenarios with large-scale data sets.

[0003] HDFS is configured with a recycle bin. When a user deletes an HDFS file, HDFS does not directly remove the file from the system. Instead, it moves the file to the HDFS recycle bin and waits. During the waiting process, if a recovery event for the file is detected, the file will be restored. If no recovery event is detected, the file will be removed from the system after the waiting process.

[0004] Due to the characteristics of HSDF, the permission information of the HSDF file will be lost when the HSDF file is moved to the recycle bin of HDFS. When it is necessary to restore a deleted HDFS file, only the data of the HDFS file can be restored, and the lost permission information cannot be restored. As a result, the restored HDFS file cannot be directly called, and professionals are required to rebuild the HDFS file. At the same time, the current recovery method in HDFS can only restore the version of the HDFS file when it was deleted, and other versions of the HDFS file cannot be restored. If the HDFS file is modified before it is deleted, only the modified HDFS file can be restored, and the HDFS file before the modification cannot be restored.

[0005] It can be seen from this that the current HDFS recovery method has low flexibility, low recovery efficiency, and high recovery cost. After the HDFS file is restored, it needs to be manually rebuilt before it can be used again, and production cannot be restored quickly. Summary of the invention

[0006] The implementation mode of the present application proposes a data processing method and related devices. The implementation mode of the present application can realize the recovery of data of a specified version, making the data recovery function more flexible. At the same time, it is conducive to quickly recovering the permission information of the data, thereby achieving the purpose of quickly restoring production business.

[0007] This application is implemented using the following technical solutions.

[0008] In a first aspect, an embodiment of the present application provides a data processing method, including:

[0009] When a deletion event for first data is detected, obtaining basic information of the first data;

[0010] Determine k second data based on the basic information, and save address information and permission information of the first data and the k second data to a recycle bin, wherein each second data is version data corresponding to the first data in a first time period, the first time period is a time period before the moment when the first data is created, and k is an integer greater than or equal to 0;

[0011] When a recovery event for the first data is monitored, determining third data from the first data and the k second data;

[0012] The third data is restored based on the address information of the third data in the recycle bin, and the authority of the third data is restored based on the authority information of the third data.

[0013] It can be seen that when a deletion event for the first data is detected, k second data are determined in the version data corresponding to the first data in the time period before the creation time of the first data based on the basic information of the first data, and the address information and permission information of the first data and the k second data are saved to the recycle bin. When the first data needs to be restored, the third data is determined from the first data and the k second data, and the third data is restored based on the address information and permission information of the third data saved in the recycle bin. In this way, the recovery of the specified version of data is achieved, making the data recovery function more flexible. At the same time, it is conducive to quickly recovering the permission information of the data, thereby achieving the purpose of quickly restoring production business.

[0014] In a second aspect, an embodiment of the present application provides a data processing device, including:

[0015] An acquisition module, configured to acquire basic information of the first data when a deletion event for the first data is detected;

[0016] a saving module, configured to determine k second data based on the basic information, and save address information and permission information of the first data and the k second data to a recycle bin, wherein each second data is version data corresponding to the first data in a first time period, the first time period is a time period before the moment when the first data is created, and k is an integer greater than or equal to 0;

[0017] A recovery module is used to determine third data from the first data and the k second data when a recovery event for the first data is monitored, restore the third data based on address information of the third data in the recycle bin, and restore permissions of the third data based on permission information of the third data.

[0018] A third aspect is an electronic device provided for an embodiment of the present application, comprising a processor, a memory, and a computer program or instructions stored on the memory, wherein the processor executes the computer program or instructions to implement the steps in the method designed in the first aspect above.

[0019] In a fourth aspect, a computer-readable storage medium is provided for an embodiment of the present application, wherein a computer program or instruction is stored therein, and when the computer program or instruction is executed, the steps in the method designed in the first aspect are implemented. For example, the computer program or instruction is executed by a processor.

[0020] In a fifth aspect, a computer program product provided by an embodiment of the present application includes a computer program or an instruction, wherein when the computer program or the instruction is executed, the steps in the method designed in the first aspect are executed. For example, the computer program or the instruction is executed by a processor.

[0021] The beneficial effects brought about by the technical solutions of the second to fifth aspects can be referred to the technical effects brought about by the technical solution of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the implementation methods of the present application will be briefly introduced below. Obviously, the drawings described below are only some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 A schematic diagram of a system architecture provided for an implementation method of the present application;

[0024] Figure 2 Another schematic diagram of a system architecture provided for an implementation method of the present application;

[0025] Figure 3 A flowchart of a data processing method provided for an embodiment of the present application;

[0026] Figure 4 A schematic diagram of a table data backup process provided for an implementation method of the present application;

[0027] Figure 5 A schematic diagram of a process for table data recovery provided in an embodiment of the present application;

[0028] Figure 6 A block diagram of the functional modules of a data processing device provided in an embodiment of the present application;

[0029] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0031] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms of "a", "said" and "the" used in the embodiments of the present application, the accompanying drawings and the claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should be understood that the term "and / or" used in this article is only a kind of association relationship describing the associated objects, indicating that there may be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. It should be understood that although the terms "first", "second" and similar words may be used in the embodiments of the present application, they do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, similar words such as "one" or "one" do not indicate a quantitative limitation, but indicate that there is at least one. The "multiple" involved in the embodiments of the present application refers to greater than or equal to two.

[0032] See also Figure 1 , Figure 1 A schematic diagram of a system architecture provided for the implementation of this application. Figure 1 As shown, the system includes a server 101 and a user device 102 .

[0033] Among them, the server 101 is a device for data storage and data processing, such as a cloud server, a distributed server, a cabinet server, a blade server, a tower server, a server cluster, etc.

[0034] The user device 102 is a device that provides data connectivity to the user, such as a handheld device with wireless connection function, a vehicle-mounted device, etc. Common user devices 102 include, for example, mobile phones, tablet computers, laptop computers, PDAs, mobile internet devices (MIDs), wearable devices such as smart watches and smart bracelets, etc.

[0035] The server 101 stores data, such as table data, document data, audio data, video data, etc. A user can access the server 101 through the user device 102 to perform operations on the data stored in the server 101, such as read operations, write operations, delete operations, permission settings, or permission adjustments. Figure 2 As shown, the server 101 is also deployed with a lifecycle management system, which is used to manage the data stored on the server 101.

[0036] In one example, if Figure 2 As shown, the lifecycle management system includes data recycling and data recovery functions. The user sends a deletion instruction for the first data to the server 101 through the user device 102. After receiving the deletion instruction, the server 101 starts to execute the deletion operation for the first data; the pre-execution hook program is deployed in the metadata management module of the lifecycle management system to monitor the deletion event; when the pre-execution hook program monitors the deletion event for the first data, the server 101 blocks the execution of the deletion event for the first data, and the metadata management module obtains the monitoring context, and determines the basic information of the first data based on the monitoring context, and then determines k second data in the version data corresponding to the first data based on its basic information. Then, the backup data of the first data and the k second data is obtained, and the backup data includes the storage address information and permission information of the data file of the corresponding data; optionally, the backup data also includes the structural information of the corresponding data. The metadata management module sends the backup data to the message middleware; after the backup data is sent to the message middleware, the server 101 continues to execute the deletion event. The lifecycle management system monitors the messages received by the message middleware, saves the backup data,

[0037] When the user needs to restore the first data, the user indicates a restore instruction of restoring the first data to the server 101 through the user device 102. The lifecycle management system of the server 101 determines the third data to be restored from the first data and the k second data, obtains the backup data corresponding to the third data, and restores the third data based on the backup data, including the data file and permission information of the third data. Optionally, the lifecycle management system of the server 101 can also restore the structural information of the third data based on the backup data.

[0038] It can be seen that in the scheme of this embodiment, before deleting the first data with permission, k second data are determined in the version data corresponding to the first data in the time period before the creation time of the first data based on the basic information of the first data, and the data files and permission information of the first data and the k second data are saved; when it is necessary to restore the first data, the third data is determined based on the first data and the k second data, and the third data with permission is restored based on the data files and permission information of the third data, so as to realize the recovery of the specified version of data, make the data recovery function more flexible, and at the same time, facilitate the rapid recovery of the permission information of the data, thereby achieving the purpose of rapid recovery of production business.

[0039] The solution of this application is described in detail below.

[0040] See also Figure 3 , Figure 3 A flowchart of a data processing method provided in the embodiment of the present application is shown in FIG. Figure 3 As shown, the method includes:

[0041] S301: When a data processing device detects a deletion event for first data, the data processing device obtains basic information of the first data.

[0042] It should be pointed out here that Figure 3 The data processing device of the embodiment shown can be Figure 1 The server 101 or the functional module in the server 101.

[0043] In this embodiment, a lifecycle management system is deployed in the data processing device to manage the data stored on the server 101, where the data includes but is not limited to table data, document data, audio data, video data, etc. The first data can be table data, document data, audio data, video data.

[0044] In this embodiment, the lifecycle management system includes a metadata management module, in which a pre-hook program is deployed and executed. The data processing device monitors operations on data stored in the server 101 by executing the pre-hook program. When a deletion event is detected, the data processing device first blocks the execution of the deletion event and obtains a monitoring context; wherein the deletion event is used to delete the first data, in other words, the deletion event is a deletion event for the first data; the data processing device obtains basic information of the first data from the monitoring context.

[0045] In this embodiment, the basic information of the first data may include: the data size and usage frequency of the first data, and associated data representing the association relationship between the first data and each task, and the above tasks may be each task running in the system.

[0046] S302: The data processing device determines k second data based on the basic information, and saves the address information and permission information of the first data and the k second data into a recycle bin.

[0047] In this embodiment, the second data is the version data corresponding to the first data in the first time period, and the first time period is the time period before the moment when the first data is created. Specifically, each time the first data is added, deleted, checked, modified, read, written, etc., the first data before the operation can be saved as a version data of the first data. For example: if the first data is modified once, the first data before the modification can be saved as a version data of the first data, and the modified first data can be saved as the latest version of the first data. The moment of saving the modification can be understood as the moment of creating the latest version of the first data. Therefore, the first time period is the time period before the moment of saving the latest version of the first data, and the version data corresponding to the first data in the first time period is the various version data of the first data saved in the time period.

[0048] For example, the first data 1.0 is modified at the first moment, and the modification is completed at the second moment. At the first moment, the first data 1.0 is saved as a version of data 1.0, and at the second moment, the modified first data 1.0 is saved as first data 2.0. The second moment is the moment when the first data 2.0 is created.

[0049] The first data 2.0 is written with new data at the third moment, and the writing is completed at the fourth moment. At the third moment, the first data 2.0 is saved as a version data 2.0, and at the fourth moment, the first data 2.0 that has been written is saved as the first data 3.0. The fourth moment is the moment when the first data 3.0 is created.

[0050] At this time, if the first data 3.0 is deleted, the first time period is the time period before the moment when the first data 3.0 is created, that is, the time period before the fourth moment, and the version data related to the first data 3.0 in this time period is version data 1.0 and version data 2.0. Then the second data is the version data 1.0 and / or version data 2.0.

[0051] In this implementation, the version data of the first data may be imperceptible to the user, that is, the version data is merely an archive of the first data by the system, and the user may not be aware that an archive of the first data is retained in the system.

[0052] In this implementation, firstly, a value k is determined based on basic information of the first data, where k may be an integer greater than or equal to 0. Then, k version data are selected from all version data corresponding to the first data in the first time period as k second data.

[0053] In this implementation, the value k is determined based on the basic information of the first data. Specifically, the value k may be determined as follows:

[0054] First, the data size of the first data can be input into the first formula to obtain the first score. Specifically, the data size can reflect the importance of the data to a certain extent. Generally speaking, the larger the data, the more difficult it is to retrieve or reconstruct it after deletion than the smaller data. Therefore, the larger the data, the greater the importance, and the two can be in direct proportion. In this embodiment, the first formula can be shown as formula ①:

[0055] S1=1-e -αD ………①

[0056] Among them, S1 is the first score, D is the data size of the first data, and α is a hyper parameter, which can be determined by the difference between the data sizes of the maximum data and the minimum data in the system.

[0057] Then, the usage frequency of the first data may be input into a second formula to obtain a second score.

[0058] Specifically, the frequency of use of data can also reflect the importance of the data to a certain extent. Generally speaking, the more frequently used data is, the greater its importance is. The two can be in direct proportion. In this embodiment, the second formula can be shown as formula ②:

[0059]

[0060] Among them, S2 is the second score, F is the usage frequency of the first data, B, ε and h are hyper parameters, which can be determined by the distribution of the usage frequency of each data in the system.

[0061] Then, based on the associated data of the first data, multiple first tasks associated with the first data can be obtained to determine the importance score of each first task. Based on the weights of the multiple first tasks, the importance scores of the multiple first tasks are weighted and summed to obtain a third score.

[0062] In this embodiment, the importance score of each first task can be determined based on a multi-factor evaluation method. Specifically, the multi-factors can include: task frequency, data scale, priority, timeliness, failure cost, etc. By normalizing each factor and then weighted summing the normalized results of each factor based on the weight of each factor, the importance score of the first task can be obtained. Of course, other methods in the art for determining the importance score of tasks in the system can also be applied to this application, and this application does not limit this.

[0063] Finally, input the first score, the second score, and the third score into the third formula to obtain the value k. In this embodiment, the third formula can be as shown in Formula ③:

[0064]

[0065] Where S1 is the first score, S2 is the second score, S3 is the third score, p1 is the weight of the first score, p2 is the weight of the second score, p3 is the weight of the third score, u and q are hyperparameters, and || is the operation symbol for taking the absolute value.

[0066] In this embodiment, after determining the value k, k second data can be determined from all the version data corresponding to the first data in the first time period. First, it should be noted that the system can retain all the version data corresponding to the first data in the first time period, or when generating the version data, calculate the importance score of the version data and only retain the version data whose importance score is greater than a preset threshold to save storage space.

[0067] Specifically, the determination method of the importance score of each version data can be as follows:

[0068] First, the usage frequency of each version data can be input into the fourth formula to obtain the fourth score. The fourth formula can follow the above Formula ②, and this application will not repeat it here.

[0069] Then, based on the associated data of each version data, multiple second tasks associated with each version data can be obtained, the importance score of each second task can be determined, and based on the weights of the multiple second tasks, the importance scores of the multiple second tasks can be weighted and summed to obtain the fifth score of each version data. The determination method of this fifth score can refer to the determination method of the above third score, and this application will not repeat it here.

[0070] Then, keyword extraction can be performed on the update log of each version data to obtain multiple keywords, and matching can be performed in a preset keyword library based on the multiple keywords, and the sixth score can be determined based on the matching result. Specifically, the matching result can be a set of multiple keywords and first keywords in the keyword library. Then, the weights and preset scores of these first keywords in the keyword library are obtained, and the sixth score is determined by weighted summation.

[0071] Finally, the fourth score, the fifth score and the sixth score can be input into the fifth formula to obtain the importance score of each version of data. The fifth formula can use the above formula ③, and this application will not repeat it here.

[0072] Meanwhile, in this embodiment, k second data can also be determined by the importance score of each version data. Specifically, all version data of the first data can be sorted in descending order according to the importance score of each version data. Then, the first k version data in the sorting are determined as k second data.

[0073] In this embodiment, after determining k second data, the address information and permission information of the first data and k second data need to be saved in the recycle bin. The permission information refers to the user's operation permission on the data, such as add, delete, check, modify, read, write, etc.

[0074] In this embodiment, the metadata object corresponding to each of the first data and k second data can be obtained through the metadata management module; wherein the metadata object corresponding to each data includes the storage address of the data file of the data in the data storage space of the server 101, that is, the original address of the data file, and the permission information of the data.

[0075] The data processing device can determine the original address of the data from the metadata object corresponding to the data, move the data file of the data from the storage space corresponding to the original address to the recycle bin based on the original address of the data, and record the storage address of the data file of the data in the recycle bin. The data file can be the data itself, or a source file used to generate the data, such as a data file of table data is a database (data base, DB) used to generate table data. The address information of the data file of the data includes the original address of the data file and the storage address of the data file in the recycle bin; the data processing device determines the permission information of the data from the metadata object corresponding to the data.

[0076] Optionally, before or after the movement, the data processing device may compress the data files of the first data and the k second data.

[0077] In one example, in the data storage space of the server, the data processing device compresses the data files of the first data and k second data to obtain the compression results of the data files, and moves the compression results of the data files to the recycle bin of the server. In this way, it is helpful to reduce the storage space occupied by the data files in the recycle bin, and at the same time, it is helpful to reduce the transmission resources for moving the data files from the storage space of the server to the recycle bin.

[0078] In another example, after moving the data files of the first data and k second data to a recycle bin, the data processing device compresses the data files of the first data and k second data in the recycle bin, and the data files are stored in the recycle bin in a compressed manner, which helps to reduce the storage space occupied by the data files in the recycle bin.

[0079] In a feasible implementation, when the first data and the k second data are structured data, such as table data, the structure information of the table data can also be saved in the recycle bin. Specifically, the metadata object of each of the first data and the k second data can also include metadata information of the table data, wherein the metadata information of the table data includes but is not limited to library name, table name, table field name, field type, partition name, table data file storage address information. The data processing device obtains the metadata information of the table data from the metadata object corresponding to the table data, and constructs the structure information of the table data based on the metadata information of the table data, such as the structured query language (SQL) of the table.

[0080] In this embodiment, after determining the above information of the first data and the k second data, the information can be saved in the recycle bin. Specifically, the data processing device stores the address information, permission information and structure information (optional) of the data files of the first data and the k second data in the recycle bin.

[0081] In a specific example, the data processing device packages the address information of the data files of the first data and k second data and the permission information of the target data into JSON files. Optionally, the JSON file also includes the structure information of the corresponding data.

[0082] The data processing device sends the JSON file to the message middleware; the recycle bin management system for managing the recycle bin monitors the message in the message middleware, obtains the JSON message from the message middleware after monitoring that the message middleware receives the JSON message, and parses the JSON message to obtain the address information, permission information and structure information (optional) of the data files of the first data and k second data; and backs up the address information, permission information and structure information (optional) of the data files of the first data and k second data to the recycle bin management database, i.e., the recycle bin. At this point, the backup of the table data file, the structure information of the table data and the permission information is completed.

[0083] It should be pointed out here that the recycling bin management system can be a part of the life cycle management system.

[0084] After completing the data backup, the data processing device continues to execute the deletion event for the first data, deletes the first data from the storage space corresponding to the original address, and deletes the permission information of the first data. If the first data is structured data, the data processing device also deletes the structure information of the first data.

[0085] S303: When the data processing device detects a recovery event for the first data, the data processing device determines third data from the first data and k second data.

[0086] In this embodiment, the data processing device can calculate the importance score of each of the first data and the k second data, and then determine the data with the highest importance score as the third data. Specifically, the calculation method of the importance score can be expressed by formula ④:

[0087]

[0088] Among them, Ri is the importance score of the i-th data among the first data and k second data, Ji is the first important score of the i-th data, Td is the current time, Ti is the generation time of the i-th data, β is the hyperparameter, and i is an integer greater than 0 and less than or equal to k+1.

[0089] Furthermore, the first importance score Ji of the i-th data can be calculated by the above-mentioned method of determining the importance score of each version of data, which will not be described in detail in this application.

[0090] In this embodiment, the data processing device may also display the first data and k second data to the user, and the user may select between the first data and the k second data, and the data processing device may determine the data selected by the user as the third data.

[0091] In this embodiment, a plurality of third data may be determined.

[0092] S304: Restore the third data based on the address information of the third data in the recycle bin, and restore the authority of the third data based on the authority information of the third data.

[0093] In this embodiment, the data processing device obtains the address information and permission information of the third data from the recycle bin. The data processing device parses the original address of the third data and the storage address of the third data in the recycle bin based on the address information, and then moves the data file of the third data from the recycle bin to the storage space corresponding to the original address based on the original address and the storage address, and associates the data file with the permission information of the third data.

[0094] Optionally, when the third data is table data, the data processing device may also obtain structural information of the third data from the recycle bin. The data processing device constructs the table data based on the structural information and the data file of the third data, and associates the table data with the permission information of the table data.

[0095] Optionally, when the data file of the third data is stored in a compressed form in the recycle bin, the data processing device decompresses the compression result of the data file in the recycle bin to obtain the data file, and moves the data file to the storage space corresponding to the original address.

[0096] Optionally, when the data file of the third data is stored in the recycle bin in a compressed form, the data processing device moves the compressed result of the data file to the storage space corresponding to the original address and then decompresses the compressed result of the data file. This method is conducive to reducing the transmission resources for moving the data file from the storage space of the server to the recycle bin.

[0097] It can be seen that in the scheme of the present application, when a deletion event for the first data is detected, k second data are determined in the version data corresponding to the first data in the time period before the creation time of the first data based on the basic information of the first data, and the address information and permission information of the first data and the k second data are saved to the recycle bin. When the first data needs to be restored, the third data is determined in the first data and the k second data, and the third data is restored based on the address information and permission information of the third data saved in the recycle bin. Thus, the restoration of the data of the specified version is realized, making the data recovery function more flexible, and at the same time, it is conducive to quickly restoring the permission information of the data, thereby achieving the purpose of quickly restoring the production business. For the table data, not only the data files and permission information of the table data are saved, but also the structural information of the table data is saved, so that when restoring the table data, the table data with permission information can be restored based on the data files, structural information and permission information of the saved table data, which is conducive to achieving the purpose of quickly restoring the production business based on the restored table data. When saving the data files of the first data and the k second data, the data files are compressed, which is conducive to reducing the storage resources occupied by saving the data files.

[0098] In a specific implementation, the solution of the present application is specifically described by taking the first data as table data as an example.

[0099] like Figure 4 As shown, the user sends a deletion instruction to the server through the user device to request to delete the table data db1.table1. After receiving the deletion instruction, the server's computing engine calls the metadata management module deployed with the pre-execution hook program to perform the operation of deleting the table data db1.table1. Since the pre-execution hook program deployed in the metadata management module is used to monitor the deletion operation performed by the metadata management module, when the pre-execution hook program monitors the deletion event for the table data db1.table1, it will prevent the metadata management module from performing the operation of deleting the table data db1.table1. The metadata management module also obtains the basic information of the table data db1.table1 from the monitoring context. The basic information includes: the data size and usage frequency of the table data db1.table1, and the associated data that characterizes the association relationship between the first data and each task. The above tasks can be each task running in the system.

[0100] Then, the metadata management module determines a value k based on the basic information of the table data db1.table1, and obtains k version table data from the version table data corresponding to the table data db1.table1 in the first time period.

[0101] Then, the metadata management module obtains the metadata objects of the table data db1.table1 and the k versions of the table data, wherein the metadata objects include the database name, table name, table field information, table field type, table data file storage address, etc.

[0102] The metadata management module obtains the metadata information of the table from the metadata object, and constructs the structural information of the table data based on the metadata information of the table. The structural information is used to create the table structure before the table data is deleted when restoring the table data. For example, the structural information can be the SQL of the table data.

[0103] Taking the table data db1.table1 as an example, the metadata information of the table includes but is not limited to the database name db1, table name table1, table field name column1, field type string, partition name, and table data file storage address information. The metadata management module constructs the structure information of the table data based on the metadata information of the table. The structure information of the table data is as follows:

[0104] CREATE TABLE `db1.table1`(

[0105] `column1` string COMMENT 'field comment 1',

[0106] `column2` string COMMENT 'Field comment 2' )

[0108] COMMENT'Table comments'

[0109] PARTITIONED BY(

[0110] `dt` string )

[0112] ROW FORMAT

[0113] SERDE'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'

[0114] STORED AS

[0115] INPUTFORMAT'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'

[0116] OUTPUTFORMAT'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'

[0117] LOCATION'hdfs: / / nameservice / user / hive / warehouse / db1.db / table1'

[0118] Among them, CREATE TABLE indicates the creation of a new table, and db1.table1 specifies that the database of the created table is db1 and the table name is table1. The data types of column1 and column2 are string, and each column contains a comment (COMMENT) to describe the meaning of the field. For example, the comment of column1 is "field comment 1". COMMENT 'table comment' indicates that a comment is provided for the table to describe the purpose or meaning of the table. Comments are optional. PARTITIONED BY defines a partition field, indicating that the table will be partitioned based on a certain field. Here, dt is specified as the partition field. The data will be partitioned according to the value of the dt field. The field type is string. ROW FORMAT SERDE defines the serialization and deserialization format (SerDe) for data storage. ParquetHiveSerDe is used here, which means that the data will be stored in Parquet format. STORED AS specifies that the data is stored in Parquet format. By specifying INPUTFORMAT and OUTPUTFORMAT, Hive is told to use the Parquet format for input and output processing: MapredParquetInputFormat: for reading Parquet files; MapredParquetOutputFormat: for writing Parquet files. LOCATION defines the data storage path of the Hive table. Here, the data is specified to be stored on the Hadoop Distributed File System (HDFS), and the specific path is hdfs: / / nameservice / user / hive / warehouse / db1.db / table1.

[0119] Specifically, the purpose of the above structural information is to create a Hive table named table1, which is located in the database db1. The table contains two fields: column1 and column2, both of which use the string type and have field annotations. The table is partitioned using the dt field. The storage format of the table is Parquet, and the relevant serialization and deserialization mechanism (ParquetHiveSerDe) is configured. The data of the table is stored in the specified path hdfs: / / nameservice / user / hive / warehouse / db1.db / table1 on HDFS. Therefore, when restoring the table data db1.table1, the table structure of the table data db1.table1 can be reconstructed through the structural data, and the data file of the table data db1.table1 can be placed in the table structure to complete the recovery of the table data db1.table1.

[0120] The metadata management module obtains the storage address of the table data file from the metadata object of the table. The storage address is the original address of the table data file. The metadata management module calls hdfs to perform a file move operation to move the table data file from the storage location corresponding to the original address to the recycle bin. The metadata management module records the storage address of the table data file in the recycle bin, such as / recycle / {timestamp} / {table data file}. Optionally, the metadata management module compresses the table data file and stores the compression result in the recycle bin to save storage space.

[0121] The metadata management module obtains the library table name from the metadata object of the table, such as db1.table1, and calls the permission management component sentry based on the library table name to obtain the data role bound to the table data (i.e. db1.table1), such as role1, role2. It should be pointed out that different data roles correspond to different permissions, and the metadata management module can use the data role bound to the table data as the permission information of the table data.

[0122] The metadata management module packages the information of the table deletion executor (i.e. user), the library name, the table name, the original address of the table data file and the storage address in the recycle bin, the structure information of the table data and the data role bound to the table data into a JSON message, and sends the JSON message to the message middleware, such as Kafka.

[0123] The recycle bin management system monitors the messages in the message middleware, and when the message middleware receives the JSON message, it obtains the JSON message from the message middleware, and parses the JSON message to obtain the information of the table deletion executor (i.e., user), the library name, the table name, the original address of the table data file and the storage address in the recycle bin, the structure information of the table data, and the data role bound to the table data; and backs up the information of the table deletion executor (i.e., user), the library name, the table name, the original address of the table data file and the storage address in the recycle bin, the structure information of the table data, and the data role bound to the table data to the recycle bin management database. The information of the table deletion executor (i.e., user), the library name, the table name, the original address of the table data file and the storage address in the recycle bin, the structure information of the table data, and the data role bound to the table data can be called the backup data corresponding to the table data. At this point, the backup of the table data file, the structure information of the table data, and the permission information is completed.

[0124] When the user realizes that the table data db1.table1 is deleted by mistake and needs to restore the deleted table data db1.table1, the user accesses the server through the user device and initiates a one-key recovery of the table data in the server's recycle bin management system. Figure 5 As shown, the user accesses the server through the user device and sends a data recovery instruction to the recycle bin management system; the recycle bin management system determines the table data to be recovered from the table data db1.table1 and the k version table data (determined by importance score, or displayed to the user through the user device and selected by the user). The backup data (address information, permission information and structure information) corresponding to the table data to be restored is read from the recycle bin record, and the storage address of the table data file to be restored in the recycle bin and the original address of the table data file to be restored are obtained from the backup data. Based on the storage address of the table data file to be restored in the recycle bin and the original address of the table data file to be restored, hdfs is called to move the table data file to be restored from the recycle bin to the storage space corresponding to the original address, and the structure information of the table data to be restored is obtained from the backup data, and hiveserver2 is called to perform a table creation operation based on the structure information of the table data to be restored to restore the structure of the table data to be restored; the permission information of the table data to be restored, that is, the data role bound to the table data to be restored, is obtained from the backup data, and the permission management component is called to bind the permission information of the table data to be restored to the table data to be restored. At this point, the recovery of the table data to be restored is completed.

[0125] The above mainly introduces the scheme of the implementation mode of the present application from the perspective of the method side. It is understandable that in order to realize the above functions, the training device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the implementation mode disclosed in this article, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0126] The implementation method of the present application can divide the functional units of the terminal device according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one functional module. The above integrated module can be implemented in the form of hardware or in the form of a software program module. It should be noted that the division of modules in the implementation method of the present application is schematic and is only a logical function division, and there may be other division methods in actual implementation.

[0127] In the case of integrated modules, Figure 6 The data processing device 600 includes an acquisition module 601 , a storage module 602 and a recovery module 603 .

[0128] In this implementation, the acquisition module 601, the storage module 602 and the recovery module 603 may be a module for receiving and processing signals, information, etc. or determining a monitoring mechanism, and there is no specific limitation on this.

[0129] In this embodiment, the data processing device 600 may further include a storage module for computer program codes or instructions executed by the data processing device 600. The storage module may be a memory.

[0130] In this embodiment, the data processing device 600 may be a chip or a chip module.

[0131] In this embodiment, the acquisition module 601, the storage module 602 and the recovery module 603 may be integrated in a communication module, wherein the communication module may be a communication interface, a transceiver, a transceiver circuit and the like.

[0132] In this implementation, the acquisition module 601 , the storage module 602 , and the recovery module 603 may be integrated into a processor.

[0133] It should be noted that the processor can be a baseband processor, a baseband chip, a central processing unit (CPU), a general processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processing module can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of DSPs and microprocessors, etc.

[0134] In this embodiment, the data processing device 600 is used to execute any step executed by the terminal device / chip / chip module, etc. in the above method implementation.

[0135] In specific implementation, the acquisition module 601, the storage module 602 and the recovery module 603 are used to execute any step in the above method implementation, and when executing actions such as sending, other modules can be selectively called to complete corresponding operations.

[0136] An acquisition module 601 is used to acquire basic information of the first data when a deletion event for the first data is detected;

[0137] A saving module 602 is used to determine k second data based on the basic information, and save the address information and permission information of the first data and the k second data to a recycle bin, wherein each second data is version data corresponding to the first data in a first time period, the first time period is a time period before the moment when the first data is created, and k is an integer greater than or equal to 0;

[0138] The recovery module 603 is used to determine the third data in the first data and the k second data when a recovery event for the first data is monitored, restore the third data based on the address information of the third data in the recycle bin, and restore the permissions of the third data based on the permission information of the third data.

[0139] In this implementation, in determining the k second data based on the basic information, the saving module 602 is specifically used to:

[0140] Determine a value k based on the basic information;

[0141] Determine an importance score of each version data among all version data corresponding to the first data in the first time period;

[0142] sorting all version data of the first data in descending order according to the importance score of each version data;

[0143] The first k version data in the sorting are determined as the k second data.

[0144] In this implementation, the basic information includes: the data size and usage frequency of the first data, and associated data characterizing the association relationship between the first data and each task. Based on this, in determining the value k based on the basic information, the saving module 602 is specifically used to:

[0145] Input the data size of the first data into a first formula to obtain a first score;

[0146] Input the usage frequency of the first data into a second formula to obtain a second score;

[0147] Based on the associated data of the first data, acquiring a plurality of first tasks associated with the first data;

[0148] Determine the importance score for each first task;

[0149] Based on the weights of the multiple first tasks, performing weighted summation on the importance scores of the multiple first tasks to obtain a third score;

[0150] The first score, the second score and the third score are input into a third formula to obtain the value k.

[0151] In this implementation, in terms of determining the importance score of each version data among all the version data corresponding to the first data in the first time period, the saving module 602 is specifically configured to:

[0152] Input the usage frequency of each version data into a fourth formula to obtain a fourth score;

[0153] Based on the associated data of each version data, acquiring a plurality of second tasks associated with each version data;

[0154] Determine the importance score for each secondary task;

[0155] Based on the weights of the plurality of second tasks, performing weighted summation on the importance scores of the plurality of second tasks to obtain a fifth score for each version of the data;

[0156] Extract keywords from the update log of each version of the data to obtain multiple keywords;

[0157] Based on the multiple keywords, matching is performed in a preset keyword library, and a sixth score is determined based on the matching result;

[0158] The fourth score, the fifth score and the sixth score are input into the fifth formula to obtain the importance score of each version of the data.

[0159] In this implementation, after determining k second data based on the basic information, the saving module 602 is further used to:

[0160] Obtaining metadata objects of each of the first data and the k second data;

[0161] Determine the original address of each data from the metadata object of each data;

[0162] Determine the storage address of each data in the recycling bin; wherein the address information of each data includes the original address of each data and the storage address of each data in the recycling bin;

[0163] The permission information of each data is determined based on the metadata object of each data.

[0164] In this implementation, in determining the storage address of each data in the recycle bin, the saving module 602 is specifically used to:

[0165] Obtaining the original address of each data;

[0166] Each data is moved to the recycle bin based on the original address of each data, and the storage address of each data in the recycle bin is obtained.

[0167] In this implementation, in terms of replying to the third data based on the address information of the third data in the recycle bin and restoring the authority of the third data based on the authority information of the third data, the recovery module 603 is specifically used to:

[0168] Moving the third data to a storage space corresponding to an original address of the third data based on the storage address of the third data in the recycle bin;

[0169] The third data is associated with the authority information to restore the authority of the third data.

[0170] In this implementation, when the first data is table data, the k second data are k table data. Based on this, the saving module 602 is further used to:

[0171] When the deletion event is detected, obtaining metadata objects of the first data and each of the second data;

[0172] Acquire first metadata information of the first data from a metadata object of the first data;

[0173] Acquire the second metadata information of each second data from the metadata object of each second data;

[0174] constructing structural information of the first data based on the first metadata information, and saving the structural information of the first data to the recycle bin;

[0175] constructing the structural information of the second data corresponding to each second metadata information based on each second metadata information, and saving the structural information of each second data to the recycle bin;

[0176] When the recovery event is detected, the recovering the third data based on the address information of the third data in the recycle bin includes:

[0177] acquiring a data file of the third data based on the storage address of the third data in the recycle bin,

[0178] The third data is restored based on the structural information of the third data in the recycle bin and the data file of the third data, and the third data is saved to a storage space corresponding to an original address of the third data.

[0179] See also Figure 7 , Figure 7 7 is a schematic diagram of a structure of an electronic device proposed in an embodiment of the present application. The electronic device 700 may include a processor 710, a memory 720, and a communication bus for connecting the processor 710 and the memory 720.

[0180] Optionally, the memory 720 includes but is not limited to random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or portable read-only memory (CD-ROM), and the memory 720 is used to store the program code executed by the electronic device 700 and the transmitted data.

[0181] In this embodiment, the electronic device 700 further includes a communication interface for receiving and sending data.

[0182] In this implementation, the processor 710 may be one or more CPUs. When the processor 710 is a CPU, the CPU may be a single-core CPU or a multi-core CPU.

[0183] In this embodiment, the processor 710 can be a baseband chip, a chip, a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA or other programmable logic devices, transistor logic devices, hardware components or any combination thereof.

[0184] In a specific implementation, the processor 710 in the electronic device 700 is used to execute the computer program or instruction 721 stored in the memory 720 to perform the following operations:

[0185] When a deletion event for first data is detected, obtaining basic information of the first data;

[0186] Determine k second data based on the basic information, and save address information and permission information of the first data and the k second data to a recycle bin, wherein each second data is version data corresponding to the first data in a first time period, the first time period is a time period before the moment when the first data is created, and k is an integer greater than or equal to 0;

[0187] When a recovery event for the first data is monitored, determining third data from the first data and the k second data;

[0188] The third data is restored based on the address information of the third data in the recycle bin, and the authority of the third data is restored based on the authority information of the third data.

[0189] It should be noted that Figure 7 The specific implementation of each operation in the implementation method can be found in the description of the method implementation method shown above, and will not be described in detail here.

[0190] The present application also provides a computer-readable storage medium storing a computer program or instructions, which implements the steps described in the above method implementation when executed.

[0191] The embodiment of the present application also provides a computer program product, including a computer program or instructions, which implement the steps described in the above method embodiment when executed.

[0192] It should be noted that, for the above-mentioned various embodiments, for the sake of simple description, they are all expressed as a series of action combinations. Those skilled in the art should be aware that the present application is not limited by the described order of actions, because some steps in the embodiments of the present application can be performed in other orders or simultaneously. In addition, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions, steps, modules or units involved are not necessarily required for the embodiments of the present application.

[0193] In the above-mentioned embodiments, the description of each embodiment of the present application has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0194] The steps of the method or algorithm described in the embodiments of the present application can be implemented in hardware or by executing software instructions by a processor. The software instructions can be composed of corresponding software modules, and the software modules can be stored in RAM, flash memory, ROM, EPROM, electrically erasable programmable read-only memory (electrically EPROM, EEPROM), registers, hard disks, mobile hard disks, read-only compact disks (CD-ROMs) or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a terminal device or a management device. Of course, the processor and the storage medium can also be present in a terminal device or a management device as discrete components.

[0195] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiments of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital video disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0196] The modules / units included in the devices and products described in the above embodiments may be software modules / units or hardware modules / units, or may be partially software modules / units and partially hardware modules / units. For example, for the devices and products applied to or integrated in the chip, the modules / units included therein may all be implemented in the form of hardware such as circuits, or at least some of the modules / units may be implemented in the form of software programs, which run on the processor integrated inside the chip, and the remaining (if any) modules / units may be implemented in the form of hardware such as circuits; for the devices and products applied to or integrated in the chip module, the modules / units included therein may all be implemented in the form of hardware such as circuits, and different modules / units may be located in the same component (such as a chip, circuit module, etc.) or in different components of the chip module, or at least some of the modules / units may be implemented in the form of software programs. The software programs run on the processor integrated inside the chip, and the remaining (if any) modules / units may be implemented in the form of hardware such as circuits. It is implemented in the form of a software program, which runs on a processor integrated inside the chip module, and the remaining (if any) modules / units can be implemented in hardware such as circuits; for various devices and products applied to or integrated in the terminal equipment, the various modules / units contained therein can be implemented in hardware such as circuits, and different modules / units can be located in the same component (for example, chip, circuit module, etc.) or in different components in the terminal equipment, or, at least some modules / units can be implemented in the form of a software program, which runs on a processor integrated inside the terminal equipment, and the remaining (if any) modules / units can be implemented in hardware such as circuits.

[0197] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the implementation methods of the present application. It should be understood that the above description is only the specific implementation method of the implementation method of the present application and is not intended to limit the protection scope of the implementation method of the present application. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the implementation methods of the present application should be included in the protection scope of the implementation methods of the present application.

Claims

1. A data processing method, characterized in that: include: When a deletion event for first data is detected, obtaining basic information of the first data; Determine k second data based on the basic information, and save address information and permission information of the first data and the k second data to a recycle bin, wherein each second data is version data corresponding to the first data in a first time period, the first time period is a time period before the moment when the first data is created, and k is an integer greater than or equal to 0; When a recovery event for the first data is monitored, determining third data from the first data and the k second data; The third data is restored based on the address information of the third data in the recycle bin, and the authority of the third data is restored based on the authority information of the third data.

2. The method according to claim 1, characterized in that: The determining k second data based on the basic information includes: Determine a value k based on the basic information; Determine an importance score of each version data among all version data corresponding to the first data in the first time period; sorting all version data of the first data in descending order according to the importance score of each version data; The first k version data in the sorting are determined as the k second data.

3. The method according to claim 2, characterized in that: The basic information includes: the data size and usage frequency of the first data, and associated data representing the association relationship between the first data and each task; The determining the value k based on the basic information includes: Input the data size of the first data into a first formula to obtain a first score; Input the usage frequency of the first data into a second formula to obtain a second score; Based on the associated data of the first data, acquiring a plurality of first tasks associated with the first data; Determine the importance score for each first task; Based on the weights of the multiple first tasks, performing weighted summation on the importance scores of the multiple first tasks to obtain a third score; The first score, the second score and the third score are input into a third formula to obtain the value k.

4. The method according to claim 2 or 3, characterized in that: Determining the importance score of each version data among all version data corresponding to the first data in the first time period includes: Input the usage frequency of each version data into a fourth formula to obtain a fourth score; Based on the associated data of each version data, acquiring a plurality of second tasks associated with each version data; Determine the importance score for each secondary task; Based on the weights of the plurality of second tasks, performing weighted summation on the importance scores of the plurality of second tasks to obtain a fifth score for each version of the data; Extract keywords from the update log of each version of the data to obtain multiple keywords; Based on the multiple keywords, matching is performed in a preset keyword library, and a sixth score is determined based on the matching result; The fourth score, the fifth score and the sixth score are input into the fifth formula to obtain the importance score of each version of the data.

5. The method according to any one of claims 1 to 4, characterized in that: After determining k second data based on the basic information, the method further includes: Obtaining metadata objects of each of the first data and the k second data; Determine the original address of each data from the metadata object of each data; Determine the storage address of each data in the recycling bin; wherein the address information of each data includes the original address of each data and the storage address of each data in the recycling bin; The permission information of each data is determined based on the metadata object of each data.

6. The method according to claim 5, characterized in that Determining the storage address of each data in the recycle bin includes: Obtaining the original address of each data; Each data is moved to the recycle bin based on the original address of each data, and the storage address of each data in the recycle bin is obtained.

7. The method according to claim 5 or 6, characterized in that: The step of replying to the third data based on the address information of the third data in the recycle bin, and restoring the authority of the third data based on the authority information of the third data, includes: Moving the third data to a storage space corresponding to an original address of the third data based on the storage address of the third data in the recycle bin; The third data is associated with the authority information to restore the authority of the third data.

8. The method according to any one of claims 1 to 7, characterized in that: When the first data is table data, the k second data are k table data, and the method further includes: When the deletion event is detected, obtaining metadata objects of the first data and each of the second data; Acquire first metadata information of the first data from a metadata object of the first data; Acquire the second metadata information of each second data from the metadata object of each second data; constructing structural information of the first data based on the first metadata information, and saving the structural information of the first data to the recycle bin; constructing the structural information of the second data corresponding to each second metadata information based on each second metadata information, and saving the structural information of each second data to the recycle bin; When the recovery event is detected, the recovering the third data based on the address information of the third data in the recycle bin includes: acquiring a data file of the third data based on the storage address of the third data in the recycle bin, The third data is restored based on the structural information of the third data in the recycle bin and the data file of the third data, and the third data is saved to a storage space corresponding to an original address of the third data.

9. The method according to any one of claims 1 to 8, characterized in that: The method further comprises: Before the address information and permission information of the first data and the k second data are saved in the recycle bin, suspending execution of the deletion event; After the address information and permission information of the first data and the k second data are saved in the recycle bin, the deletion event is executed.

10. A data processing device, characterized in that: The data processing device comprises: An acquisition module, configured to acquire basic information of the first data when a deletion event for the first data is detected; a saving module, configured to determine k second data based on the basic information, and save address information and permission information of the first data and the k second data to a recycle bin, wherein each second data is version data corresponding to the first data in a first time period, the first time period is a time period before the moment when the first data is created, and k is an integer greater than or equal to 0; A recovery module is used to determine third data from the first data and the k second data when a recovery event for the first data is monitored, restore the third data based on address information of the third data in the recycle bin, and restore permissions of the third data based on permission information of the third data.

11. An electronic device comprising a processor, a memory and a computer program or instruction stored in the memory, characterized in that: The processor executes the computer program or instructions to implement the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 9.

Images