Methods and devices for data recovery, electronic devices, and storage media

By calculating concurrency and generating query statements, data is directly imported from relational databases into the data computing cluster, solving the problem of long processing times in big data recovery and enabling flexible and real-time data recovery.

CN115586988BActive Publication Date: 2026-11-17INDUSTRIAL AND COMMERCIAL BANK OF CHINA
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202211202748.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-09-29
Publication Date
2026-11-17
Estimated Expiration
2042-09-29

AI Technical Summary

Technical Problem

Existing technologies for big data recovery are time-consuming and inflexible, and cannot respond to recovery needs in real time.

Method used

By parsing the recovery request, calculating the amount of data to be recovered and the number of concurrent requests, generating a recovery query statement, querying table data from the relational database, and importing the data into various partitions of the data computing cluster based on the mapping relationship, the data copying process is avoided.

Benefits of technology

It enables flexible and real-time data recovery, reduces data recovery time, and supports recovery scenarios with large data volumes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115586988B_ABST
    Figure CN115586988B_ABST
Patent Text Reader

Abstract

This invention discloses a method, apparatus, electronic device, and storage medium for data recovery, relating to the field of big data. The method includes: parsing a recovery request to obtain a data recovery statement, wherein the data recovery statement is used to calculate the amount of data to be recovered; calculating the recovery concurrency based on processor usage information, memory usage information, and the amount of data; generating a recovery query statement based on the recovery concurrency, wherein the recovery query statement is used to query table data corresponding to the fields to be recovered from a relational database to obtain data query results; and importing the data query results into various data partitions of the data computing cluster based on the mapping relationship between the relational database and the data computing cluster. This invention solves the technical problems of long processing times and lack of flexibility in data recovery in related technologies.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of big data technology, and more specifically, to a method and apparatus for data recovery, an electronic device, and a storage medium. Background Technology

[0002] In recent years, with the development of the Internet and information industry, big data has also developed and has been widely used in various industries and fields. Big data has a significant advantage in processing massive amounts of data.

[0003] Currently, data loss due to accidental deletion or data storage failures in the massive amounts of stored data can cause huge losses to enterprises. Therefore, data recovery has become a key focus and challenge in big data development. Related technologies typically employ copying HDFS (Hadoop Distributed File System) files to achieve data recovery during the development of large-scale wide tables. This involves copying data from the production Hadoop cluster (distributed system infrastructure) as HDFS files to a relay cluster, and then copying it from the relay cluster to the target cluster in the same way, thereby achieving the goal of restoring production environment data to the target environment.

[0004] Data recovery methods that involve copying HDFS files through a relay cluster are common in related technologies. However, since these methods involve obtaining data from the production environment, in practice, a dedicated person usually collects the recovery requirements and then recovers the data from the production Hadoop environment. Furthermore, the data needs to be copied twice during the recovery process before the recovery requirements can be met. This recovery process is lengthy, time-consuming, and inflexible, and it cannot respond to recovery requirements in real time during big data development.

[0005] There is currently no effective solution to the above problems. Summary of the Invention

[0006] This invention provides a method, apparatus, electronic device, and storage medium for data recovery, to at least solve the technical problems of long time consumption and lack of flexibility in data recovery in related technologies.

[0007] According to one aspect of the present invention, a method for recovering data is provided, comprising: parsing a recovery request to obtain a recovery data statement, wherein the recovery data statement is used to calculate the amount of data to be recovered; calculating a recovery concurrency based on processor usage information, memory usage information, and the amount of data; generating a recovery query statement based on the recovery concurrency, wherein the recovery query statement is used to query table data corresponding to the field to be recovered from a relational database to obtain a data query result; and importing the data query result into various data partitions of the data computing cluster based on the mapping relationship between the relational database and the data computing cluster.

[0008] Optionally, before parsing the recovery request and obtaining the recovery data statement, the method further includes: receiving a login request initiated by the frontend, wherein the login request carries at least: the frontend address, user account, and account password; and, if the user account and account password are verified to be correct, receiving the recovery request, wherein the recovery request carries at least the recovery selection information transmitted by the frontend, wherein the information dimensions of the recovery selection information include: file identifier, table identifier of at least one table to be recovered, data source information, cluster information, data partition information, data full amount, and data volume information.

[0009] Optionally, the step of parsing the recovery request to obtain the recovery data statement includes: parsing the first set of information dimensions in the recovery request to obtain the data source, file identifier, data volume, and data partition of the data to be recovered, wherein the first set of information dimensions includes: file identifier, data source information, data partition information, and data volume information; establishing a connection with the relational database based on the data source; and generating the recovery data statement associated with the relational database according to the file identifier, the data volume, and the data partition.

[0010] Optionally, after establishing a connection with the relational database based on the data source, the method further includes: parsing the second set of information dimensions in the recovery request, wherein the second set of information dimensions includes: table identifier and cluster information; obtaining the table structure information of the table to be recovered from the data computing cluster indicated by the cluster information and the relational database based on the table identifier; generating a table mapping template based on the table structure information; and using the table mapping relationship represented by the table mapping template as the mapping relationship between the relational database and the data computing cluster.

[0011] Optionally, the step of calculating the number of concurrent connections to be restored based on processor usage information, memory usage information, and the amount of data includes: obtaining the processor utilization rate and the number of idle threads of the processor in the previous time period to obtain the processor usage information; obtaining the memory occupancy rate and memory size in the previous time period to obtain the memory usage information; counting the number of all fields to be restored; and inputting the processor usage information, the memory usage information, the amount of data, and the number of all fields to be restored into the target neural network model to obtain the number of concurrent connections to be restored output by the target neural network model.

[0012] Optionally, the step of generating a recovery query statement based on the number of concurrent recovery attempts includes: generating the recovery query statement for each concurrent recovery data thread based on the number of concurrent recovery attempts and the recovery data statement.

[0013] Optionally, when importing the data query results into each data partition of the data computing cluster, the method further includes: receiving a recovery status query request; responding to the recovery status query request by sending a progress bar for data recovery to the front end, wherein the progress bar is used to display the data table identifier, the recovery fields, and the table data being recovered at the current moment.

[0014] According to another aspect of the present invention, an apparatus for recovering data is also provided, comprising: a parsing unit, configured to parse a recovery request to obtain a recovery data statement, wherein the recovery data statement is used to calculate the amount of data to be recovered; a calculation unit, configured to calculate a recovery concurrency number based on processor usage information, memory usage information, and the amount of data; a generation unit, configured to generate a recovery query statement based on the recovery concurrency number, wherein the recovery query statement is used to query table data corresponding to the field to be recovered from a relational database to obtain a data query result; and a recovery unit, configured to import the data query result into various data partitions of the data computing cluster based on the mapping relationship between the relational database and the data computing cluster.

[0015] Optionally, the data recovery device further includes: a first receiving module, configured to receive a login request initiated by the front end, wherein the login request carries at least: the front end address, user account, and account password; and a second receiving module, configured to receive the recovery request after verifying that the user account and the account password are correct, wherein the recovery request carries at least the recovery selection information transmitted by the front end, wherein the information dimensions of the recovery selection information include: file identifier, table identifier of at least one table to be recovered, data source information, cluster information, data partition information, data full amount, and data volume information.

[0016] Optionally, the parsing unit includes: a first parsing module, used to parse the first set of information dimensions in the recovery request to obtain the data source, file identifier, data volume, and data partition of the data to be recovered, wherein the first set of information dimensions includes: file identifier, data source information, data partition information, and data volume information; a first establishing module, used to establish a connection with the relational database based on the data source; and a first generating module, used to generate the recovery data statement associated with the relational database according to the file identifier, the data volume, and the data partition.

[0017] Optionally, the parsing unit further includes: a second parsing module, used to parse the second set of information dimensions in the recovery request, wherein the second set of information dimensions includes: table identifier and cluster information; a first acquisition module, used to acquire the table structure information of the table to be recovered from the data computing cluster indicated by the cluster information and the relational database based on the table identifier; a second generation module, used to generate a table mapping template based on the table structure information; and a first mapping module, used to use the table mapping relationship represented by the table mapping template as the mapping relationship between the relational database and the data computing cluster.

[0018] Optionally, the computing unit includes: a second acquisition module, used to acquire the processor utilization rate and idle threads of the processor in the previous time period to obtain the processor usage information; and to acquire the memory occupancy rate and memory size in the previous time period to obtain the memory usage information; a first statistics module, used to count the number of all fields to be recovered; and a first input module, used to input the processor usage information, the memory usage information, the data volume, and the number of all fields to be recovered into the target neural network model to obtain the recovery concurrency number output by the target neural network model.

[0019] Optionally, the generation unit includes a third generation module, configured to generate the recovery query statement for each concurrent recovery data thread based on the recovery concurrency and the recovery data statement.

[0020] Optionally, the recovery unit includes: a third receiving module for receiving a recovery status query request; and a first sending module for responding to the recovery status query request and sending a progress bar of the recovery data to the front end, wherein the progress bar is used to display the data table identifier, the recovery fields, and the table data being recovered at the current moment.

[0021] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the data recovery method described in any one of the above embodiments.

[0022] According to another aspect of the present invention, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the data recovery method described in any one of the preceding embodiments.

[0023] In this invention, a recovery data statement is obtained by parsing the recovery request. The recovery data statement is used to calculate the amount of data to be recovered, and to calculate the recovery concurrency based on processor usage information, memory usage information, and the amount of data. Based on the recovery concurrency, a recovery query statement is generated. The recovery query statement is used to query the table data corresponding to the field to be recovered from the relational database to obtain the data query result. Based on the mapping relationship between the relational database and the data computing cluster, the data query result is imported into each data partition of the data computing cluster.

[0024] This invention proposes a flexible data recovery method. It calculates the concurrency based on the performance of the relational database server and the amount of data to be recovered. The backend server generates recovery data statements based on the data elements to be recovered. By executing the recovery query statements in the relational database, the retrieved data is automatically imported into the data computing cluster, eliminating the need for two data copies and significantly reducing data recovery time. Furthermore, it allows for flexible import of different partitions of the recovery data, supporting large-scale data recovery scenarios and achieving flexible, real-time data recovery functionality. This solves the technical problems of long recovery times and lack of flexibility in related technologies.

[0025] In this invention, the method of automatically importing data into the data computing cluster replaces the solution of directly restoring data from the production Hadoop cluster through a relay environment.

[0026] In this invention, partitions with different recovery data can be flexibly imported, and large-scale data recovery scenarios are supported, ultimately achieving flexible and real-time data recovery functionality. Attached Figure Description

[0027] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0028] Figure 1 This is a schematic diagram of an optional data recovery method according to an embodiment of the present invention;

[0029] Figure 2 This is an architecture diagram of an optional data recovery method according to an embodiment of the present invention;

[0030] Figure 3 This is a schematic diagram of an optional data recovery device according to an embodiment of the present invention;

[0031] Figure 4 This is a hardware structure block diagram of an electronic device (or mobile device) according to an embodiment of the present invention for a method of recovering data. Detailed Implementation

[0032] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0034] To facilitate understanding of the present invention by those skilled in the art, some terms or nouns involved in the various embodiments of the present invention are explained below:

[0035] HDFS, short for Hadoop Distributed File System, is a distributed file system used to store massive amounts of data.

[0036] Hadoop is a distributed system infrastructure used to solve the problems of storing and analyzing massive amounts of data.

[0037] A Hadoop cluster, referred to as a data computing cluster in this application, is used to store and analyze large amounts of unstructured data in a distributed computing environment.

[0038] SQL, short for Structured Query Language, is a database query and programming language used to access, query, update, and manage relational database systems.

[0039] It should be noted that the data recovery method and apparatus in this disclosure can be used in the field of big data. In the case of recovering massive amounts of data, it can also be used in any field other than big data. In the case of recovering massive amounts of data, this disclosure does not limit the application field of the data recovery method and apparatus.

[0040] It should be noted that all information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are information and data authorized by the user or fully authorized by all parties. For example, this system has an interface with relevant users or organizations. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain the relevant information after receiving consent information from the aforementioned user or organization.

[0041] This invention can be applied to various data recovery systems / devices / products. It allows for flexible configuration of data recovery requirements through the front end, free configuration of the file name or table name of the data to be recovered, and flexible specification of the data recovery partition. The partition can be a single partition or a range partition, and the amount of data to be recovered can be specified. There is no need to submit a data recovery application to a designated person. After submitting the application through the front end, the data can be recovered to the target cluster in real time (this application uses the target Hadoop cluster as an example), thereby realizing flexible configuration of data recovery.

[0042] The present invention will now be described in detail with reference to various embodiments.

[0043] Example 1

[0044] According to an embodiment of the present invention, a method for recovering data is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0045] Figure 1 This is a schematic diagram of an optional data recovery method according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0046] Step S101: Parse the recovery request to obtain the recovery data statement, wherein the recovery data statement is used to calculate the amount of data to be recovered;

[0047] Step S102: Calculate the number of concurrent connections to be restored based on processor usage information, memory usage information, and data volume;

[0048] Step S103: Generate a recovery query statement based on the number of concurrent recovery attempts. The recovery query statement is used to query the table data corresponding to the field to be recovered from the relational database to obtain the data query results.

[0049] Step S104: Based on the mapping relationship between the relational database and the data computing cluster, the data query results are imported into each data partition of the data computing cluster.

[0050] Through the above steps, the recovery request is first parsed to obtain the recovery data statement. This statement calculates the amount of data to be recovered. Then, based on processor usage, memory usage, and the amount of data, the recovery concurrency is calculated. Next, a recovery query statement is generated based on the recovery concurrency. This query statement retrieves table data corresponding to the fields to be recovered from the relational database, obtaining the query results. Finally, based on the mapping relationship between the relational database and the data computing cluster, the query results are imported into various data partitions of the data computing cluster. This embodiment provides a flexible data recovery method. The recovery concurrency is calculated based on the performance of the relational database server and the amount of data to be recovered. The backend server generates the recovery data statement based on the concurrency. By executing the recovery query statement in the relational database, the retrieved data is automatically imported into the data computing cluster, eliminating the need for two data copies and significantly reducing data recovery time. It also allows for flexible import into different recovery data partitions, supporting large-scale data recovery scenarios and achieving flexible, real-time data recovery functionality. This solves the technical problems of long recovery times and lack of flexibility in related technologies.

[0051] The following section will explain in detail each of the above implementation steps.

[0052] Step S101: Parse the recovery request to obtain the recovery data statement, wherein the recovery data statement is used to calculate the amount of data to be recovered.

[0053] The recovery request is initiated by the user through the terminal page of the front-end terminal (including but not limited to: laptop, PC, panel). The user logs in to the terminal page / application / address of the front-end terminal, fills in the source table name or file name of the data to be recovered, the partition field or partition range of the data to be recovered, the amount of data in the table to be recovered, and the target Hadoop cluster. In this embodiment of the invention, there is no limit to the number of data recovery tasks, and multiple data recovery tasks can be configured at the same time. The user clicks the submit button on the front end to send a recovery request to the back end, thereby triggering the background data recovery process.

[0054] It should be noted that before parsing the recovery request and obtaining the recovery data statement, the process also includes: receiving a login request initiated by the frontend, wherein the login request carries at least the following information: frontend address, user account, and password; and, if the user account and password are verified to be correct, receiving a recovery request, wherein the recovery request carries at least the recovery selection information transmitted by the frontend. The recovery selection information includes the following dimensions: file identifier, table identifier of at least one table to be recovered, data source information, cluster information, data partition information, full data volume, and data volume information (e.g., data size, occupied space, etc.).

[0055] The login request includes, but is not limited to: front-end address, user account (account type includes, but is not limited to, numbers, letters, characters, numbers and letters, numbers and characters), static password (i.e., the password set by the user in advance), dynamic password (i.e., the real-time password obtained by the user through SMS or authentication APP), and user name.

[0056] In this embodiment, there is no limit to the number of data items or tables to be restored each time, and multiple tables can be restored simultaneously. If a user submits a restoration request or order through the front-end terminal page, clicking the submit button will trigger the data restoration process on the back-end server.

[0057] In this embodiment of the invention, the step of parsing the recovery request to obtain the recovery data statement includes: parsing the first set of information dimensions in the recovery request to obtain the data source, file identifier, data volume, and data partition of the data to be recovered, wherein the first set of information dimensions includes: file identifier, data source information, data partition information, and data volume information; establishing a connection with a relational database based on the data source; and generating a recovery data statement related to the relational database according to the file identifier, data volume, and data partition.

[0058] It should be noted that the data source stores all information related to establishing database connections. A connection is established with a relational database operating in a similar production environment based on this data source information. The types of relational databases include, but are not limited to, Oracle, DB2, and MySQL. The retrieved data is stored in rows and columns within the database. A link is established between the data to be recovered and the relational database based on the data source, thereby achieving the goal of exporting data from the relational database.

[0059] After establishing a connection with a relational database, data recovery statements can be generated in the relational database based on file or table identifiers, data volume, partitions, and other information.

[0060] The file identifier types mentioned above include, but are not limited to: text, numbers, characters, and letters; the file identifier content includes, but is not limited to: the name of the data to be recovered, the sequence number, and the recovery date. Data partitions include, but are not limited to: random partitions, hash partitions, and range partitions.

[0061] In this embodiment of the invention, after establishing a connection with the relational database based on the data source, the method further includes: parsing the second set of information dimensions in the recovery request, wherein the second set of information dimensions includes: table identifier and cluster information; obtaining the table structure information of the table to be recovered from the data computing cluster indicated by the cluster information and the relational database based on the table identifier; generating a table mapping template based on the table structure information; and using the table mapping relationship represented by the table mapping template as the mapping relationship between the relational database and the data computing cluster.

[0062] It should be noted that for large amounts of data to be recovered, it should be stored in a data table and transmitted and recovered as a recovery table to prevent the loss of individual data during data transmission and recovery. The table identifier type includes, but is not limited to: text, numbers, characters, and letters, and the table identifier content includes, but is not limited to: the data type to be recovered, the name of the table to be recovered, the sequence number, and the recovery date.

[0063] The table structure information carries a large amount of data from the data source. By establishing a mapping relationship between the data computing cluster and the relational database through the table structure information, the purpose of exporting data from the relational database and restoring the data can be achieved.

[0064] Through the above implementation method, a mapping template from a relational database to a data computing cluster (illustrated in this embodiment using a Hadoop cluster) can be generated. Specifically, a table mapping relationship from the relational database to the data computing cluster can be formed based on the table structure information of the data table to be recovered obtained from the relational database and the data computing cluster.

[0065] In this embodiment, when calculating the amount of data to be recovered, the amount of data to be recovered can be calculated in a relational database corresponding to a production-equivalent environment using the data recovery statement.

[0066] Step S102: Calculate the number of concurrent connections to be restored based on processor usage information, memory usage information, and data volume.

[0067] In this embodiment, after calculating the amount of data to be recovered, the calculation can be performed using the amount of data, processor usage information, and memory usage information.

[0068] In this embodiment of the invention, the step of calculating the number of concurrent connections to be restored based on processor usage information, memory usage information, and data volume includes: obtaining the processor utilization rate and the number of idle threads of the processor in the previous time period to obtain processor usage information; obtaining the memory occupancy rate and memory size in the previous time period to obtain memory usage information; counting the number of all fields to be restored; and inputting the processor usage information, memory usage information, data volume, and the number of all fields to be restored into the target neural network model to obtain the number of concurrent connections to be restored output by the target neural network model.

[0069] It should be noted that the number of concurrent recovery threads must be selected appropriately to ensure that the data recovery thread can work normally. If the number of concurrent recovery threads is too large, it will cause the recovery thread to freeze, resulting in the failure of the data task. In this embodiment of the invention, the memory to be used is obtained based on the processor's memory usage rate and memory size. Then, based on the total amount of data to be recovered, combined with processor usage information, memory usage information, and data volume, the neural network model automatically configures the number of concurrent recovery threads to ensure that the recovery task proceeds normally.

[0070] Step S103: Generate a recovery query statement based on the number of concurrent recovery attempts. The recovery query statement is used to query the table data corresponding to the field to be recovered from the relational database to obtain the data query results.

[0071] It should be noted that the steps of generating recovery query statements based on the number of concurrent recovery attempts include: generating recovery query statements for each concurrent data recovery thread based on the number of concurrent recovery attempts and the recovery data statements.

[0072] The recovery query statement includes, but is not limited to: the database table name, key fields of the data to be recovered, the submission time, and the deletion time of the data to be recovered. The data to be recovered can be found in the database based on the recovery query statement.

[0073] Step S104: Based on the mapping relationship between the relational database and the data computing cluster, the data query results are imported into each data partition of the data computing cluster.

[0074] The system concurrently executes recovery query statements in a relational database in a production-equivalent environment. Based on the previously determined mapping relationship, the results queried from the relational database are mapped and written to a file according to the corresponding mapping relationship. After all data is written to the file, it is automatically imported into the target data computing cluster.

[0075] It should be noted that when importing the data query results into the various data partitions of the data computing cluster, the process also includes: receiving recovery status query requests; responding to recovery status query requests and sending a progress bar of the data recovery to the front end. The progress bar is used to display the data table identifier, the fields being recovered, and the table data at the current moment.

[0076] After clicking the data recovery submit button, users can view the current recovery progress bar in real time on the front end. The progress bar displays the data table identifier, the fields to be recovered, and the table data in real time. For example, it shows which table data is being recovered.

[0077] Figure 2 This is an architecture diagram of an optional data recovery method according to an embodiment of the present invention, such as... Figure 2 As shown, the data recovery system includes a front-end page ( Figure 2 (For example, the front-end illustration) and the back-end data recovery page ( Figure 2 (Backstage illustration).

[0078] For the front end, its functions include:

[0079] Fill in the data recovery requirements. Users log in to this front-end address and can flexibly configure the data recovery requirements through the front-end, specifying the table name or file name of the data source table to be recovered. Figure 2 (Illustrated by the target table name or file name), and the data partition field or partition range to be restored. Figure 2 (The text is a mix of Chinese characters and symbols, which appear to be fragments and do not form a coherent sentence or paragraph.)

[0080] The data recovery progress is displayed. After clicking the data recovery submit button, users can view the current data recovery progress bar in real time on the front end. The progress bar shows which table's data is being recovered in real time.

[0081] The backend's functions include:

[0082] Analyze the data recovery requirements. Obtain the template for importing into the target cluster based on the table name or file name of the data to be recovered, and update the partition information in the template according to the partitions passed in from the front end.

[0083] Generate data recovery logic, and query the database for its sub-tables based on the filenames of the data tables to be recovered using SQL statements. Figure 2 See below for templates and SQL illustrations, corresponding to the aforementioned data recovery statements.

[0084] Data is generated into a template. The SQL query is executed in the production peer database, and the results are loaded into the template corresponding to the data table to be restored. If the amount of data to be restored is small, the data is generated and imported using a single thread. If the amount of data to be restored is large, the data is obtained concurrently by multiple threads using an asynchronous thread pool and written to multiple templates.

[0085] Import the data into the target cluster by calling the cluster data import interface to import the template generated in the above steps into the target cluster, thereby achieving the data recovery goal.

[0086] The above implementation provides a flexible data recovery method. It calculates the recovery concurrency based on the relational database server performance and the amount of data to be recovered. The backend server generates recovery data statements based on the concurrency, and the retrieved data is automatically imported into the data computing cluster by executing the recovery query statement in the relational database. This eliminates the need for two data copies, significantly reducing data recovery time and allowing for flexible import of different recovery data partitions. Figure 2(Illustrated by "filling partition"), it supports data recovery scenarios with large data volumes, and realizes flexible and real-time data recovery functions, thereby solving the technical problems of long time consumption and lack of flexibility when performing data recovery in related technologies.

[0087] The invention will now be described in conjunction with another alternative embodiment.

[0088] Example 2

[0089] This embodiment provides a data recovery apparatus, wherein each implementation unit of the data recovery apparatus corresponds to each implementation step in Embodiment 1.

[0090] Figure 3 This is a schematic diagram of an optional data recovery device according to an embodiment of the present invention, such as... Figure 3 As shown, it includes: a parsing unit 31, a calculation unit 32, a generation unit 33, and a recovery unit 34, wherein,

[0091] Parsing unit 31 is used to parse the recovery request and obtain the recovery data statement, wherein the recovery data statement is used to calculate the amount of data to be recovered;

[0092] The calculation unit 32 is used to calculate the number of concurrent connections to be restored based on processor usage information, memory usage information, and data volume;

[0093] The generation unit 33 is used to generate a recovery query statement based on the number of concurrent recovery attempts. The recovery query statement is used to query the table data corresponding to the field to be recovered from the relational database to obtain the data query result.

[0094] Recovery unit 34 is used to import data query results into various data partitions of the data computing cluster based on the mapping relationship between the relational database and the data computing cluster.

[0095] The aforementioned data recovery device can parse the recovery request through the parsing unit 31 to obtain a recovery data statement, wherein the recovery data statement is used to calculate the amount of data to be recovered; the calculation unit 32 calculates the recovery concurrency based on processor usage information, memory usage information, and the amount of data; the generation unit 33 generates a recovery query statement based on the recovery concurrency, wherein the recovery query statement is used to query the table data corresponding to the field to be recovered from the relational database to obtain the data query result; and the recovery unit 34 imports the data query result into each data partition of the data computing cluster based on the mapping relationship between the relational database and the data computing cluster.

[0096] This embodiment provides a flexible data recovery method. The recovery concurrency is calculated based on the performance of the relational database server and the amount of data to be recovered. The backend server generates recovery data statements based on the recovery concurrency. By executing the recovery query statement in the relational database, the queried data is automatically imported into the data computing cluster without the need for two data copies, which greatly reduces the data recovery time. It can also flexibly import different recovery data partitions, supports data recovery scenarios with large data volumes, and realizes flexible and real-time data recovery functions. This solves the technical problems of long time consumption and lack of flexibility in related technologies when performing data recovery.

[0097] Optionally, the data recovery device further includes: a first receiving module for receiving a login request initiated by the front end, wherein the login request carries at least: the front end address, the user account, and the account password; and a second receiving module for receiving a recovery request after verifying that the user account and the account password are correct, wherein the recovery request carries at least recovery selection information transmitted by the front end, wherein the information dimensions of the recovery selection information include: file identifier, table identifier of at least one table to be recovered, data source information, cluster information, data partition information, data full amount, and data volume information.

[0098] Optionally, the parsing unit 31 includes: a first parsing module, used to parse the first set of information dimensions in the recovery request to obtain the data source, file identifier, data volume, and data partition of the data to be recovered, wherein the first set of information dimensions includes: file identifier, data source information, data partition information, and data volume information; a first establishing module, used to establish a connection with the relational database based on the data source; and a first generating module, used to generate recovery data statements for the relational database based on the file identifier, data volume, and data partition.

[0099] Optionally, the parsing unit 31 further includes: a second parsing module, used to parse the second information dimension set in the recovery request, wherein the second information dimension set includes: table identifier and cluster information; a first acquisition module, used to acquire the table structure information of the table to be recovered from the data computing cluster and relational database indicated by the cluster information based on the table identifier; a second generation module, used to generate a table mapping template based on the table structure information; and a first mapping module, used to use the table mapping relationship represented by the table mapping template as the mapping relationship between the relational database and the data computing cluster.

[0100] Optionally, the computing unit 32 includes: a second acquisition module, used to acquire the processor utilization rate and idle threads of the processor in the previous time period to obtain processor usage information; acquire the memory occupancy rate and memory size in the previous time period to obtain memory usage information; a first statistics module, used to count the number of all fields to be recovered; and a first input module, used to input the processor usage information, memory usage information, data volume and the number of all fields to be recovered into the target neural network model to obtain the recovery concurrency output by the target neural network model.

[0101] Optionally, the generation unit 33 includes: a third generation module, used to generate a recovery query statement for each concurrent recovery data thread based on the number of concurrent recovery threads and the recovery data statement.

[0102] Optionally, the recovery unit 34 includes: a third receiving module for receiving a recovery status query request; and a first sending module for responding to the recovery status query request and sending a progress bar of the recovery data to the front end, wherein the progress bar is used to display the data table identifier, the recovery fields, and the table data being recovered at the current moment.

[0103] The aforementioned data recovery device may also include a processor and a memory. The parsing unit 31, calculation unit 32, generation unit 33, recovery unit 34, etc., are all stored in the memory as program units, and the processor executes the aforementioned program units stored in the memory to realize the corresponding functions.

[0104] The aforementioned processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured to enable flexible data recovery. The recovery concurrency is calculated based on the relational database server's performance and the amount of data to be recovered. The backend server generates recovery data statements based on the concurrency, and the retrieved data is automatically imported into the data processing cluster by executing the recovery query statements in the relational database. This eliminates the need for two data copies, significantly reducing data recovery time. Furthermore, it can flexibly import different recovery data partitions, supporting large-scale data recovery scenarios and achieving flexible, real-time data recovery functionality.

[0105] The aforementioned memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0106] According to another aspect of the present invention, an electronic device is also provided, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform a method for recovering data of any of the above via executing the executable instructions.

[0107] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to perform the data recovery method described above.

[0108] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program with the following method steps: parsing a recovery request to obtain a recovery data statement, wherein the recovery data statement is used to calculate the amount of data to be recovered; calculating the recovery concurrency based on processor usage information, memory usage information, and the amount of data; generating a recovery query statement based on the recovery concurrency, wherein the recovery query statement is used to query table data corresponding to the field to be recovered from a relational database to obtain data query results; and importing the data query results into various data partitions of the data computing cluster based on the mapping relationship between the relational database and the data computing cluster.

[0109] Figure 4 This is a hardware structure block diagram of an electronic device (or mobile device) according to an embodiment of the present invention for a method of recovering data. Figure 4 As shown, the electronic device may include one or more processors 402 (shown as 402a, 402b, ..., 402n in the figure) 402 (processor 402 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 404 for storing data. In addition, it may include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a keyboard, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 4 The structure shown is for illustrative purposes only and does not limit the structure of the electronic device described above. For example, the electronic device may also include components that are more... Figure 4 The more or fewer components shown, or having the same Figure 4 The different configurations shown.

[0110] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0111] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0112] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0113] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0114] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0115] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0116] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for recovering data, characterized in that, include: Parse the recovery request to obtain a recovery data statement, wherein the recovery data statement is used to calculate the amount of data to be recovered; Based on processor usage information, memory usage information, and the amount of data, the recovery concurrency count is calculated, including: obtaining the processor utilization rate and idle threads of the processor in the previous time period to obtain the processor usage information; obtaining the memory occupancy rate and memory size in the previous time period to obtain the memory usage information; counting the number of all fields to be recovered; and inputting the processor usage information, the memory usage information, the amount of data, and the number of all fields to be recovered into the target neural network model to obtain the recovery concurrency count output by the target neural network model. Based on the number of concurrent recovery attempts, a recovery query statement is generated. The recovery query statement is used to query the table data corresponding to the field to be recovered from the relational database to obtain the data query result. The recovery query statement includes: the database table name, the key fields of the data to be recovered, the submission time, and the deletion time of the data to be recovered. Based on the mapping relationship between the relational database and the data computing cluster, the data query results are imported into each data partition of the data computing cluster.

2. The method according to claim 1, characterized in that, Before parsing the recovery request and obtaining the recovery data statement, the process also includes: Receive a login request initiated by the front end, wherein the login request carries at least: front end address, user account, and account password; If the user account and the account password are verified to be correct, the recovery request is received. The recovery request carries at least the recovery selection information transmitted by the front end. The information dimensions of the recovery selection information include: file identifier, table identifier of at least one table to be recovered, data source information, cluster information, data partition information, data full amount, and data volume information.

3. The method according to claim 2, characterized in that, The steps to parse the recovery request and obtain the recovery data statement include: The first set of information dimensions in the recovery request is parsed to obtain the data source, file identifier, data volume, and data partition of the data to be recovered. The first set of information dimensions includes: file identifier, data source information, data partition information, and data volume information. Based on the data source, a connection is established with the relational database; Based on the file identifier, the data volume, and the data partition, generate the recovery data statement associated with the relational database.

4. The method according to claim 3, characterized in that, After establishing a connection with the relational database based on the data source, the process further includes: The second set of information dimensions in the recovery request is parsed, wherein the second set of information dimensions includes: table identifier and cluster information; Based on the table identifier, obtain the table structure information of the table to be recovered from the data computing cluster indicated by the cluster information and the relational database; Based on the table structure information, a table mapping template is generated; The table mapping relationship represented by the table mapping template is used as the mapping relationship between the relational database and the data computing cluster.

5. The method according to claim 1, characterized in that, The steps for generating a recovery query statement based on the recovery concurrency number include: Based on the number of concurrent recovery attempts and the recovery data statements, the recovery query statements for each concurrent recovery data thread are generated.

6. The method according to claim 1, characterized in that, When importing the data query results into the various data partitions of the data computing cluster, the process also includes: Receive recovery status query request; In response to the recovery status query request, a progress bar for data recovery is sent to the front end. The progress bar is used to display the data table identifier, the fields being recovered, and the table data at the current moment.

7. A device for recovering data, characterized in that, include: The parsing unit is used to parse the recovery request and obtain the recovery data statement, wherein the recovery data statement is used to calculate the amount of data to be recovered; A calculation unit is used to calculate the number of concurrent connections to be restored based on processor usage information, memory usage information, and the amount of data. The calculation unit includes: a second acquisition module, used to acquire the processor utilization rate and idle threads of the processor in the previous time period to obtain the processor usage information; and to acquire the memory occupancy rate and memory size in the previous time period to obtain the memory usage information; a first statistics module, used to count the number of all fields to be restored; and a first input module, used to input the processor usage information, the memory usage information, the amount of data, and the number of all fields to be restored into a target neural network model to obtain the number of concurrent connections to be restored output by the target neural network model. The generation unit is used to generate a recovery query statement based on the recovery concurrency. The recovery query statement is used to query the table data corresponding to the field to be recovered from the relational database to obtain the data query result. The recovery query statement includes: the database table name, the key fields of the data to be recovered, the submission time, and the deletion time of the data to be recovered. The recovery unit is used to import the data query results into each data partition of the data computing cluster based on the mapping relationship between the relational database and the data computing cluster.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the data recovery method according to any one of claims 1 to 6.

9. An electronic device, characterized in that, It includes one or more processors and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the data recovery method according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • Concurrency elastic control method based on distributed database

    CN111782396A

  • Data recovery method and device

    CN113448777A