Data file recovery method and device, computer equipment and storage medium

Abstract

The invention discloses a data file recovery method and device, computer equipment and a storage medium, and belongs to the technical field of big data and the field of internet medical treatment. According to the method, key classes related to data writing in the Spark source codes are recognized, clearing path codes and / or deleting path codes in the key classes are modified into moving path codes, new Spark source codes are generated, the targets of the moving path codes are HDFS file recycle bins, data recovery tools are constructed, data files generated in the Spark task execution process are obtained, and the data files are stored in the Spark source codes. And the data file is stored in an HDFS file recycle bin, an HDFS partition path is inquired, and the data file is restored by using a data recovery tool based on the HDFS partition path. The invention further relates to the technical field of block chains, and the data file can be stored on the block chain node. According to the method, a data backup and recovery mechanism is provided, the reliability and fault tolerance of data processing are improved, and it is ensured that the data can still be recovered even under the condition that task execution fails or errors occur.

Description

Technical Field

[0001] The present application belongs to the field of big data technology and Internet medical care, and specifically relates to a data file recovery method, device, computer equipment and storage medium. Background Art

[0002] Apache Spark is an open source distributed big data memory computing engine. Its internal implementation uses in-memory caching and optimized query execution. It can quickly analyze and query data of any size. It has been widely used in big data application scenarios such as offline batch computing, data ETL processing, data mining, and machine learning. It has become one of the benchmark products in the big data field.

[0003] Spark SQL is a standard computer language for accessing and processing databases in Spark tasks for data operations in large data distributed clusters. It is the most widely used module in Spark. Users can process large-scale data through simple SQL statements.

[0004] A typical application scenario of Spark SQL is to run offline batch calculations every day, read the incremental data of the previous day, and finally insert it into the target partition path after a series of conversion and calculation operations, so as to finally update the data of the entire table. For example, the medical property insurance system needs to process medical insurance claim data, which contains daily claim records. The system needs to process the daily incremental data to update the insurance claim data table. One of the most important statements is insert overwrite, which is an SQL statement used to overwrite new data to an existing table or a specified partition of a table to update or replace data.

[0005] In the native Spark of the open source community, the internal implementation logic of the insert overwrite statement is: first clear the target insertion path, then submit the specific computing task, and after each distributed computing node completes the operation of each task, modify the final generated data file and move it to the target path to complete the operation of the entire task. This implementation logic has a relatively important hidden danger: once an error occurs during the computing task and causes the overall task to fail, the data in the original partition path may have been cleared and cannot be retrieved, resulting in the risk of data loss. Summary of the invention

[0006] The purpose of the embodiments of the present application is to propose a data file recovery method, apparatus, computer equipment and storage medium to solve the technical problem that when the existing medical property insurance system performs data update or replacement tasks, the target insertion path is first cleared and then the specific calculation task is submitted. When the task fails, the data in the original partition path may have been cleared and cannot be retrieved, resulting in the risk of data loss.

[0007] In order to solve the above technical problems, the present application embodiment provides a data file recovery method, which adopts the following technical solution:

[0008] A data file recovery method, comprising:

[0009] Obtain the Spark source code of the project, and identify key classes related to data writing in the Spark source code;

[0010] Modify the clear path code and / or delete path code in the key class into a move path code to generate a new Spark source code, wherein the target of the move path code is the HDFS file recycle bin;

[0011] Compile the new Spark source code to obtain a Spark binary distribution package;

[0012] Build a data recovery tool, and install the Spark binary distribution package and the data recovery tool into the production environment of the project;

[0013] Monitor the Spark task, obtain the data files generated during the execution of the Spark task, and store the data files in the HDFS file recycle bin;

[0014] Obtain a data file recycling instruction, query the HDFS partition path where the data file is stored in the HDFS file recycling bin, and restore the data file using the data recovery tool based on the HDFS partition path.

[0015] Furthermore, the key classes related to data writing include InsertIntoHadoopFsRelationCommand class, HadoopMapReduceCommitProtocol class and InsertIntoHiveTable class.

[0016] Furthermore, the clearing path code and / or deleting path code in the key class is modified into a moving path code to generate a new Spark source code, specifically including:

[0017] Identify the clear path code in the InsertIntoHadoopFsRelationCommand class;

[0018] Respectively identify the deletion path codes in the HadoopMapReduceCommitProtocol class and the InsertIntoHiveTable class;

[0019] The clear path code and / or the delete path code are modified into a move path code for moving the file to the HDFS file recycle bin.

[0020] Furthermore, before obtaining the Spark source code of the project and identifying key classes related to data writing in the Spark source code, the method further includes:

[0021] Obtain a Hadoop distribution package, and deploy the operating environment of the HDFS file recycle bin based on the Hadoop distribution package;

[0022] Obtain a Hadoop dependency file from a Hadoop distribution package, and build a dependency between the production environment of the project and the HDFS file recycle bin based on the Hadoop dependency file.

[0023] Further, the monitoring of the Spark task, obtaining the data files generated during the execution of the Spark task, and storing the data files in the HDFS file recycle bin specifically includes:

[0024] Monitor the Spark task and obtain the data files generated during the execution of the Spark task;

[0025] Define a first source file path and a first target file path, wherein the first source file path is defined as an original file path, the first target file path is defined as an HDFS partition path in the HDFS file recycle bin, and the original file path is a path of a data file generated during the execution of the Spark task;

[0026] The Java API in the Hadoop distribution package is called, and the data file is moved to the HDFS file recycle bin based on the original file path to the HDFS partition path using the Java API.

[0027] Further, obtaining a data file recycling instruction, querying the HDFS partition path where the data file is stored in the HDFS file recycling bin, and restoring the data file using the data recovery tool based on the HDFS partition path, specifically includes:

[0028] Obtain a data file recycling instruction, and query the HDFS partition path where the data file is stored in the HDFS file recycling bin;

[0029] Define a second source file path and a second target file path, wherein the second source file path is an HDFS partition path, and the second target file path is the original file path;

[0030] The Java API is called to move the data file to the production environment of the project based on the HDFS partition path to the original file path.

[0031] In order to solve the above technical problems, the embodiment of the present application also provides a data file recovery device, which adopts the following technical solution:

[0032] A data file recovery device, comprising:

[0033] A key class identification module, used to obtain the Spark source code of the project and identify key classes related to data writing in the Spark source code;

[0034] A code replacement module is used to modify the clearing path code and / or the deleting path code in the key class into a moving path code to generate a new Spark source code, wherein the target of the moving path code is an HDFS file recycle bin;

[0035] A code compilation module, used to compile the new Spark source code to obtain a Spark binary distribution package;

[0036] An environment deployment module, using a built data recovery tool, installs the Spark binary distribution package and the data recovery tool into the production environment of the project;

[0037] A task monitoring module is used to monitor Spark tasks, obtain data files generated during the execution of the Spark tasks, and store the data files in the HDFS file recycle bin;

[0038] The data recovery module uses the acquisition data file recovery instruction to query the HDFS partition path where the data file is stored in the HDFS file recycling bin, and restores the data file using the data recovery tool based on the HDFS partition path.

[0039] Furthermore, the key classes related to data writing include InsertIntoHadoopFsRelationCommand class, HadoopMapReduceCommitProtocol class and InsertIntoHiveTable class, and the clearing path code and / or deleting path code in the key classes are modified into moving path codes to generate new Spark source code, specifically including:

[0040] Identify the clear path code in the InsertIntoHadoopFsRelationCommand class;

[0041] Respectively identify the deletion path codes in the HadoopMapReduceCommitProtocol class and the InsertIntoHiveTable class;

[0042] The clear path code and / or the delete path code are modified into a move path code for moving the file to the HDFS file recycle bin.

[0043] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0044] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of any of the above-mentioned data file recovery methods when executing the computer-readable instructions.

[0045] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0046] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of any one of the above-mentioned data file recovery methods.

[0047] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0048] The present application discloses a data file recovery method, device, computer equipment and storage medium, which belongs to the field of big data technology and Internet medical care. The present application obtains the Spark source code of the project, identifies the key classes related to data writing in the Spark source code, modifies the clear path code and / or delete path code in the key class into a mobile path code, generates a new Spark source code, wherein the target of the mobile path code is the HDFS file recycle bin, compiles the new Spark source code, obtains the Spark binary distribution package, builds a data recovery tool, installs the Spark binary distribution package and the data recovery tool into the production environment of the project, monitors the Spark task, obtains the data files generated during the execution of the Spark task, and stores the data files in the HDFS file recycle bin, obtains the data file recovery instruction, queries the HDFS partition path where the data files are stored in the HDFS file recycle bin, and restores the data files based on the HDFS partition path using the data recovery tool. The present application modifies the Spark source code so that the files of the original partition path will be moved to the recycle bin, thereby providing a data backup and recovery mechanism, improving the reliability and fault tolerance of data processing in the medical property insurance system, and ensuring that the data can still be recovered even if the task fails or an error occurs. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 An exemplary system architecture diagram is shown in which the present application can be applied;

[0051] Figure 2 A flow chart showing an embodiment of a data file recovery method according to the present application is shown;

[0052] Figure 3 A schematic diagram showing the structure of an embodiment of a data file recovery device according to the present application is shown;

[0053] Figure 4 A schematic structural diagram of an embodiment of a computer device according to the present application is shown. DETAILED DESCRIPTION

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0055] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0056] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0057] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0058] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0059] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0060] Server 105 can be a server that provides various services, such as a background server that provides support for pages displayed on terminal devices 101, 102, and 103. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0061] It should be noted that the data file recovery method provided in the embodiment of the present application is generally executed by a server, and accordingly, the data file recovery device is generally set in the server.

[0062] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0063] Continue to refer Figure 2 , shows a flow chart of an embodiment of the data file recovery method according to the present application. The embodiment of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0064] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0065] The medical property insurance system needs to process medical insurance claim data, which includes daily claim records. The system needs to process daily incremental data to update the insurance claim data table. At present, the data table update and replacement of the medical property insurance system is mainly based on the insert overwrite statement. The internal implementation logic of the insert overwrite statement is: first clear the target insertion path, then submit the specific calculation task, and after each distributed computing node completes the calculation of each task, change the final generated data file, move it to the target path, and complete the operation of the entire task. This implementation logic has a relatively important hidden danger. Once an error occurs during the calculation task and causes the overall task to fail, the data in the original partition path may have been cleared and cannot be retrieved, resulting in the risk of data loss.

[0066] In order to solve the above technical problems, the present application discloses a data file recovery method, device, computer equipment and storage medium, which belong to the field of big data technology and Internet medical care. The present application modifies the Spark source code so that the files in the original partition path will be moved to the recycle bin, thereby providing a data backup and recovery mechanism, improving the reliability and fault tolerance of data processing, and ensuring that data can still be recovered even in the event of task failure or error.

[0067] The data file recovery method comprises the following steps:

[0068] S201, obtaining the Spark source code of the project, and identifying key classes related to data writing in the Spark source code.

[0069] In this embodiment, the server needs to obtain the Spark source code of the project in advance, and analyze the source code to identify key classes related to data writing. Specifically, the server can determine the classes related to data writing by checking the file writing operations in the source code, such as by keyword searching or checking the source code directory structure.

[0070] Furthermore, before obtaining the Spark source code of the project and identifying the key classes related to data writing in the Spark source code, the following steps are also included:

[0071] Obtain the Hadoop distribution package and deploy the operating environment of the HDFS file recycle bin based on the Hadoop distribution package;

[0072] Get Hadoop dependency files from the Hadoop distribution package, and build dependencies between the project's production environment and the HDFS file recycle bin based on the Hadoop dependency files.

[0073] Hadoop distribution package refers to the compilation result of Hadoop software, which includes Hadoop binary files, configuration files, dependency libraries and other related resources. Hadoop distribution package can be deployed to each node in the cluster to execute Hadoop tasks and manage Hadoop cluster.

[0074] In this embodiment, the server obtains the Hadoop distribution package in advance, deploys the operating environment of the HDFS file recycle bin based on the Hadoop distribution package, obtains Hadoop dependency files from the Hadoop distribution package, and builds the dependency between the project's production environment and the HDFS file recycle bin based on the Hadoop dependency files.

[0075] In the above embodiment, the role of the Hadoop distribution package is to provide the operating environment and related tools and libraries of the Hadoop distributed file system HDFS. The existence of the Hadoop distribution package is to ensure that Spark can interact with HDFS when processing data files and use the features of HDFS, such as file operations and path management. By using the appropriate version of the Hadoop distribution package, the compatibility between Spark and HDFS and the correct file operation behavior can be ensured.

[0076] S202, modifying the clearing path code and / or the deleting path code in the key class into the moving path code, and generating a new Spark source code, wherein the target of the moving path code is the HDFS file recycle bin.

[0077] In this embodiment, the server identifies and modifies the clear path code and / or deletion path code in the key class to a mobile path code, modifies the original clear path code and / or deletion path code to a mobile path code, and moves the data file to the HDFS file recycle bin. In this way, the data file can be retained in the HDFS file recycle bin to avoid permanent data loss.

[0078] Furthermore, key classes related to data writing include InsertIntoHadoopFsRelationCommand class, HadoopMapReduceCommitProtocol class, and InsertIntoHiveTable class.

[0079] Furthermore, the clearing path code and / or deleting path code in the key class is modified into moving path code to generate a new Spark source code, which specifically includes:

[0080] Identify the clear path code in the InsertIntoHadoopFsRelationCommand class;

[0081] Identify the delete path codes in the HadoopMapReduceCommitProtocol class and the InsertIntoHiveTable class respectively;

[0082] The empty path code and / or delete path code are modified to a move path code for moving files to the HDFS file recycle bin.

[0083] In this embodiment, the InsertIntoHadoopFsRelationCommand class is responsible for processing the external table to perform the specified partition insertion operation. It is necessary to change the operation of clearing the target partition file path in the original insertoverwrite statement to the operation of moving the file to the HDFS file recycle bin, that is, replace the code for clearing the target partition file path in the InsertIntoHadoopFsRelationCommand class with the new code for moving the file to the HDFS file recycle bin using the HDFS file move operation.

[0084] The HadoopMapReduceCommitProtocol class is responsible for handling the final commit operation when dynamically writing to the partition table. The operation of deleting the partition path needs to be modified to move the file to the HDFS file recycle bin. That is, replace the code for deleting the partition path in the HadoopMapReduceCommitProtocol class with the new code that uses the HDFS file move operation to move the file to the HDFS file recycle bin.

[0085] The InsertIntoHiveTable class is responsible for processing the specified partition writing operation of the internal table, or when the configuration parameter spark.sql.hive.convertInsertingPartitionedTable=false is involved, the operation of deleting the partition path needs to be changed to the operation of moving the file to the HDFS file recycle bin, that is, the new code that uses the HDFS file move operation to move the file to the HDFS file recycle bin needs to be replaced.

[0086] In the above embodiment, at the corresponding position of each class, the deletion operation is replaced by the operation of moving the file to the HDFS file recycle bin as required. Through the modification of the above code, during the task execution process, the file of the original partition path will be moved to the HDFS file recycle bin, avoiding the direct emptying of the target insertion path, thereby providing a data backup and recovery mechanism.

[0087] S203, compile the new Spark source code to obtain a Spark binary distribution package.

[0088] In this embodiment, a compilation tool is used to compile the new Spark source code to obtain a Spark binary distribution package. Specifically, the server needs to use a compilation tool to compile the modified Spark source code to generate an executable Spark binary distribution package. The compilation process involves the installation and configuration of dependent libraries and the execution of compilation commands.

[0089] S204, build a data recovery tool, and install the Spark binary distribution package and the data recovery tool into the production environment of the project.

[0090] In this embodiment, the server builds a data recovery tool and installs the Spark binary distribution package and the data recovery tool into the production environment of the project. Specifically, the server needs to pre-build a data recovery tool that can restore data files according to the partition path of the data files in HDFS. At the same time, the server also needs to install the modified Spark binary distribution package and the data recovery tool into the production environment of the project so that they can be used directly when needed later.

[0091] In one embodiment of the present application, a new Java project can be created in the Java programming language to implement a data recovery tool. Specifically, in the new Java project, the Java API dependency of Hadoop is introduced to interact with Hadoop, and the functions of connecting to Hadoop and interacting with the HDFS file system are implemented, including operations such as obtaining a file list and moving files, and the function of querying the data file recycle bin is implemented, which can be achieved by traversing the recycle bin directory or using Hadoop's API to query the file list, and the function of moving the data file from the HDFS file recycle bin back to the original partition path is implemented. The above Java project is packaged to implement a data recovery tool.

[0092] S205, monitor the Spark task, obtain the data files generated during the execution of the Spark task, and store the data files in the HDFS file recycle bin.

[0093] In this embodiment, the server monitors the Spark task, obtains the data files generated during the execution of the Spark task, and stores the data files in the HDFS file recycle bin. Specifically, during the execution of the Spark task, the server needs to monitor the generation of data files, and can obtain data files by listening to file system events or regularly checking the file system. Once a data file is detected, it needs to be moved to the HDFS file recycle bin for storage.

[0094] Furthermore, the Spark task is monitored, the data files generated during the execution of the Spark task are obtained, and the data files are stored in the HDFS file recycle bin, including:

[0095] Monitor Spark tasks and obtain data files generated during the execution of Spark tasks;

[0096] Define a first source file path and a first target file path, wherein the first source file path is defined as an original file path, the first target file path is defined as an HDFS partition path in an HDFS file recycle bin, and the original file path is a path of a data file generated during the execution of a Spark task;

[0097] The Java API in the Hadoop distribution package is called, and the data file is moved to the HDFS file recycle bin based on the original file path to the HDFS partition path using the Java API.

[0098] In this embodiment, the server can obtain relevant information of the Spark task being executed through Spark's monitoring mechanism or preset related trigger points. During the task execution process, the path information of the data file generated by the task can be obtained through Spark's API or event listener.

[0099] For each acquired data file path, you can use Hadoop's Java API to perform file move operations, moving the data file from the original path to the HDFS file recycle bin. Here, you can use Hadoop's FileSystem class and Path class to implement file move operations. Specifically, get Hadoop's configuration object and HDFS instance, define the source file path and target file path, set the source file path to the data file path generated by the task, set the target file path to the path of the HDFS file recycle bin, that is, the HDFS partition path, and call the file system object's rename(sourcePath,targetPath) method to perform file move operations, moving the data file to the HDFS file recycle bin.

[0100] In the above embodiment, the Java API in the Hadoop distribution package is called, and the data file is moved to the HDFS file recycle bin based on the original file path to the HDFS partition path using the Java API, so as to realize data backup and improve the reliability and fault tolerance of data processing.

[0101] S206, obtaining a data file recycling instruction, querying the HDFS partition path where the data file is stored in the HDFS file recycling bin, and restoring the data file using a data recovery tool based on the HDFS partition path.

[0102] In this embodiment, a data file recycling instruction is obtained, the HDFS partition path where the data file is stored in the HDFS file recycling bin is queried, and the data file is restored using a data recovery tool based on the HDFS partition path. Specifically, a data file recycling instruction needs to be obtained, and the location information of the stored data file can be obtained by querying the HDFS partition path in the HDFS file recycling bin, and then the data recovery tool is used to restore the data file based on the queried path information.

[0103] Furthermore, a data file recycling instruction is obtained, the HDFS partition path where the data file is stored in the HDFS file recycling bin is queried, and the data file is restored using a data recovery tool based on the HDFS partition path, specifically including:

[0104] Get the data file recycling instruction and query the HDFS partition path where the data file is stored in the HDFS file recycling bin;

[0105] A second source file path and a second target file path are defined, wherein the second source file path is an HDFS partition path, and the second target file path is an original file path.

[0106] Call the Java API to move data files to the project's production environment based on the HDFS partition path to the original file path.

[0107] In this embodiment, the server uses the interface or command line tool provided by the data recovery tool to query the information of the data files in the HDFS file recycle bin based on the HDFS partition path, including the storage path, file name, etc. For the storage path of each data file, the Hadoop Java API can be used to perform the file move operation to move the data file from the HDFS file recycle bin back to the original partition path. Specifically, the Hadoop configuration object and HDFS instance are obtained, the source file path and the target file path are redefined, the source file path is set to the path of the data file in the HDFS file recycle bin, that is, the HDFS partition path, the target file path is set to the original partition path, that is, the data file path generated by the task, and the rename (sourcePath, targetPath) method of the file system object is called to perform the file move operation, and the data file is moved back to the original partition path, so as to realize the restoration of the data file, realize the recovery of the data, and improve the reliability and fault tolerance of data processing.

[0108] In a specific embodiment of the present application, a medical property insurance system uses Spark for data processing and analysis. The processed data includes daily medical claim records. The system needs to process daily incremental data to update the insurance claim data table. For the medical property insurance system, its Spark source code can be modified so that the files in the original partition path will be moved to the recycle bin, thereby providing a data backup and recovery mechanism and improving the reliability and fault tolerance of data processing.

[0109] In the above embodiment, the present application is a data file recovery method, which belongs to the field of big data technology and the field of Internet medical care. The present application obtains the Spark source code of the project, identifies the key classes related to data writing in the Spark source code, modifies the clear path code and / or delete path code in the key class into a mobile path code, generates a new Spark source code, wherein the target of the mobile path code is the HDFS file recycle bin, compiles the new Spark source code, obtains the Spark binary distribution package, builds a data recovery tool, installs the Spark binary distribution package and the data recovery tool into the production environment of the project, monitors the Spark task, obtains the data files generated during the execution of the Spark task, and stores the data files in the HDFS file recycle bin, obtains the data file recovery instruction, queries the HDFS partition path where the data file is stored in the HDFS file recycle bin, and restores the data file based on the HDFS partition path using the data recovery tool. The present application modifies the Spark source code so that the files of the original partition path will be moved to the recycle bin, thereby providing a data backup and recovery mechanism, improving the reliability and fault tolerance of data processing, and ensuring that the data can still be recovered even if the task fails or an error occurs.

[0110] In this embodiment, the electronic device (eg, Figure 1 The server shown in the figure) can receive instructions or obtain data through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0111] It should be emphasized that in order to further ensure the privacy and security of the above data files, the above data files can also be stored in a node of a blockchain.

[0112] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0113] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the computer-readable instructions are executed, they can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0114] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0115] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a data file recovery device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0116] like Figure 3 As shown, the data file recovery device 300 described in this embodiment includes:

[0117] A key class identification module 301 is used to obtain the Spark source code of the project and identify key classes related to data writing in the Spark source code;

[0118] The code replacement module 302 modifies the clearing path code and / or the deleting path code in the key class into the moving path code to generate a new Spark source code, wherein the target of the moving path code is the HDFS file recycle bin;

[0119] A code compiling module 303 is used to compile the new Spark source code to obtain a Spark binary distribution package;

[0120] The environment deployment module 304 uses the built data recovery tool to install the Spark binary distribution package and the data recovery tool into the production environment of the project;

[0121] The task monitoring module 305 is used to monitor the Spark task, obtain the data files generated during the execution of the Spark task, and store the data files in the HDFS file recycle bin;

[0122] The data recovery module 306 obtains the data file recovery instruction, queries the HDFS partition path where the data file is stored in the HDFS file recovery bin, and restores the data file using a data recovery tool based on the HDFS partition path.

[0123] Furthermore, key classes related to data writing include InsertIntoHadoopFsRelationCommand class, HadoopMapReduceCommitProtocol class and InsertIntoHiveTable class. The code replacement module 302 specifically includes:

[0124] A first path code identification unit, used to identify a clear path code in the InsertIntoHadoopFsRelationCommand class;

[0125] A second path code identification unit is used to identify the deletion path codes in the HadoopMapReduceCommitProtocol class and the InsertIntoHiveTable class respectively;

[0126] The code modification and replacement unit is used to modify the clear path code and / or the delete path code into a move path code for moving the file to the HDFS file recycle bin.

[0127] Furthermore, the data file recovery device 300 further includes:

[0128] Hadoop distribution package module, used to obtain Hadoop distribution package and deploy the operating environment of HDFS file recycling bin based on Hadoop distribution package;

[0129] The Hadoop dependency module is used to obtain Hadoop dependency files from the Hadoop distribution package and build the dependency between the project's production environment and the HDFS file recycle bin based on the Hadoop dependency files.

[0130] Furthermore, the task monitoring module 305 specifically includes:

[0131] The task monitoring unit is used to monitor the Spark task and obtain the data files generated during the execution of the Spark task;

[0132] A first path definition unit is used to define a first source file path and a first target file path, wherein the first source file path is defined as an original file path, the first target file path is defined as an HDFS partition path in an HDFS file recycle bin, and the original file path is a path of a data file generated during the execution of a Spark task;

[0133] The first file moving unit is used to call the Java API in the Hadoop distribution package, and use the Java API to move the data file to the HDFS file recycling bin based on the original file path to the HDFS partition path.

[0134] Furthermore, a data file recycling instruction is obtained, the HDFS partition path where the data file is stored in the HDFS file recycling bin is queried, and the data file is restored using a data recovery tool based on the HDFS partition path, specifically including:

[0135] A path query unit, used to obtain a data file recycling instruction and query the HDFS partition path where the data file is stored in the HDFS file recycling bin;

[0136] The second path definition unit is used to define a second source file path and a second target file path, wherein the second source file path is an HDFS partition path, and the second target file path is an original file path.

[0137] The second file moving unit is used to call the Java API to move the data file to the production environment of the project based on the HDFS partition path to the original file path.

[0138] In the above embodiment, the present application is a data file recovery device, which belongs to the field of big data technology and the field of Internet medical care. The present application obtains the Spark source code of the project, identifies the key classes related to data writing in the Spark source code, modifies the clear path code and / or delete path code in the key class into a mobile path code, generates a new Spark source code, wherein the target of the mobile path code is the HDFS file recycle bin, compiles the new Spark source code, obtains the Spark binary distribution package, builds a data recovery tool, installs the Spark binary distribution package and the data recovery tool into the production environment of the project, monitors the Spark task, obtains the data files generated during the execution of the Spark task, and stores the data files in the HDFS file recycle bin, obtains the data file recovery instruction, queries the HDFS partition path where the data file is stored in the HDFS file recycle bin, and restores the data file based on the HDFS partition path using the data recovery tool. The present application modifies the Spark source code so that the files of the original partition path will be moved to the recycle bin, thereby providing a data backup and recovery mechanism, improving the reliability and fault tolerance of data processing, and ensuring that the data can still be recovered even if the task execution fails or an error occurs.

[0139] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0140] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0141] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0142] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of a data file recovery method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0143] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for running the data file recovery method.

[0144] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0145] In the above embodiment, the present application is a computer device, which belongs to the field of big data technology and the field of Internet medical care. The present application obtains the Spark source code of the project, identifies the key classes related to data writing in the Spark source code, modifies the clear path code and / or delete path code in the key class into a mobile path code, generates a new Spark source code, wherein the target of the mobile path code is the HDFS file recycle bin, compiles the new Spark source code, obtains the Spark binary distribution package, builds a data recovery tool, installs the Spark binary distribution package and the data recovery tool into the production environment of the project, monitors the Spark task, obtains the data files generated during the execution of the Spark task, and stores the data files in the HDFS file recycle bin, obtains the data file recovery instruction, queries the HDFS partition path where the data file is stored in the HDFS file recycle bin, and restores the data file based on the HDFS partition path using the data recovery tool. The present application modifies the Spark source code so that the files of the original partition path will be moved to the recycle bin, thereby providing a data backup and recovery mechanism, improving the reliability and fault tolerance of data processing, and ensuring that the data can still be recovered even if the task execution fails or an error occurs.

[0146] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the data file recovery method as described above.

[0147] In the above embodiment, the present application is a computer-readable storage medium, which belongs to the field of big data technology and the field of Internet medical care. The present application obtains the Spark source code of the project, identifies the key classes related to data writing in the Spark source code, modifies the clear path code and / or delete path code in the key class into a mobile path code, generates a new Spark source code, wherein the target of the mobile path code is the HDFS file recycle bin, compiles the new Spark source code, obtains the Spark binary distribution package, builds a data recovery tool, installs the Spark binary distribution package and the data recovery tool into the production environment of the project, monitors the Spark task, obtains the data files generated during the execution of the Spark task, and stores the data files in the HDFS file recycle bin, obtains the data file recovery instruction, queries the HDFS partition path where the data file is stored in the HDFS file recycle bin, and restores the data file based on the HDFS partition path using the data recovery tool. The present application modifies the Spark source code so that the files of the original partition path will be moved to the recycle bin, thereby providing a data backup and recovery mechanism, improving the reliability and fault tolerance of data processing, and ensuring that the data can still be recovered even if the task execution fails or an error occurs.

[0148] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0149] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0150] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A data file recovery method, characterized in that: include: Obtain the Spark source code of the project, and identify key classes related to data writing in the Spark source code; Modify the clear path code and / or delete path code in the key class into a move path code to generate a new Spark source code, wherein the target of the move path code is the HDFS file recycle bin; Compile the new Spark source code to obtain a Spark binary distribution package; Building a data recovery tool, and installing the Spark binary distribution package and the data recovery tool into the production environment of the project; Monitor the Spark task, obtain the data files generated during the execution of the Spark task, and store the data files in the HDFS file recycle bin; Obtain a data file recycling instruction, query the HDFS partition path where the data file is stored in the HDFS file recycling bin, and restore the data file using the data recovery tool based on the HDFS partition path.

2. The data file recovery method according to claim 1, characterized in that: The key classes related to data writing include InsertIntoHadoopFsRelationCommand class, HadoopMapReduceCommitProtocol class and InsertIntoHiveTable class.

3. The data file recovery method according to claim 2, characterized in that: The step of modifying the clearing path code and / or deleting path code in the key class into moving path code to generate a new Spark source code specifically includes: Identify the clear path code in the InsertIntoHadoopFsRelationCommand class; Respectively identify the deletion path codes in the HadoopMapReduceCommitProtocol class and the InsertIntoHiveTable class; The clear path code and / or the delete path code are modified into a move path code for moving the file to the HDFS file recycle bin.

4. The data file recovery method according to claim 1, characterized in that: Before obtaining the Spark source code of the project and identifying key classes related to data writing in the Spark source code, the method further includes: Obtain a Hadoop distribution package, and deploy the operating environment of the HDFS file recycle bin based on the Hadoop distribution package; Obtain a Hadoop dependency file from a Hadoop distribution package, and build a dependency between the production environment of the project and the HDFS file recycle bin based on the Hadoop dependency file.

5. The data file recovery method according to claim 4, characterized in that: The monitoring of the Spark task, obtaining the data files generated during the execution of the Spark task, and storing the data files in the HDFS file recycle bin specifically includes: Monitor the Spark task and obtain the data files generated during the execution of the Spark task; Define a first source file path and a first target file path, wherein the first source file path is defined as an original file path, the first target file path is defined as an HDFS partition path in the HDFS file recycle bin, and the original file path is a path of a data file generated during the execution of the Spark task; A Java API in the Hadoop distribution package is called, and the data file is moved to the HDFS file recycle bin based on the original file path to the HDFS partition path using the Java API.

6. The data file recovery method according to claim 5, characterized in that: Obtaining a data file recycling instruction, querying the HDFS partition path where the data file is stored in the HDFS file recycling bin, and restoring the data file using the data recovery tool based on the HDFS partition path, specifically includes: Obtain a data file recycling instruction, and query the HDFS partition path where the data file is stored in the HDFS file recycling bin; Define a second source file path and a second target file path, wherein the second source file path is an HDFS partition path, and the second target file path is the original file path; The Java API is called to move the data file to the production environment of the project based on the HDFS partition path to the original file path.

7. A data file recovery device, characterized in that: include: A key class identification module, used to obtain the Spark source code of the project and identify key classes related to data writing in the Spark source code; A code replacement module is used to modify the clearing path code and / or the deleting path code in the key class into a moving path code to generate a new Spark source code, wherein the target of the moving path code is an HDFS file recycle bin; A code compilation module, used to compile the new Spark source code to obtain a Spark binary distribution package; An environment deployment module, using a built data recovery tool, installs the Spark binary distribution package and the data recovery tool into the production environment of the project; A task monitoring module is used to monitor Spark tasks, obtain data files generated during the execution of the Spark tasks, and store the data files in the HDFS file recycle bin; The data recovery module uses the acquisition data file recovery instruction to query the HDFS partition path where the data file is stored in the HDFS file recycling bin, and restores the data file using the data recovery tool based on the HDFS partition path.

8. The data file recovery device according to claim 7, characterized in that: The key classes related to data writing include InsertIntoHadoopFsRelationCommand class, HadoopMapReduceCommitProtocol class and InsertIntoHiveTable class. The clearing path code and / or deleting path code in the key classes are modified into moving path code to generate new Spark source code, which specifically includes: Identify the clear path code in the InsertIntoHadoopFsRelationCommand class; Respectively identify the deletion path codes in the HadoopMapReduceCommitProtocol class and the InsertIntoHiveTable class; The clear path code and / or the delete path code are modified into a move path code for moving the file to the HDFS file recycle bin.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the data file recovery method according to any one of claims 1 to 6 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the data file recovery method according to any one of claims 1 to 6 are implemented.

Images