A method and system for data supplementing based on a big data platform

Abstract

The present application relates to the technical field of data supplementing, in particular to a data supplementing method and system based on a big data middle platform, comprising the following steps: S1: a user performs relevant operations on table files or library files in a file set through a file item operation unit; in the present application, a step recording unit records the relevant operations; in a work log, a tree-shaped logical window is grouped according to the relevant operations and constructs a tree-shaped logical structure, so as to group the operation records of the relevant operations, facilitate later calling, analyze operation file stubs, and reconstruct the changed content to restore the database; this kind of way constructs an integrated data middle platform structure, can record and group the operations in the operation process of the database, judge whether there is packet loss, and automatically reconstruct, avoids a large number of manual operation processes, and provides protection for the saving of the database files.

Description

Technical Field

[0001] This invention relates to the field of data supplementation technology, and in particular to a method and system for data supplementation based on a big data platform. Background Technology

[0002] Data supplementation refers to the process of adding missing data during database file operations to ensure the integrity of the database's data preservation. Traditional data supplementation methods often require manual verification of missing files, retrieval of historical files based on the missing file's content, and re-execution of related operations based on those historical files to achieve the data supplementation function. This process is poorly automated, requires significant manual intervention, and is prone to errors, necessitating improvements. Summary of the Invention

[0003] The purpose of this invention is to address the shortcomings of existing technologies by proposing a data supplementation method and system based on a big data platform.

[0004] To achieve the above objectives, the present invention adopts the following technical solution: a data supplementation method based on a big data platform, comprising the following steps:

[0005] S1: Users can perform related operations on table files or library files within the file set through the file item operation unit. These operations include creating files, deleting files, editing files, indexing files, and deleting indexes.

[0006] S2: During the execution of the operation by the file item operation unit, the step recording unit records the relevant operations and saves them in the work log;

[0007] S3: The file calibration unit compares the original database and the database after the completion of this step based on the work log to check for file loss. If a file is lost, it determines whether the user has performed the corresponding file deletion or index deletion operation. If so, the lost item restoration is not performed; otherwise, the lost item restoration is performed.

[0008] S4: In the recovery of lost items, the timeline is used to locate the timeline of packet loss, and the changed files and associated changed files are located by locating the lost items;

[0009] S5: After the timeline positioning is completed, the operation file stub is parsed by parsing the operation file stub and the changed content is reconstructed to restore the database.

[0010] As a further aspect of the present invention, in step S2, the recorded content of the work log consists of three parts: a timeline system, operation content records, and operation file stubs. In the timeline system, the tree-like logic pane groups related operations based on grouping criteria and constructs a tree-like logic structure, which is recorded based on the timeline. The operation content records record the flow of the workflow, and the operation file stubs record changed file names and related changed file names, and record the logical formula of the changed content between the two, which facilitates subsequent analysis.

[0011] As a further aspect of the present invention, in step S5, parsing the operation file stub specifically involves interpreting the changed content by parsing the changed content, thereby obtaining the relevant operations that need to be performed for the changed content, as well as the changed files and associated changed files corresponding to the relevant operations.

[0012] A data supplementation system based on a big data platform consists of a database, a step recording unit, a work log, a file calibration unit, and a missing item restoration unit. The output of the database is electrically connected to the input of the step recording unit, the output of the step recording unit is electrically connected to the input of the work log, the output of the work log is electrically connected to the input of the file calibration unit, and the output of the file calibration unit is electrically connected to the input of the missing item restoration unit and the database.

[0013] As a further aspect of the present invention, the database includes a file set, which includes library files and table files, wherein the output end of the library files is electrically connected to the input end of the table files.

[0014] As a further embodiment of the present invention, the output end of the file set is electrically connected to a file item operation unit, the file item operation unit including creating a file, deleting a file, editing a file, indexing a file, and deleting an index, and the output end of the file item operation unit is electrically connected to the input end of the step recording unit.

[0015] As a further embodiment of the present invention, the work log includes a timeline system, operation content records, and operation file stubs. The output end of the timeline system is electrically connected to the input end of the operation content records, and the output end of the operation content records is electrically connected to the input end of the operation file stubs.

[0016] As a further embodiment of the present invention, the output end of the time axis system is electrically connected to a tree-like logic pane, the output end of the tree-like logic pane is electrically connected to a grouping basis and a time axis, the output end of the operation content record is electrically connected to a workflow, the operation file stub includes a changed file name, a related changed file name, and changed content, and the output ends of the changed file name and the related changed file name are electrically connected to the input end of the changed content.

[0017] As a further embodiment of the present invention, the lost item restoration includes lost item location and timeline location. The lost item location includes locating the change file and locating the associated change file. The timeline location includes parsing the operation file stub. The output end of the parsing operation file stub is electrically connected to the parsed change content. The output end of the parsed change content is electrically connected to the reconstructed change content. The output end of the reconstructed change content is electrically connected to the input ends of the located change file and the located associated change file.

[0018] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0019] In this invention, relevant operations are recorded by a step recording unit and stored in a work log. Within the work log, a tree-like logical pane groups relevant operations based on grouping criteria, constructing a tree-like logical structure to group operation records for easy later retrieval. The operation file stubs are parsed, and the changed content is reconstructed to restore the database. This method constructs an integrated data platform structure that records and groups operations during database operations, determines packet loss, and automatically reconstructs data, avoiding extensive manual processes and ensuring the database's file preservation. Attached Figure Description

[0020] Figure 1 This invention proposes a method and system for data supplementation based on a big data middle platform, and provides a schematic diagram of the workflow.

[0021] Figure 2 This invention proposes a method for data supplementation based on a big data middle platform and a system framework flowchart;

[0022] Figure 3 This invention proposes a method for data supplementation based on a big data middle platform and a flowchart of the file item operation unit of the system;

[0023] Figure 4 This invention proposes a method for data supplementation based on a big data middle platform and a flowchart of the system's work log.

[0024] Figure 5 This invention proposes a method for data supplementation based on a big data middle platform and a flowchart of the system for restoring lost items. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0026] In the description of this invention, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, in the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0027] Example 1

[0028] Please see Figure 1 This invention provides a technical solution: a method for data supplementation based on a big data middle platform, comprising the following steps:

[0029] S1: Users can perform related operations on table files or library files within the file set through the file item operation unit. These operations include creating files, deleting files, editing files, indexing files, and deleting indexes.

[0030] S2: During the execution of the operation by the file item operation unit, the step recording unit records the relevant operations and saves them in the work log;

[0031] S3: The file calibration unit compares the original database and the database after the completion of this step based on the work log to check for file loss. If a file is lost, it determines whether the user has performed the corresponding file deletion or index deletion operation. If so, the lost item restoration is not performed; otherwise, the lost item restoration is performed.

[0032] S4: In the recovery of lost items, the timeline is used to locate the timeline of packet loss, and the changed files and associated changed files are located by locating the lost items;

[0033] S5: After the timeline positioning is completed, the operation file stub is parsed by parsing the operation file stub and the changed content is reconstructed to restore the database.

[0034] In S2, the work log consists of three parts: a timeline system, operation content records, and operation file stubs. The timeline system uses a tree-like logical pane to group related operations based on grouping criteria, constructing a tree-like logical structure and recording data based on the timeline. The operation content records the workflow flow, while the operation file stubs record changed filenames and associated changed filenames, along with the logical formulas for the changed content, facilitating later parsing. The purpose of this design is to define the specific content and items recorded in the work log. The tree-like logical pane groups related operations based on grouping criteria and constructs a tree-like logical structure, thereby grouping the operation records of related operations for easy later retrieval.

[0035] In S5, parsing the operation file stub involves interpreting the changed content to obtain the specific operations that need to be performed, as well as the corresponding changed files and associated changed files. This design limits the specific content parsed from the operation file stub, thereby refining the execution items for restoring the database and enabling automated reconstruction of the changed content.

[0036] Please see Figure 2 A data supplementation system based on a big data platform consists of a database, a step recording unit, a work log, a file calibration unit, and a lost item restoration unit. The output of the database is electrically connected to the input of the step recording unit, the output of the step recording unit is electrically connected to the input of the work log, the output of the work log is electrically connected to the input of the file calibration unit, and the output of the file calibration unit is electrically connected to the input of the lost item restoration unit and the database. The database includes a set of files, which includes library files and table files. The output of the library files is electrically connected to the input of the table files. During the execution of the file item operation unit, the step recording unit records the relevant operations and saves them in the work log. The file calibration unit compares the original database and the database after the completion of the step based on the work log to check for file loss. In the case of file loss, it determines whether the user has performed corresponding file deletion or index deletion operations. If so, the lost item restoration is not performed; otherwise, the lost item restoration is performed.

[0037] Please see Figure 3 The output of the file set is electrically connected to a file item operation unit, which includes functions for creating files, deleting files, editing files, indexing files, and deleting indexes. The output of the file item operation unit is electrically connected to the input of the step recording unit. Users can perform related operations on table files or library files within the file set through the file item operation unit. These related operations include creating files, deleting files, editing files, indexing files, and deleting indexes.

[0038] Please see Figure 4 The work log includes a timeline system, operation content records, and operation file stubs. The output of the timeline system is electrically connected to the input of the operation content records, and the output of the operation content records is electrically connected to the input of the operation file stubs. The output of the timeline system is electrically connected to a tree-like logic pane, and the output of the tree-like logic pane is electrically connected to the grouping basis and the timeline. The output of the operation content records is electrically connected to the workflow. The operation file stubs include changed file names, associated changed file names, and changed content. The outputs of the changed file names and associated changed file names are electrically connected to the input of the changed content. The work log's recorded content consists of three parts: the timeline system, operation content records, and operation file stubs. In the timeline system, the tree-like logic pane groups related operations based on the grouping basis and constructs a tree-like logic structure, recording based on the timeline. The operation content records the workflow's flow content. The operation file stubs record changed file names and associated changed file names, and record the logical formulas of the changed content between them, facilitating later analysis.

[0039] Please see Figure 5 Lost item restoration includes lost item location and timeline location. Lost item location includes locating the changed file and the associated changed file. Timeline location includes parsing the operation file stub. The output of the parsed operation file stub is electrically connected to the parsed changed content. The output of the parsed changed content is electrically connected to the reconstructed changed content. The output of the reconstructed changed content is electrically connected to the input of the located changed file and the located associated changed file. In lost item restoration, the timeline of packet loss is located through timeline location, and the changed file and associated changed file are located through lost item location. After the timeline location is completed, the operation file stub is parsed through parsing, and the changed content is reconstructed, thereby restoring the database.

[0040] Working Principle: During database operations, users perform related operations on table files or database files within the file set through the file item operation unit. These operations include creating, deleting, editing, indexing, and deleting indexes. During the execution of these operations, the step recording unit records the relevant operations and saves them in the work log. The file calibration unit compares the original database with the database after the step is completed based on the work log to check for file loss. If files are lost, it determines whether the user performed the corresponding file deletion or index deletion operation. If so, the lost item restoration is not performed; otherwise, it is. In the lost item restoration process, the timeline of the packet loss is located using timeline positioning, and the changed files and associated changed files are located using lost item positioning. After the timeline positioning is completed, the operation file stubs are parsed, and the changed content is reconstructed, thereby restoring the database.

[0041] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for data supplement based on a big data platform, characterized in that, It comprises the following steps: S1: the user performs relevant operations on the table file or library file in the file set through the file item operation unit, and the relevant operations include creating a file, deleting a file, editing a file, indexing a file, and deleting an index; S2: the file item operation unit records the relevant operations in the process of performing the operations, and saves them in the work log; S3: the file calibration unit compares the original database and the database after S2 based on the work log to determine whether there is a file packet loss, and determines whether the user has performed a corresponding delete file or delete index operation in the case of file loss, and if so, does not perform the missing item restoration, and if not, performs the missing item restoration; S4: In the missing item restoration, the time axis is located by locating the time axis of the packet loss, and the changed file and the associated changed file are located by the missing item positioning; S5: After the time axis positioning is completed, the operation file stub is parsed by parsing the operation file stub, and the changed content is reconstructed to restore the database; In S2, the recording content of the work log is composed of three parts: the time axis system, the operation content record, and the operation file stub. In the time axis system, the tree-like logical window is grouped according to the relevant operations, and a tree-like logical structure is constructed based on the time axis. In the operation content record, the flow content of the work flow is recorded. In the operation file stub, the changed file name and the associated changed file name are recorded, and the logical formula of the changed content is recorded between them to facilitate later analysis. 2.The method of claim 1, wherein: In S5, the operation file stub is parsed by parsing the changed content to interpret the changed content, so as to obtain the relevant operations required by the changed content and the changed file and the associated changed file corresponding to the relevant operations.

3. A system for data supplementing based on a big data platform, for implementing the method of claim 1 or 2, characterized in that: The system for data supplement recording based on a big data platform is composed of a database, a step recording unit, a work log, a file calibration unit, and a missing item restoration. The output end of the database is electrically connected to the input end of the step recording unit. The output end of the step recording unit is electrically connected to the input end of the work log. The output end of the work log is electrically connected to the input end of the file calibration unit. The output end of the file calibration unit is electrically connected to the input end of the missing item restoration and the database. 4.The system of data supplementing based on big data middle platform according to claim 3, characterized in that: The database comprises a file set, the file set comprises a library file and a table file, and the output end of the library file is electrically connected to the input end of the table file. 5.The system of data supplementing based on big data middle platform according to claim 4, characterized in that: The output end of the file set is electrically connected to a file item operation unit, the file item operation unit comprises creating a file, deleting a file, editing a file, indexing a file, and deleting an index, and the output end of the file item operation unit is electrically connected to the input end of the step recording unit. 6.The system of data supplementing based on big data middle platform according to claim 3, characterized in that: The work log comprises a time axis system, an operation content record, and an operation file stub. The output end of the time axis system is electrically connected to the input end of the operation content record. The output end of the operation content record is electrically connected to the input end of the operation file stub. 7.The system of data supplementing based on big data middle platform according to claim 6, characterized in that: The output end of the time axis system is electrically connected with a tree logic window, the output end of the tree logic window is electrically connected with a grouping basis and time axis, the output end of the operation content record is electrically connected with a work flow, the operation file stub includes a change file name, a change file name and a change content, and the output end of the change file name and the change file name is electrically connected with the input end of the change content. 8.The system of data supplementing based on big data middle platform according to claim 3, characterized in that: The lost item recovery includes lost item positioning and time axis positioning, the lost item positioning includes positioning a change file and positioning a change file, the time axis positioning includes analyzing an operation file stub, the output end of the analyzing operation file stub is electrically connected with the output end of the analyzing change content, the output end of the analyzing change content is electrically connected with the output end of the reestablishing change content, and the output end of the reestablishing change content is electrically connected with the input end of the positioning change file and the positioning change file.

Images