A method and device for processing merged data

Abstract

The present invention provides a method and device for demerging data processing, which relates to the field of data processing technology and can be used in the financial field or other technical fields. The method includes: determining a target reference file corresponding to the target source file currently being processed according to pre-set demerging configuration information; performing a back-up process on the target source file, and if it is determined that the backup process is successful, demerging the target source file according to the target reference file; if it is determined that the demerging process is successful, verifying the demerging process result according to the backup process result, and if the verification process result is a pass, restoring the stopped running state of the target job to the started running state. The device executes the above method. The demerging data processing method and device provided by the embodiment of the present invention can not only improve the efficiency and accuracy of demerging data processing, but also improve the timeliness of external services.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method and device for processing merged and unmerged data. Background Art

[0002] Institutional branches are widely distributed, and sometimes they need to be merged or closed. When a cross-regional branch merger occurs, all operations at that branch must be migrated to the new branch, necessitating the processing of the merged data. However, this data processing is tedious and complex, requiring manual intervention, time-consuming and labor-intensive, and prone to a range of issues, including data inaccuracies. Summary of the Invention

[0003] In response to the problems in the prior art, embodiments of the present invention provide a method and apparatus for processing merged data, which can at least partially solve the problems in the prior art.

[0004] In one aspect, the present invention provides a method for processing merged data, comprising:

[0005] Determining a target reference file corresponding to a target source file currently being processed based on pre-set merge configuration information; wherein the target source file has previously completed a target job batch run, and the job status is a stopped running state; the target reference file has previously been loaded; and the merge configuration information includes a correspondence between the source file of the merged data and the reference file;

[0006] Backing up the target source file, and if the backup process is successful, performing a merge and unmerge process on the target source file according to the target reference file;

[0007] If it is determined that the unmerge process is successful, the unmerge process result is verified according to the backup process result. If the verification process result is that the verification passes, the stopped running state of the target job is restored to the started running state.

[0008] The step of determining the target reference file corresponding to the currently processed target source file according to the preset merge configuration information includes:

[0009] According to the preset merge configuration information, a polling processing method is used to determine the target reference file corresponding to the currently processed target source file.

[0010] The pre-set merge and unmerge code is called, and the merge and unmerge code includes a data backup script; accordingly, the backing up process of the target source file includes:

[0011] The data backup script is called to perform backup processing on the target source file.

[0012] The merging code includes a source file merging script; accordingly, the merging process of the target source file according to the target control file includes:

[0013] The source file merging script is called, and the target source file is merging-processed according to the target comparison file.

[0014] The merging and unmerging code includes a data verification script; accordingly, the verification process of the merging and unmerging result according to the backup process result includes:

[0015] The data verification script is called, and the merge and unmerge processing result is verified according to the backup processing result.

[0016] The merging configuration information further includes a correspondence between the source file, the reference file, and the job of the merged data; accordingly, before the step of determining the target reference file corresponding to the target source file currently being processed according to the preset merging configuration information, the merging data processing method further includes:

[0017] Determine a target job corresponding to the target source file according to the merge configuration information;

[0018] The target jobs are run in batches, and after the batch runs are completed, the job status of the target jobs is set to the stopped running state.

[0019] Wherein, before the target reference file is loaded in advance, the merge data processing method further includes determining whether the reference file has been locally arrived; accordingly, the determining whether the reference file has been locally arrived includes:

[0020] If it is determined that the control file has not arrived locally within t-t1 to t, then at t+t2, polling is performed to determine whether the control file has arrived locally until it is determined that the control file has arrived locally; t is the current time, t1 is the first preset time length, and t2 is the second preset time length, and the first preset time length is greater than the second preset time length.

[0021] In one aspect, the present invention provides a demerging data processing device, comprising:

[0022] a determination unit, configured to determine a target reference file corresponding to a currently processed target source file based on pre-set merge configuration information; wherein the target source file has previously completed a target job batch run, and the job status is a stopped running state; the target reference file has previously been loaded; and the merge configuration information includes a correspondence between the source file of the merged data and the reference file;

[0023] a decomposition unit, configured to perform a backup process on the target source file, and if the backup process is determined to be successful, decomposition the target source file according to the target reference file;

[0024] The recovery unit is configured to verify the merging and unmerging result according to the backup result if the merging and unmerging process is successful, and to restore the stopped running state of the target job to the started running state if the verification result is that the verification passes.

[0025] On the other hand, an embodiment of the present invention provides an electronic device, comprising: a processor, a memory, and a bus, wherein:

[0026] The processor and the memory communicate with each other via the bus;

[0027] The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the following method:

[0028] Determining a target reference file corresponding to a target source file currently being processed based on pre-set merge configuration information; wherein the target source file has previously completed a target job batch run, and the job status is a stopped running state; the target reference file has previously been loaded; and the merge configuration information includes a correspondence between the source file of the merged data and the reference file;

[0029] Backing up the target source file, and if the backup process is successful, performing a merge and unmerge process on the target source file according to the target reference file;

[0030] If it is determined that the unmerge process is successful, the unmerge process result is verified according to the backup process result. If the verification process result is that the verification passes, the stopped running state of the target job is restored to the started running state.

[0031] An embodiment of the present invention provides a non-transitory computer-readable storage medium, including:

[0032] The non-transitory computer-readable storage medium stores computer instructions, which cause the computer to execute the following method:

[0033] Determining a target reference file corresponding to a target source file currently being processed based on pre-set merge configuration information; wherein the target source file has previously completed a target job batch run, and the job status is a stopped running state; the target reference file has previously been loaded; and the merge configuration information includes a correspondence between the source file of the merged data and the reference file;

[0034] Backing up the target source file, and if the backup process is successful, performing a merge and unmerge process on the target source file according to the target reference file;

[0035] If it is determined that the unmerge process is successful, the unmerge process result is verified according to the backup process result. If the verification process result is that the verification passes, the stopped running state of the target job is restored to the started running state.

[0036] The demerging data processing method and device provided by the embodiment of the present invention determine the target reference file corresponding to the currently processed target source file according to pre-set demerging configuration information; back up the target source file, and if it is determined that the backup processing is successful, demerge the target source file according to the target reference file; if it is determined that the demerging processing is successful, verify the demerging processing result according to the backup processing result, and if the verification processing result is that the verification passes, restore the stopped running state of the target job to the started running state, which not only improves the efficiency and accuracy of demerging data processing, but also improves the timeliness of external services. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0038] Figure 1 It is a flowchart of a method for processing merged data provided by one embodiment of the present invention.

[0039] Figure 2 A schematic diagram of a process for generating a demerger code and demerger configuration information according to an embodiment of the present invention.

[0040] Figure 3 It is a flowchart of a data merging processing method provided by another embodiment of the present invention.

[0041] Figure 4 It is a structural diagram of a data merging and demerging processing device provided by one embodiment of the present invention.

[0042] Figure 5 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] To make the purpose, technical solutions and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention. It should be noted that, unless there is a conflict, the embodiments and features in the embodiments of this application can be combined with each other in any manner.

[0044] Figure 1 FIG. 1 is a flow chart of a method for processing data merging provided by an embodiment of the present invention. Figure 1 As shown, the method for processing merging and removing data provided by the embodiment of the present invention includes:

[0045] Step S1: Determine a target reference file corresponding to the target source file currently being processed based on pre-set merge configuration information; wherein the target source file has previously completed a target job batch run, and the job status is a stopped running state; the target reference file has previously been loaded; and the merge configuration information includes a correspondence between the source file and the reference file of the merged data.

[0046] Step S2: backing up the target source file. If the backup process is successful, the target source file is merged according to the target reference file.

[0047] Step S3: If it is determined that the merging and unmerging process is successful, the merging and unmerging process result is verified according to the backup process result. If the verification process result is that the verification passes, the stopped running state of the target job is restored to the started running state.

[0048] In step S1, the device determines a target reference file corresponding to the currently processed target source file based on pre-set merge configuration information; wherein the target source file has previously completed a target job batch run, and the job status is stopped; the target reference file has previously been loaded; and the merge configuration information includes the correspondence between the source file and the reference file of the merged data. The device can be a computer device that executes the method, for example, a server that is deployed with a database, specifically a data lake. It should be noted that the acquisition and analysis of merged data in the embodiments of the present invention are authorized by the user.

[0049] When importing source file data into the data lake, if the data involves mergers and reorganizations, the data already imported into the data lake needs to be changed when cross-regional branch mergers and reorganizations occur. For the source file tables involved in the merger and reorganization, a reference file is provided, and the data lake changes the data in the data lake based on the reference relationship in the reference file.

[0050] Taking Account Table A as an example, this table contains fields for region code and account number. When a merger occurs, the region code and account number in Account Table A are updated to the new region and account number based on the mapping between the old and new regions in the comparison file. For non-merged regions, the records in Account Table A remain unchanged. Because many source file tables involve mergers and demergers, and each comparison file has different data feed methods and logic, and each has its own unique merger and demerger comparison logic, developing and implementing merger and demerger data is complex and tedious, which can easily lead to data issues.

[0051] like Figure 2 As shown, the pre-set merger and revocation configuration information and merger and revocation codes are explained as follows:

[0052] If the source files imported into the data lake involve regional mergers, after registering information such as the source file name, reference file name, reference file entry table name, and merger logic description, the merger code (data backup script, source file merger script, data verification script) and merger configuration information (the correspondence between the source file and reference file, or the correspondence between the source file, reference file, and job) can be automatically generated. If the source file structure changes, the merger reference file changes, or the merger logic is adjusted, the corresponding registration information can be adjusted to automatically generate the latest merger code and merger configuration information.

[0053] The target job can be understood as the job corresponding to the target source file, which can be run in batches. Figure 3 As shown, the target source file has completed the target job batch run in advance, and the job status is stopped, corresponding to Figure 3 In the "Complete T-1 day batch" and "Stop batch" options, T is the day for withdrawing and processing data, and T-1 is the day before the withdrawal and processing day. This means that the target batch job can be completed one day in advance. This prevents data errors caused by continued batch job execution.

[0054] The target control file has been loaded in advance, corresponding to Figure 3 In the "Is the file loaded successfully?" judgment branch, the answer is "yes".

[0055] In the above step S2, the device performs a backup process on the target source file. If the backup process is successful, the target source file is merged according to the target reference file. Figure 3 In the "Is the backup successful?" judgment branch, the answer is "Yes".

[0056] The target source file is merged and decompressed according to the target reference file. The example of the account table A mentioned above can be referred to and will not be described in detail.

[0057] In the above step S3, if the device determines that the merging process is successful, the merging process result is verified according to the backup process result. If the verification process result is verified, the stopped operation state of the target job is restored to the started operation state. Figure 3 The judgment branch for "whether the execution is successful" is "yes".

[0058] Verify the merge result based on the backup result. Since the embodiment of the present invention does not change the data content but the data quantity, this step may further include: determining whether the number of data records in the backup is consistent with the number of data records in the source file after the merge. If they are consistent, the verification process is considered to be passed, and the corresponding Figure 3 In the "Is the verification passed" section, the judgment branch is "yes".

[0059] If they are inconsistent, the verification will be considered as failed, corresponding to Figure 3 In the "Is the verification passed" judgment branch, the answer is "No".

[0060] Restore the stopped running state of the target job to the started running state, corresponding to Figure 3 "Start the operation to restore service" indicates that the source file removal and consolidation is complete and external services can be restored.

[0061] The demerging data processing method provided by an embodiment of the present invention determines a target reference file corresponding to a currently processed target source file according to pre-set demerging configuration information; performs back-up processing on the target source file, and if it is determined that the backup processing is successful, demerges the target source file according to the target reference file; if it is determined that the demerging processing is successful, verifies the demerging processing result according to the backup processing result, and if the verification processing result is that the verification passes, restores the stopped running state of the target job to the started running state, which not only improves the efficiency and accuracy of demerging data processing, but also improves the timeliness of external services.

[0062] Furthermore, the step of determining the target reference file corresponding to the currently processed target source file according to the preset merge configuration information includes:

[0063] According to the pre-set merge configuration information, the polling processing method is used to determine the target reference file corresponding to the target source file currently being processed. Figure 3, automatically scans for source files that meet the two conditions of "target job batch run completed on day T-1" and "file loading successful," and then polls for unmerge processing. If the number of reference file loading records in the unmerge process control table is 0, it means that there is no business data for the unmerge area in the source file, and the source file unmerge status is directly set to "Completed." If the number of records is not 0, it means that the reference file corresponding to this number of records is the target reference file corresponding to the target source file, and the subsequent data backup steps will continue.

[0064] Furthermore, a pre-set unmerge code is called, and the unmerge code includes a data backup script; accordingly, the target source file is backed up, including:

[0065] The data backup script is called to perform a backup process on the target source file. Referring to the above description, after determining the target reference file corresponding to the target source file, the data backup script is called to perform a backup process on the target source file.

[0066] Furthermore, the merging code includes a source file merging script; accordingly, the merging process of the target source file according to the target control file includes:

[0067] Call the source file unmerge script and unmerge the target source file according to the target reference file. Referring to the above description, after the backup process is determined to be successful, call the source file unmerge script and unmerge the target source file according to the target reference file.

[0068] Furthermore, the merging and unmerging code includes a data verification script; accordingly, the verification process of the merging and unmerging result according to the backup process result includes:

[0069] Call the data verification script and verify the unmerge processing result according to the backup processing result. Referring to the above description, after determining that the unmerge processing is successful, call the data verification script and verify the unmerge processing result according to the backup processing result.

[0070] Furthermore, the merging configuration information further includes a correspondence between the source file, the reference file, and the job of the merged data; accordingly, before the step of determining the target reference file corresponding to the currently processed target source file according to the preset merging configuration information, the merging data processing method further includes:

[0071] Determine the target job corresponding to the target source file according to the merge configuration information; that is, determine the target job corresponding to the target source file according to the corresponding relationship between the above source files, reference files and jobs, Figure 3 "Source Tables and Source Table Jobs" in the

[0072] The target jobs are run in batches, and after the batch runs are completed, the job status of the target jobs is set to the stopped state.

[0073] Furthermore, before the target reference file is loaded in advance, the merge data processing method further includes determining whether the reference file has been locally received. Accordingly, the determining whether the reference file has been locally received includes:

[0074] If it is determined that the reference file has not arrived locally within t-t1 to t, then at t+t2, polling is performed to determine whether the reference file has arrived locally until it is determined that the reference file has arrived locally; t is the current time, t1 is the first preset time, and t2 is the second preset time, and the first preset time is greater than the second preset time. Considering a longer time dimension, t can also be Figure 3 The first preset duration and the second preset duration can be set according to the actual situation, and can be selected as 7 days and 5 minutes respectively. Figure 3 In the process, the check is performed on whether the file has arrived, the judgment branch is "no", and "wait for the next polling".

[0075] The data merging and unmerging processing method of the embodiment of the present invention can be realized by modularization, including a reference file loading module, a source table merging and unmerging module, a checking and verifying module and an automatic alarm module. Figure 3 The embodiments of the present invention are described as follows:

[0076] The first step is to set the job status of the data lake source table registered as "involved in merger" to "stop running" after the batch run is completed on T-1 according to the merger configuration information to prevent data errors caused by continued batch job running.

[0077] In the second step, the reference file loading module begins polling to check whether each reference file has arrived at the data lake server. Reference file data can be supplied in two scenarios. In the first scenario, reference files are supplied daily. Since the supply frequency is fixed, a regular daily data loading operation can be established. In the first scenario, the reference file loading module automatically checks whether the batch loading operation for reference files on day T has been completed. If completed, the reference file has been successfully loaded.

[0078] The second scenario involves data being provided only within the withdrawal and consolidation window. The time and frequency of data provision are not fixed, making it impossible to establish a lake entry operation. For the second scenario, if the specific date of the reference file provision cannot be determined, the server will automatically check whether the reference file exists in the file receiving path for the past seven days. If not, polling will continue after five minutes. If the reference file has arrived, the loading process will be triggered and the reference file will be loaded into the T-day partition of the reference file.

[0079] Automatically check the number of loaded records in the comparison file, and synchronize the loading status (not loaded, loaded successfully, loaded failed) and the number of loaded records to the merging process control table.

[0080] In the third step, the source table unmerge module automatically scans for source tables that meet the following two conditions: "the corresponding job has completed batch execution on T-1" and "the reference file has been loaded successfully." The unmerge process is then performed in a round-robin fashion. If the number of reference file loaded records in the unmerge process control table is 0, this indicates that the source table contains no business data for the unmerged region, and the unmerge status can be set to "Completed."

[0081] If the number of records is not 0, it means that the reference file corresponding to the record number is the reference file corresponding to the source table. In this case, the source table data backup script is executed first. If the execution is successful, the source table unmerge script is executed. If the execution is successful, "Unmerge script execution successful" is marked in the unmerge process control table. If the backup script fails to execute, the unmerge script is not triggered.

[0082] In the fourth step, the check and verification module polls and checks the source tables whose merge status is "Merge script executed successfully". If the conditions are met, the check process is triggered to check the number of records in the backup table and the number of records in the source table after the merge. If the number of records in the backup table and the source table after the merge is inconsistent, the verification status is "Verification failed".

[0083] If the backup table and the merged source table have the same number of records, the verification status is "Verification Passed." For source tables that pass verification, the corresponding batch job is set to "Started," indicating that the merge is complete and external services are restored.

[0084] The automatic alarm module automatically captures exceptions in each module and issues an alarm. This includes: source tables whose batch runs failed to complete on day T-1 in step 1; source tables whose status in the reference file loading module is "File timeout not reached" or "Load failed" in step 2; "Data backup failure" or "Unmerge script execution failure" in step 3; and "Verification failure" in step 4.

[0085] It should be noted that the data merging and demerging processing method provided in the embodiment of the present invention can be used in the financial field, and can also be used in any technical field other than the financial field. The embodiment of the present invention does not limit the application field of the data merging and demerging processing method.

[0086] Figure 4 FIG. 1 is a structural diagram of a data processing device provided by an embodiment of the present invention. Figure 4 As shown, the demerging data processing device provided by the embodiment of the present invention includes a determining unit 401, a demerging unit 402 and a restoring unit 403, wherein:

[0087] The determination unit 401 is used to determine the target reference file corresponding to the target source file currently being processed based on the pre-set merging configuration information; wherein, the target source file has completed the batch operation of the target job in advance, and the job status is the stopped running state; the target reference file has been loaded in advance; the merging configuration information includes the correspondence between the source file and the reference file of the merging data; the merging unit 402 is used to back up the target source file, and if it is determined that the backup processing is successful, the target source file is merging according to the target reference file; the recovery unit 403 is used to verify the merging processing result according to the backup processing result if it is determined that the merging processing is successful, and if the verification processing result is verification passed, the stopped running state of the target job is restored to the started running state.

[0088] Specifically, the determination unit 401 in the device is used to determine the target reference file corresponding to the target source file currently being processed based on the pre-set merging configuration information; wherein, the target source file has completed the batch operation of the target job in advance, and the job status is the stopped running state; the target reference file has been loaded in advance; the merging configuration information includes the correspondence between the source file and the reference file of the merging data; the merging unit 402 is used to back up the target source file, and if it is determined that the backup processing is successful, the target source file is merging according to the target reference file; the recovery unit 403 is used to verify the merging processing result according to the backup processing result if it is determined that the merging processing is successful, and if the verification processing result is verification passed, the stopped running state of the target job is restored to the started running state.

[0089] The merging data processing device provided by an embodiment of the present invention determines a target reference file corresponding to a currently processed target source file according to pre-set merging configuration information; performs back-up processing on the target source file, and if it is determined that the backup processing is successful, performs merging processing on the target source file according to the target reference file; if it is determined that the merging processing is successful, performs verification processing on the merging processing result according to the backup processing result, and if the verification processing result is verification passed, restores the stopped running state of the target job to the started running state, which not only improves the efficiency and accuracy of merging data processing, but also improves the timeliness of external services.

[0090] The embodiment of the present invention provides an embodiment of a demerging data processing device which can be used to execute the processing flow of the above-mentioned method embodiments. Its functions are not described in detail here, and reference can be made to the detailed description of the above-mentioned method embodiments.

[0091] Figure 5 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as Figure 5As shown, the electronic device includes: a processor 501, a memory 502 and a bus 503;

[0092] The processor 501 and the memory 502 communicate with each other via a bus 503.

[0093] The processor 501 is configured to call the program instructions in the memory 502 to execute the methods provided by the above method embodiments, for example, including:

[0094] Determining a target reference file corresponding to a target source file currently being processed based on pre-set merge configuration information; wherein the target source file has previously completed a target job batch run, and the job status is a stopped running state; the target reference file has previously been loaded; and the merge configuration information includes a correspondence between the source file of the merged data and the reference file;

[0095] Backing up the target source file, and if the backup process is successful, performing a merge and unmerge process on the target source file according to the target reference file;

[0096] If it is determined that the unmerge process is successful, the unmerge process result is verified according to the backup process result. If the verification process result is that the verification passes, the stopped running state of the target job is restored to the started running state.

[0097] This embodiment discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can perform the methods provided in the above-mentioned method embodiments, for example, including:

[0098] Determining a target reference file corresponding to a target source file currently being processed based on pre-set merge configuration information; wherein the target source file has previously completed a target job batch run, and the job status is a stopped running state; the target reference file has previously been loaded; and the merge configuration information includes a correspondence between the source file of the merged data and the reference file;

[0099] Backing up the target source file, and if the backup process is successful, performing a merge and unmerge process on the target source file according to the target reference file;

[0100] If it is determined that the unmerge process is successful, the unmerge process result is verified according to the backup process result. If the verification process result is that the verification passes, the stopped running state of the target job is restored to the started running state.

[0101] This embodiment provides a computer-readable storage medium storing a computer program. The computer program enables the computer to execute the methods provided in the above method embodiments, for example, including:

[0102] Determining a target reference file corresponding to a target source file currently being processed based on pre-set merge configuration information; wherein the target source file has previously completed a target job batch run, and the job status is a stopped running state; the target reference file has previously been loaded; and the merge configuration information includes a correspondence between the source file of the merged data and the reference file;

[0103] Backing up the target source file, and if the backup process is successful, performing a merge and unmerge process on the target source file according to the target reference file;

[0104] If it is determined that the unmerge process is successful, the unmerge process result is verified according to the backup process result. If the verification process result is that the verification passes, the stopped running state of the target job is restored to the started running state.

[0105] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0106] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0107] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0108] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0109] Throughout this specification, reference to terms such as "one embodiment," "a specific embodiment," "some embodiments," "for example," "example," "specific example," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0110] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for processing merged data, characterized in that: include: If the source file imported into the data lake involves merging, the merging code and merging configuration information will be automatically generated after registering the source file name, reference file name, reference file entry table name, and merging logic description information. When the source file structure changes, the merge comparison file changes, or the merge logic is adjusted, the corresponding registration information is adjusted to automatically generate the latest merge code and merge configuration information; Determining a target reference file corresponding to a currently processed target source file based on the pre-set merge configuration information; wherein the target source file has previously completed a target job batch run, and the job status is a stopped running state; the target reference file has previously been loaded; the merge configuration information includes a correspondence between the source file and the reference file of the merged data; the target reference file includes the number of loaded records; Backing up the target source file, and if the backup process is successful, performing a merge and unmerge process on the target source file according to the target reference file; If it is determined that the merging and unmerging process is successful, the merging and unmerging process result is verified according to the backup process result. If the verification process result is that the verification passes, the stopped running state of the target job is restored to the started running state; The target source file has already completed the target job batch running, and the job status is stopped, including: If the target job corresponding to the target source file completes batch operation on T-1, the job status is set to stopped; T is the day for withdrawing and merging data processing, and T-1 is the day before the day for withdrawing and merging data processing; The step of determining the target reference file corresponding to the currently processed target source file according to the preset merge configuration information includes: According to the preset merge configuration information, a polling processing method is used to determine the target reference file corresponding to the currently processed target source file.

2. The method for processing merged data according to claim 1, characterized in that: Calling a preset unmerge code, the unmerge code including a data backup script; accordingly, backing up the target source file includes: The data backup script is called to perform backup processing on the target source file.

3. The method for processing merged data according to claim 2, characterized in that: The merging code includes a source file merging script; accordingly, the merging process of the target source file according to the target control file includes: The source file merging script is called, and the target source file is merging-processed according to the target comparison file.

4. The method for processing merged data according to claim 3, characterized in that: The demerging code includes a data verification script; accordingly, the verification process of the demerging result according to the backup process result includes: The data verification script is called, and the merge and unmerge processing result is verified according to the backup processing result.

5. The method for processing merged data according to any one of claims 1 to 4, characterized in that: The merging configuration information also includes a correspondence between the source file, the reference file, and the job of the merged data; accordingly, before the step of determining the target reference file corresponding to the target source file currently being processed according to the preset merging configuration information, the merging data processing method further includes: Determine a target job corresponding to the target source file according to the merge configuration information; The target jobs are run in batches, and after the batch runs are completed, the job status of the target jobs is set to the stopped running state.

6. The method for processing merged data according to any one of claims 1 to 4, characterized in that: Before the target reference file is loaded in advance, the merge data processing method further includes determining whether the reference file has been locally arrived; accordingly, the determining whether the reference file has been locally arrived includes: If it is determined that the control file has not arrived locally within t-t1 to t, then at t+t2, polling is performed to determine whether the control file has arrived locally until it is determined that the control file has arrived locally; t is the current time, t1 is the first preset time length, and t2 is the second preset time length, and the first preset time length is greater than the second preset time length.

7. A data processing device for merging and demerging, characterized in that: include: A determination unit is configured to automatically generate a merge code and a merge configuration information after registering the source file name, the reference file name, the reference file entry table name, and the merge logic description information if the source file imported into the data lake involves a merge; when the source file structure changes, the merge reference file changes, or the merge logic is adjusted, adjust the corresponding registration information and automatically generate the latest merge code and merge configuration information; determine a target reference file corresponding to the currently processed target source file based on the pre-set merge configuration information; wherein the target source file has previously completed a target job batch run, and the job status is a stopped running state; the target reference file has previously been loaded; the merge configuration information includes the correspondence between the source file and the reference file of the merged data; and the target reference file includes the number of loaded records; a decomposition unit, configured to perform a backup process on the target source file, and if the backup process is determined to be successful, decomposition the target source file according to the target reference file; A recovery unit is configured to, if it is determined that the unmerge process is successful, verify the unmerge process result according to the backup process result, and if the verification result is that the verification passes, restore the stopped running state of the target job to the started running state; The target source file has already completed the target job batch running, and the job status is stopped, including: If the target job corresponding to the target source file completes batch operation on T-1, the job status is set to stopped; T is the day for withdrawing and merging data processing, and T-1 is the day before the day for withdrawing and merging data processing; The step of determining the target reference file corresponding to the currently processed target source file according to the preset merge configuration information includes: According to the preset merge configuration information, a polling processing method is used to determine the target reference file corresponding to the currently processed target source file.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images