Data merging method and device, equipment and storage medium
By determining the real time range of the merge process in the data lake and data warehouse environment and filtering log files, the version errors and duplicate merge problems during writing and merging concurrency are solved, and the accuracy and consistency of data merge is achieved.
Patent Information
- Application Number
- CN202411744174.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-30
- Publication Date
- 2025-03-11
AI Technical Summary
In data lake and data warehouse environments, when writing incremental log files and merge incremental log files are concurrent, it is easy to cause the target modified base file version errors or repeated merging, affecting the accuracy of the merge result.
Ensure the accuracy of the merge process's accuracy by determining the real time range for the process start of the merge process and filtering log files that meet the preset merge conditions based on this time range.
It effectively avoids duplicate merging and version errors, improves the accuracy and consistency of data merging, and ensures the accuracy of the merging process.
Smart Images
Figure CN119597779A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and more specifically, to a data merging method, device, equipment and storage medium. Background Art
[0002] Merge-On-Read (MOR) is a strategy used in data processing and storage when reading data, especially in data lake and data warehouse environments. Specifically, in the merge-on-read strategy, the newly inserted data is first stored in an incremental log file, such as a delta log file. When the data needs to be read, the database system reads the incremental log and the basic file, and merges the incremental log and the basic file to obtain the latest data file. The basic file can be understood as the historical data file stored before the new data is inserted.
[0003] However, in data lake and data warehouse environments, each time a log file is written, the version of the target modified base file needs to be determined, and each time a log file is merged, a new version of the base file is generated. As a result, in the concurrent process of writing incremental log files and merging incremental log files, the version of the target modified base file written to the log file is incorrect, or log files that are not in this merge process are repeatedly merged, thus affecting the accuracy of the result of merging log files. Summary of the invention
[0004] In view of the above problems, the present application provides a data merging method, device, equipment and storage medium to achieve the purpose of accurately merging log files in the concurrent process of writing incremental log files and merging incremental log files. The specific scheme is as follows:
[0005] The first aspect of the present application provides a data merging method, which is applied to a server and includes:
[0006] When the current merge process is in the startup state and the server meets the preset conditions, determining the physical start time of the current merge process;
[0007] Determine the real time range of the process start when the server starts to execute the merge process based on the physical time of the process start and the time error between the physical time and the real time of the server;
[0008] Obtain the real time when the last merge process started, and the file merged by the last merge process as the basic file;
[0009] Determine at least one log file that meets a preset merging condition according to the process start real time range and the last process start real time, wherein the preset merging condition is that the submission time of the log file is within the time range defined by the process start real time range and the last process start real time;
[0010] The basic file and at least one of the log files are merged to obtain a target file corresponding to the current merging process.
[0011] In a possible implementation, determining the process start real time range at which the server starts to execute the current merge process based on the process start physical time and the time error between the physical time and the real time of the server includes:
[0012] Determining the current physical time of the server;
[0013] Determine the physical start time of the process when the server starts to execute the current merge process based on the time error between the current physical time of the server and the real time;
[0014] According to the real time when the process starts and the time error, the physical time when the process starts is corrected to determine the real time range when the process starts for the current merged process.
[0015] In a possible implementation, determining the physical start time of the process at which the server starts to execute the current merge process based on the time error between the physical time of the server and the real time includes:
[0016] Based on the time error between the physical time of the server and the real time, determining the initial process start physical time when the server starts to execute the current merge process;
[0017] Get the real time when the last merge process started;
[0018] Determine whether the difference between the physical time when the initial process starts and the real time when the last process starts is greater than a preset threshold;
[0019] When the difference between the initial process start physical time and the last process start real time is greater than the preset threshold, the initial process start physical time is determined as the process start physical time of the current merged process;
[0020] When the difference between the physical time when the initial process starts and the real time when the last process starts is not greater than the preset threshold, the step of determining the physical time when the server starts to execute the initial process of the merged process based on the time error between the physical time and the real time of the server is performed until the difference between the physical time when the initial process starts and the real time when the last process starts is greater than the preset threshold.
[0021] In a possible implementation, the process of determining the submission time corresponding to the log file includes:
[0022] Determine the time when the log file completes registration of the metadata of the server as the real submission time of the log file;
[0023] The submission real time is corrected based on the time error between the physical time and the real time of the server to obtain a submission time range, and the submission time range is used as the submission time of the log file.
[0024] In a possible implementation, the preset condition is that there is no merge process in execution state in the server.
[0025] A second aspect of the present application provides a data merging device, applied to a server, comprising:
[0026] A real time determination unit, configured to determine the real time of the start of the current merging process when the current merging process is in the startup state and the server meets a preset condition;
[0027] a start time range determining unit, configured to determine a process start real time range for the server to start executing the current merge process based on the process start real time and a time error between the physical time of the server and the real time;
[0028] The last process information acquisition unit is used to acquire the real time when the last process of the last merge process started, and the file merged by the last merge process as the basic file;
[0029] A log file determination unit, configured to determine at least one log file that meets a preset merging condition according to the process start real time range and the last process start real time, wherein the preset merging condition is that the submission time of the log file is within the time range defined by the process start real time range and the last process start real time;
[0030] The file merging unit is used to merge the basic file and at least one of the log files to obtain a target file corresponding to the current merging process.
[0031] In a possible implementation, the start time range determining unit includes:
[0032] A current physical time determining unit, used to determine the current physical time of the server;
[0033] a process start physical time determination unit, configured to determine the process start physical time at which the server starts to execute the current merge process based on a time error between the physical time of the server and the real time;
[0034] The process start real time determining unit is used to correct the process start physical time according to the process start real time and the time error, and determine the process start real time range of the current merged process.
[0035] In a possible implementation, the process starts a physical time determination unit, including:
[0036] an initial process determination subunit, configured to determine the initial process start physical time at which the server starts to execute the current merge process based on a time error between the physical time of the server and the real time;
[0037] The last process time acquisition subunit is used to obtain the real time when the last process of the last merge process started;
[0038] A time judgment subunit, used to judge whether the difference between the physical time when the initial process starts and the real time when the last process starts is greater than a preset threshold;
[0039] a process start physical time determination subunit, configured to determine the initial process start physical time as the process start physical time of the current merged process if the judgment result of the time judgment subunit is yes;
[0040] The time loop determination subunit is used to jump to the initial process determination subunit when the judgment result of the time judgment subunit is no, and execute the step of determining the physical time when the server starts to execute the initial process of the merged process based on the time error between the physical time and the physical time of the server and the real time, until the judgment result of the time judgment subunit is yes.
[0041] A third aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:
[0042] The memory is used to store computer programs;
[0043] The processor is used to execute the computer program so that the electronic device can implement any one of the data merging methods.
[0044] A fourth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any one of the data merging methods described.
[0045] The data merging method provided in the embodiment of the present application uses, among all the log files written to the server, the log files whose submission time is between the time range specified by the start time of the last merge process and the start time of this merge process as the log files used for merging in this merge process, thereby filtering out the log files written to the server by the writing process concurrent with the current merge process, avoiding repeated merging.
[0046] In addition, the present application takes into account the error between the actual start time of the merge process and the machine execution time of the server, and thereby determines the reference time range for the server to start executing this merge process, thereby correcting the start time of this merge process, and filtering log files based on the reference time range and the start time of the last merge process, which can improve the accuracy of the resulting log files and achieve the purpose of avoiding missing log files belonging to this merge process as much as possible.
[0047] Based on this, while supporting the concurrent merging process and writing process, this application accurately filters out the log files used for this merging process through time screening and time error correction, so as to achieve the purpose of no duplication and no omission, and improve the accuracy of data merging. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0049] Figure 1 A flowchart of a method for implementing data merging provided in an embodiment of the present application;
[0050] Figure 2 A schematic diagram of the structure of a data merging device provided in an embodiment of the present application;
[0051] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0052] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0053] In the prior art, in order to realize data lake or data warehouse, the process locking method is usually adopted to separate the time of writing process and merging process, so that only one process is executed at the same time, avoiding conflicts between writing process and concurrent process, which may lead to incorrect file merging results. However, although the prior art can avoid the problem of file merging errors, it cannot achieve concurrency of writing and merging, which makes the server's response efficiency to business low.
[0054] Therefore, the embodiment of the present application provides a data merging method, which is applied to a server running a data lake, data warehouse, or other data storage environment, and supports the concurrency of the writing process and the merging process while ensuring the correctness of the file merging. Figure 1 , a flow chart of a method for implementing data merging provided in an embodiment of the present application, and the specific steps of the method for data merging are described in conjunction with the following embodiment.
[0055] Step S110, when the current merge process is in the startup state and the server meets the preset conditions, determine the physical start time of the current merge process.
[0056] Step S120, based on the physical time of the process start and the time error between the physical time of the server and the real time, determine the real time range of the process start when the server starts to execute the current merge process.
[0057] The embodiment of the present application responds to the start request of this merge process, first checks whether there is a merge process in execution state in the server, and starts this merge process when the preset merge condition that there is no merge process in execution state in the server is met, and records the real start time of this merge process.
[0058] Among them, real time refers to the 24-hour timekeeping method in daily life, which is not affected by any specific device or system and exists objectively. The physical time of a machine (such as a server) is the time converted by the clock cycle and count value of the internal chip of the simulation system. Each computer has its own physical time. However, due to the clock setting of the server, there is a certain error between the actual physical time executed by the server and the real time.
[0059] Based on this, the real time for starting the merge process fed back by the server may not be the accurate process start time. Therefore, the embodiment of the present application determines a real time range for starting the merge process based on the time error between the physical time of the server and the real time, that is, the real time range for the start of the process, and regards all moments within this time range as all possible merge start times.
[0060] In one possible implementation, the process of determining the real start time range of the process when the server starts to execute the current merge process may include: determining the current physical time of the server; determining the physical start time of the process when the server starts to execute the current merge process based on the time error between the current physical time of the server and the real time; correcting the physical start time of the process based on the real start time of the process and the time error, and determining the real start time range of the process of the current merge process.
[0061] First, obtain the current physical time of the server where the merge process is located, and set the physical time at which the merge starts. Optionally, the physical time at which the merge starts = the current physical time of the server where the merge process is located - 2 × time error.
[0062] Furthermore, considering that there is an error between the physical time and the real time of the server where the merge process is located, the real time range of the merge start time is determined, such as: [real time of process start - 3× time error, real time of process start - time error].
[0063] Furthermore, in order to prevent the extreme situation where the interval between two merge processes is very short, resulting in confusion in log file merging and concurrent merging processes, in another possible implementation, the process of determining the physical start time of the process when the server starts to execute this merge process can include: based on the time error between the server's physical time and the real time, determining the initial process start physical time when the server starts to execute this merge process; obtaining the last process start real time of the last merge process; and determining whether the difference between the initial process start physical time and the last process start real time is greater than a preset threshold.
[0064] When the difference between the initial process start physical time and is greater than a preset threshold, is determined as the process start physical time of the current merged process.
[0065] When the difference between the physical time when the initial process starts and the real time when the last process starts is not greater than the preset threshold, the steps of determining the physical time when the server starts to execute the initial process of this merge process based on the time error between the physical time and the real time of the server are executed until the difference between the physical time when the initial process starts and the real time when the last process starts is greater than the preset threshold.
[0066] Optionally, the physical time of the start of the merge of this merge process, that is, the physical time of the initial process start, is determined based on "merge start time = current physical time of the server where the merge process is located - 2 × time error". Compare the initial process start physical time that is initially set for this merge process with the real time when the last process started to determine whether the two merge processes are similar. It is understandable that due to the error between physical time and real time, when comparing the merge start times of two merge processes, the time error between physical time and real time can also be superimposed. For example, it is determined that the physical time of the initial process start > the real time of the last process start + 2 × time error. Optionally, when the physical time of the last merge process start is still available, the two physical times can also be compared to determine whether the time interval between the two merge processes is small.
[0067] If the time interval between two merge processes is small, you can jump back to the time error between the server's physical time and the real time, determine the step of the physical time when the server starts to execute the initial process of this merge process, and re-determine the physical time when this merge process starts. For example, based on the last determined physical time of the initial process start, delay it by two or more time errors, thereby widening the interval between the two merge processes and avoiding concurrent merge processes.
[0068] Step S130, obtaining the real time when the last merge process started and the file merged by the last merge process as the basic file.
[0069] Step S140: determining at least one log file that meets a preset merging condition according to the process start real time range and the last process start real time.
[0070] Based on the actual start time range of this merge process determined in the above steps, the log files that need to be merged in this merge process are determined from the server. It can be understood that the log files merged by each merge process are new log files generated between the start time of the last merge process and the start time of this merge process, and the new log files generated between this time range are all modifications to the files obtained by the last merge process. Therefore, the files obtained by the last merge process are obtained as the basic files for this merge process.
[0071] Based on this, it is also necessary to obtain the real start time of the last merge process and determine the preset merge condition for filtering log files. The preset merge condition is that the submission time of the log file is within the time range limited by the real start time range of the process and the real start time of the last process.
[0072] Regarding the log files stored in the server, in the process of writing log files, the embodiment of the present application names each log file according to its submission time. When filtering log files, the submission time in the name of the log file can be compared with the preset merge conditions to determine whether the log file belongs to this merge process.
[0073] Specifically, the process of writing log files can refer to the example below. First, when the data lake receives an instruction to modify data, it determines which partition in the distributed structure of the database the data belongs to, and then determines which file group in the partition to write, where the file group is uniquely marked by the file id. In an embodiment of the present application, a log file is generated in advance, where the name of the log file can be set to: ${file_id}.log.${number}_${task_token}, where file_id is used to uniquely determine the file group, number is used to distinguish multiple files split from the same write, and task_token is used to uniquely mark the write of a certain operation of the client to avoid the same name for different clients or the same name for different operations (transactions) of the same client.
[0074] Furthermore, the content of the data modified by the client this time is written into the above-mentioned pre-generated log file. After the data is written into the log file, the log file is written into the metadata to complete the registration of the log file. At this time, it indicates that the log file has been submitted successfully. Subsequent queries need to read the metadata first to determine whether the log file is in a submitted state or a completed state. Only log files in a completed state can participate in the merge process.
[0075] Among them, the process of determining the submission time may include: determining the time when the log file completes registration in the metadata of the server, as the time error between the physical time and the real time of the log file, correcting the real submission time, obtaining the submission time range, and using the submission time range as the submission time of the log file.
[0076] Optionally, the range of the submission time can be: [real submission time - time error, real submission time + time error].
[0077] Based on this, when filtering log files, log files whose submission time range is within the time range limited by the real time range of the process start and the real time of the last process start can be determined as log files of this merge process.
[0078] Based on this, continuing the above example, we use "the process start time of this merge process >= the submission time of the log file > the process start time of the last merge process" as the preset merge condition to filter the log files.
[0079] Because the process start time range of this merge process is [real process start time - 3 × time error, real process start time - time error], the submission time range of the log file is: [real submission time - time error, + time error]. After the start of this merge process, if there are still log writes completed, then the real submission time of the log file at this time > the real process start time. Therefore, the submission time of the log file submitted after the start of this merge process will not be greater than the start time of this merge process. This ensures that the log file processing that meets the condition of "process start time of this merge process >= submission time of log file >" will not be missed.
[0080] Step S150: merge the basic file and at least one log file to obtain a target file corresponding to this merge process.
[0081] The log file and the base file determined above are merged to generate a target file, which is the base file for the next merge process. The merge process can be understood as synchronizing the editing operations such as data insertion, data modification, and data deletion recorded in the log file to the base file to complete the editing of the base file. After the log file merge is completed, the status of the merged base file is set to completed merge in the metadata, so as to facilitate the judgment of step S110 when the next merge process is started.
[0082] To summarize, the data merging method provided in the embodiment of the present application uses, among all the log files written to the server, the log files whose submission time is between the time range specified by the start time of the last merge process and the start time of this merge process as the log files used for merging in this merge process, thereby filtering out the log files written to the server by the writing process concurrent with the current merge process, and avoiding repeated merging.
[0083] In addition, the present application takes into account the error between the actual time when the merge process starts and the physical time of the server's machine execution, and thereby determines the reference time range for the server to start executing this merge process, thereby correcting the start time of this merge process, and filtering log files based on the reference time range and the start time of the last merge process, which can improve the accuracy of the resulting log files and achieve the purpose of avoiding missing log files belonging to this merge process as much as possible.
[0084] Based on this, while supporting the concurrent merging process and writing process, this application accurately filters out the log files used for this merging process through time screening and time error correction, so as to achieve the purpose of no duplication and no omission, and improve the accuracy of data merging.
[0085] The data merging device provided in the embodiment of the present application is described below. The data merging device described below and the data merging method described above can refer to each other.
[0086] First, combine Figure 2 , the data merging device is introduced, such as Figure 2 As shown, the data merging device may include:
[0087] The real time determination unit 100 is used to determine the real time of the start of the current merging process when the current merging process is in the startup state and the server meets the preset conditions;
[0088] A start time range determining unit 200 is used to determine the process start real time range for the server to start executing the current merge process based on the process start real time and the time error between the physical time of the server and the real time;
[0089] The last process information acquisition unit 300 is used to acquire the real time when the last process of the last merge process started, and the file merged by the last merge process as the basic file;
[0090] The log file determination unit 400 is used to determine at least one log file that meets a preset merging condition according to the process start real time range and the last process start real time, wherein the preset merging condition is that the submission time of the log file is within the time range defined by the process start real time range and the last process start real time;
[0091] The file merging unit 500 is used to merge the basic file and at least one of the log files to obtain a target file corresponding to the current merging process.
[0092] In a possible implementation, the start time range determining unit includes:
[0093] A current physical time determining unit, used to determine the current physical time of the server;
[0094] a process start physical time determination unit, configured to determine the process start physical time at which the server starts to execute the current merge process based on a time error between the physical time of the server and the real time;
[0095] The process start real time determining unit is used to correct the process start physical time according to the process start real time and the time error, and determine the process start real time range of the current merged process.
[0096] In a possible implementation, the process starts a physical time determination unit, including:
[0097] The process starts a physical time determination unit, including:
[0098] an initial process determination subunit, configured to determine the initial process start physical time at which the server starts to execute the current merge process based on a time error between the physical time of the server and the real time;
[0099] The last process time acquisition subunit is used to obtain the real time when the last process of the last merge process started;
[0100] A time judgment subunit, used to judge whether the difference between the physical time when the initial process starts and the real time when the last process starts is greater than a preset threshold;
[0101] a process start physical time determination subunit, configured to determine the initial process start physical time as the process start physical time of the current merged process if the judgment result of the time judgment subunit is yes;
[0102] The time loop determination subunit is used to jump to the initial process determination subunit when the judgment result of the time judgment subunit is no, and execute the steps of determining the physical time when the server starts to execute the initial process of the merged process based on the time error between the physical time and the real time of the server, until the judgment result of the time judgment subunit is yes.
[0103] In a possible implementation, the functional unit for determining the submission time corresponding to the log file includes:
[0104] a submission real time determination unit, used to determine the time when the log file completes registration of the metadata of the server as the submission real time of the log file;
[0105] The submission time determination unit is used to correct the submission real time based on the time error between the physical time and the real time of the server to obtain a submission time range, and use the submission time range as the submission time of the log file.
[0106] In a possible implementation, the preset condition is that there is no merge process in execution state in the server.
[0107] To summarize, the embodiment of the present application writes all log files to the server, and uses the log files whose submission time is between the time range specified by the start time of the last merge process and the start time of this merge process as the log files used for merging in this merge process, thereby filtering out the log files written to the server by the writing process concurrent with this merge process to avoid repeated merging.
[0108] In addition, the present application takes into account the error between the actual time when the merge process starts and the physical time of the server's machine execution, and thereby determines the reference time range for the server to start executing this merge process, thereby correcting the start time of this merge process, and filtering log files based on the reference time range and the start time of the last merge process, which can improve the accuracy of the resulting log files and achieve the purpose of avoiding missing log files belonging to this merge process as much as possible.
[0109] The data merging device provided in the embodiment of the present application can be applied to electronic devices. Figure 3 A schematic diagram of the structure of an electronic device is shown, referring to Figure 3 , the structure of the electronic device may include: at least one processor 10, at least one memory 20, at least one communication bus 30 and at least one communication interface 40;
[0110] In the embodiment of the present application, the number of the processor 10, the memory 20, the communication bus 30, and the communication interface 40 is at least one, and the processor 10, the memory 20, and the communication interface 40 communicate with each other through the communication bus 30;
[0111] The processor 10 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention;
[0112] The memory 20 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory;
[0113] The memory stores a program, and the processor can call the program stored in the memory, and the program is used to implement each processing flow in the aforementioned data merging method.
[0114] An embodiment of the present application further provides a computer storage medium, which can store a program suitable for execution by a processor, wherein the program is used to implement each processing flow in the aforementioned data merging method.
[0115] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.
[0116] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0117] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0118] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A data merging method, characterized in that: Applicable to servers, including: When the current merge process is in the startup state and the server meets the preset conditions, determining the physical start time of the current merge process; Determine the real time range of the process start when the server starts to execute the merge process based on the physical time of the process start and the time error between the physical time and the real time of the server; Obtain the real time when the last merge process started, and the file merged by the last merge process as the basic file; Determine at least one log file that meets a preset merging condition according to the process start real time range and the last process start real time, wherein the preset merging condition is that the submission time of the log file is within the time range defined by the process start real time range and the last process start real time; The basic file and at least one of the log files are merged to obtain a target file corresponding to the current merging process.
2. The data merging method according to claim 1, characterized in that: The determining, based on the physical time of the process start and the time error between the physical time of the server and the real time, the real time range of the process start at which the server starts to execute the merge process includes: Determining the current physical time of the server; Determine the physical start time of the process when the server starts to execute the current merge process based on the time error between the current physical time of the server and the real time; According to the real time when the process starts and the time error, the physical time when the process starts is corrected to determine the real time range when the process starts for the current merged process.
3. The data merging method according to claim 2, characterized in that: The determining, based on the time error between the physical time of the server and the real time, the physical time at which the server starts to execute the process of the current merge process comprises: Based on the time error between the physical time of the server and the real time, determining the initial process start physical time when the server starts to execute the current merge process; Get the real time when the last merge process started; Determine whether the difference between the physical time when the initial process starts and the real time when the last process starts is greater than a preset threshold; When the difference between the initial process start physical time and the last process start real time is greater than the preset threshold, the initial process start physical time is determined as the process start physical time of the current merged process; When the difference between the physical time when the initial process starts and the real time when the last process starts is not greater than the preset threshold, the step of determining the physical time when the server starts to execute the initial process of the merged process based on the time error between the physical time and the real time of the server is performed until the difference between the physical time when the initial process starts and the real time when the last process starts is greater than the preset threshold.
4. The data merging method according to claim 1, characterized in that: The process of determining the submission time corresponding to the log file includes: Determine the time when the log file completes registration of the metadata of the server as the real submission time of the log file; The submission real time is corrected based on the time error between the physical time and the real time of the server to obtain a submission time range, and the submission time range is used as the submission time of the log file.
5. The data merging method according to claim 1, characterized in that: The preset condition is that there is no merge process in execution state in the server.
6. A data merging device, characterized in that: Applicable to servers, including: A real time determination unit, configured to determine the real time of the start of the current merging process when the current merging process is in the startup state and the server meets a preset condition; a start time range determining unit, configured to determine a process start real time range for the server to start executing the current merge process based on the process start real time and a time error between the physical time of the server and the real time; The last process information acquisition unit is used to acquire the real time when the last process of the last merge process started, and the file merged by the last merge process as the basic file; a log file determination unit, configured to determine at least one log file that meets a preset merging condition according to the process start real time range and the last process start real time, wherein the preset merging condition is that the submission time of the log file is within the time range defined by the process start real time range and the last process start real time; The file merging unit is used to merge the basic file and at least one of the log files to obtain a target file corresponding to the current merging process.
7. The data merging device according to claim 6, characterized in that: The starting time range determining unit comprises: A current physical time determining unit, used to determine the current physical time of the server; a process start physical time determination unit, configured to determine the process start physical time at which the server starts to execute the current merge process based on a time error between the physical time of the server and the real time; The process start real time determining unit is used to correct the process start physical time according to the process start real time and the time error, and determine the process start real time range of the current merged process.
8. The data merging device according to claim 7, characterized in that: The process starts a physical time determination unit, including: an initial process determination subunit, configured to determine the initial process start physical time at which the server starts to execute the current merge process based on a time error between the physical time of the server and the real time; The last process time acquisition subunit is used to obtain the real time when the last process of the last merge process started; A time judgment subunit, used to judge whether the difference between the physical time when the initial process starts and the real time when the last process starts is greater than a preset threshold; a process start physical time determination subunit, configured to determine the initial process start physical time as the process start physical time of the current merged process if the judgment result of the time judgment subunit is yes; The time loop determination subunit is used to jump to the initial process determination subunit when the judgment result of the time judgment subunit is no, and execute the steps of determining the physical time when the server starts to execute the initial process of the merged process based on the time error between the physical time and the real time of the server, until the judgment result of the time judgment subunit is yes.
9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the data merging method as described in any one of claims 1 to 5.
10. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the data merging method as described in any one of claims 1 to 5.