Log synchronization method, device, and computer-readable storage medium

Abstract

The present application relates to the technical field of Fintech (Financial Technology). The present application discloses a log synchronization method, device and computer readable storage medium. The present application introduces the concept of date division in the synchronization process, divides the log data to be synchronized into batches according to the date to which the log data belongs, then defines the corresponding date sub-table type, and executes the log synchronization task by using the type, so that the dynamic synchronization of log data can be realized in the time dimension, thereby improving the intelligence of the log data synchronization operation. The synchronization operation trigger event of the log data is automatically identified, so that the user only needs to perform the triggering operation, or even does not need to manually trigger. The system can automatically start creating and executing the corresponding log data synchronization task according to the synchronization event automatically triggered by the preset program, thereby saving the manual operation required in the log synchronization process in the prior art, and further improving the intelligent level of the log data synchronization operation.

Description

Technical Field

[0001] This invention relates to the field of financial technology (Fintech), and more particularly to log synchronization methods, devices, and computer-readable storage media. Background Technology

[0002] With the development of computer technology, more and more technologies (big data, distributed systems, blockchain, artificial intelligence, etc.) are being applied in the financial sector. The traditional financial industry is gradually transforming into Fintech. However, due to the security and real-time requirements of the financial industry, higher demands are being placed on automated testing technologies. In the field of automated testing, daily continuous regression testing generates a large amount of automated test logs. These logs are generated by the automated testing platform and stored in the source data storage source. Because automated test logs need to be retrieved and analyzed quickly, it is necessary to synchronize the log data from the source data storage source to the target data storage source.

[0003] In existing log synchronization solutions, the same table is used regardless of when the synchronization operation is performed. This table records various information related to log synchronization updates, and the log data is updated by updating the corresponding data in the table each time. However, in the above synchronization solution, each synchronization update requires manual synchronization or modification of corresponding configurations, resulting in a lack of intelligence in log data synchronization operations in the field of automated testing. Summary of the Invention

[0004] The main objective of this invention is to propose a log synchronization method, device, and computer-readable storage medium, aiming to solve the technical problem of low intelligence in log data synchronization operations in the field of automated testing.

[0005] To achieve the above objectives, the present invention provides a log synchronization method, the log synchronization method comprising:

[0006] Identify log synchronization trigger events with a synchronization type of date-based table partitioning, wherein the date-based table partitioning type determines the synchronization type of the log synchronization task based on the date to which the log data belongs;

[0007] Based on the log synchronization trigger event, create and execute a target log synchronization task corresponding to the target date of the log synchronization trigger event, so as to synchronize the log data to be synchronized in the target log synchronization task to the target data source.

[0008] Optionally, the target log synchronization task includes a daily log synchronization task and / or a cross-day log synchronization task.

[0009] The step of creating and executing a target log synchronization task corresponding to the target date of the log synchronization trigger event, in order to synchronize the log data to be synchronized in the target log synchronization task to the target data source, includes:

[0010] The pre-configured synchronization metadata is determined based on the log synchronization trigger event, and the test log data for the current day and / or the test log data for the next day are obtained, wherein the test log data for the next day is log data with a time span of multiple days;

[0011] Based on the synchronized metadata and the test log data of the day, and / or the test log data of the day, create a log synchronization task for the day and / or a log synchronization task for the day corresponding to the date of the target;

[0012] Execute the same-day log synchronization task and / or cross-day log synchronization task to synchronize the log data to be synchronized in the same-day log synchronization task and / or cross-day log synchronization task to the target data source.

[0013] Optionally, the step of creating a log synchronization task for the current day and / or a log synchronization task for the day corresponding to the target date based on the synchronization metadata and the test log data for the current day, and / or the test log data for the next day includes:

[0014] The data table prefix is ​​obtained based on the synchronized metadata, and the date to which the target belongs is formatted to obtain built-in variables;

[0015] Based on the test log data of the day, the data table prefix, and the built-in variables, a log synchronization data table for the day is generated to create the log synchronization task for the day; and / or,

[0016] Based on the synchronization metadata, traverse the historical log synchronization record queue before the date to which the target belongs, and obtain the synchronization date table and the latest synchronization tracking column value from the historical log synchronization record queue;

[0017] The cross-day log synchronization task is created based on the cross-day test log data, the synchronization date table, and the latest synchronization tracking column value.

[0018] Optionally, before the step of identifying log synchronization trigger events of the date-based table type, the method further includes:

[0019] Receive initial configuration information sent by the user, wherein the initial configuration information includes initial source data source configuration information, initial target data source configuration information, and initial synchronization task configuration information;

[0020] The initial configuration information is used to initialize the synchronization metadata so that the configured synchronization metadata can be reused in the next log synchronization task.

[0021] Optionally, the step of executing the same-day log synchronization task and / or the cross-day log synchronization task to synchronize the log data to be synchronized in the same-day log synchronization task and / or the cross-day log synchronization task to the target data source includes:

[0022] To acquire the first distributed lock and / or the second distributed lock for the same day's log synchronization task and / or the cross-day log synchronization task;

[0023] The test log data for the current day and / or the test log data for the next day are segmented according to the data slicing method, so as to push the test log data for the current day and / or the test log data for the next day to the target data source in batches;

[0024] When the daily test log data and / or cross-day test log data are detected to be pushed out, the first distributed lock and / or the second distributed lock are released.

[0025] Optionally, after the step of releasing the first distributed lock and / or the second distributed lock when the test log data for the current day and / or the test log data for the next day has been pushed out, the method further includes:

[0026] Iterate through the synchronization success record table obtained after the test log data of the day and / or the test log data of the cross-day are pushed, and obtain the tracking column value of the last synchronization success in the synchronization success record table;

[0027] Based on the source data table and target data table in the synchronization metadata, obtain the source tracking column value and target tracking column value respectively, and determine whether the source tracking column value and target tracking column value are both greater than the tracking column value of the last successful synchronization.

[0028] If so, the log data in the target data source is updated synchronously to complete the verification of the daily log synchronization task and / or cross-day log synchronization task.

[0029] Optionally, the step of identifying log synchronization trigger events of the date-based table partitioning type includes:

[0030] When new node information for a synchronization task configuration is detected, the log synchronization trigger event is determined based on the new node information. The triggering conditions for the log synchronization trigger event include manual triggering and / or automatic triggering based on preset rules.

[0031] Obtain the synchronization type field of the log synchronization trigger event, identify the synchronization type of the log synchronization trigger event to determine that the synchronization type is a date table partitioning type.

[0032] Optionally, after the step of creating and executing a target log synchronization task corresponding to the target date of the log synchronization trigger event, the method further includes:

[0033] When a log data synchronization failure is detected, retrieve the log data of the failed synchronization.

[0034] Log data that fails to synchronize is pushed to the retry data synchronization queue for the corresponding date, so that the log data in the retry data synchronization queue is resynchronized to the target data source in turn.

[0035] Furthermore, to achieve the above objectives, the present invention also provides a log synchronization device, the log synchronization device comprising:

[0036] The synchronization trigger identification module is used to identify log synchronization trigger events with a synchronization type of date-based table partitioning, wherein the date-based table partitioning type determines the synchronization type of the log synchronization task based on the date to which the log data belongs;

[0037] The synchronization task execution module is used to create and execute a target log synchronization task corresponding to the target date of the log synchronization trigger event, so as to synchronize the log data to be synchronized in the target log synchronization task to the target data source.

[0038] Optionally, the target log synchronization task includes a daily log synchronization task and / or a cross-day log synchronization task, and the synchronization task execution module includes:

[0039] The log data acquisition unit is used to determine the pre-configured synchronization metadata based on the log synchronization trigger event, and to acquire the test log data for the current day and / or the test log data for the next day, wherein the test log data for the next day is log data with a time span of multiple days;

[0040] The synchronization task creation unit is used to create a log synchronization task for the current day and / or a log synchronization task for the cross-day based on the synchronization metadata and the test log data for the current day, and / or the test log data for the cross-day.

[0041] The synchronization task execution unit is used to execute the daily log synchronization task and / or cross-day log synchronization task to synchronize the log data to be synchronized in the daily log synchronization task and / or cross-day log synchronization task to the target data source.

[0042] Optionally, the synchronization task creation unit is further configured to:

[0043] The data table prefix is ​​obtained based on the synchronized metadata, and the date to which the target belongs is formatted to obtain built-in variables;

[0044] Based on the test log data of the day, the data table prefix, and the built-in variables, a log synchronization data table for the day is generated to create the log synchronization task for the day; and / or,

[0045] Based on the synchronization metadata, traverse the historical log synchronization record queue before the date to which the target belongs, and obtain the synchronization date table and the latest synchronization tracking column value from the historical log synchronization record queue;

[0046] The cross-day log synchronization task is created based on the cross-day test log data, the synchronization date table, and the latest synchronization tracking column value.

[0047] Optionally, the log synchronization device further includes:

[0048] The configuration information receiving module is used to receive initial configuration information sent by the user, wherein the initial configuration information includes initial source data source configuration information, initial target data source configuration information, and initial synchronization task configuration information;

[0049] The synchronization data configuration module is used to initialize the synchronization metadata configuration based on the initial configuration information, so as to reuse the configured synchronization metadata in the next log synchronization task.

[0050] Optionally, the synchronous task execution unit is further configured to:

[0051] To acquire the first distributed lock and / or the second distributed lock for the same day's log synchronization task and / or the cross-day log synchronization task;

[0052] The test log data for the current day and / or the test log data for the next day are segmented according to the data slicing method, so as to push the test log data for the current day and / or the test log data for the next day to the target data source in batches;

[0053] When the daily test log data and / or cross-day test log data are detected to be pushed out, the first distributed lock and / or the second distributed lock are released.

[0054] Optionally, the synchronous task execution unit is further configured to:

[0055] Iterate through the synchronization success record table obtained after the test log data of the day and / or the test log data of the cross-day are pushed, and obtain the tracking column value of the last synchronization success in the synchronization success record table;

[0056] Based on the source data table and target data table in the synchronization metadata, obtain the source tracking column value and target tracking column value respectively, and determine whether the source tracking column value and target tracking column value are both greater than the tracking column value of the last successful synchronization.

[0057] If so, the log data in the target data source is updated synchronously to complete the verification of the daily log synchronization task and / or cross-day log synchronization task.

[0058] Optionally, the synchronization trigger identification module includes:

[0059] A new node detection unit is used to determine the log synchronization trigger event based on the new node information when new node information with a synchronization task configuration is detected. The triggering conditions of the log synchronization trigger event include manual triggering and / or automatic triggering based on preset rules.

[0060] The type field identification unit is used to obtain the synchronization type field of the log synchronization trigger event, identify the synchronization type field, and determine that the synchronization type of the log synchronization trigger event is a date table type.

[0061] Optionally, the synchronous task execution module further includes:

[0062] The synchronization failure detection unit is used to obtain the log data of the synchronization failure when a log data synchronization failure is detected.

[0063] The failed data retry unit is used to push the log data of the synchronization failure to the retry data synchronization queue of the corresponding date, so as to resynchronize the log data in the retry data synchronization queue to the target data source in turn.

[0064] In addition, to achieve the above objectives, the present invention also provides a log synchronization device, the log synchronization device comprising: a memory, a processor, and a log synchronization program stored in the memory and executable on the processor, wherein the log synchronization program, when executed by the processor, implements the steps of the log synchronization method as described above.

[0065] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a log synchronization program, which, when executed by a processor, implements the steps of the log synchronization method as described above.

[0066] This invention provides a log synchronization method, device, and computer-readable storage medium. By introducing the concept of date segmentation into the synchronization process, the log data to be synchronized is divided into batches according to their respective dates. Then, a corresponding date-based table type is defined, and log synchronization tasks are executed using this type. This enables dynamic synchronization of log data over time, thereby improving the intelligence of log data synchronization operations. By automatically identifying log data synchronization operation trigger events, users only need to perform a trigger operation, or even no manual triggering is required. The system can automatically start creating and executing the corresponding log data synchronization tasks based on preset rules (e.g., automatic triggering based on date updates). This eliminates the manual operation required in the log synchronization process of existing technologies, further improving the intelligence level of log data synchronization operations and solving the technical problem of low intelligence in log data synchronization operations in the field of automated testing. Attached Figure Description

[0067] Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of the present invention;

[0068] Figure 2 This is a flowchart illustrating the first embodiment of the log synchronization method of the present invention;

[0069] Figure 3 This is a schematic diagram of the cross-day log synchronization task creation process in the second embodiment of the log synchronization method of the present invention;

[0070] Figure 4 This is a list of source data source configuration parameters in the second embodiment of the log synchronization method of the present invention;

[0071] Figure 5 This is a diagram showing the target data source (configuration parameter list) in the second embodiment of the log synchronization method of the present invention.

[0072] Figure 6 This is a list of configuration parameters for the synchronization task in the second embodiment of the log synchronization method of the present invention;

[0073] Figure 7 This is a schematic diagram of phased synchronization based on data slicing in the second embodiment of the log synchronization method of the present invention;

[0074] Figure 8 This is a schematic diagram of the overall flow of a specific embodiment of the log synchronization method of the present invention in the third embodiment;

[0075] Figure 9 This is a schematic diagram of the functional modules of the log synchronization device of the present invention.

[0076] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0077] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0078] like Figure 1 As shown, Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of the present invention.

[0079] like Figure 1 As shown, the log synchronization device may include: a processor 1001, such as a CPU; a user interface 1003; a network interface 1004; a memory 1005; and a communication bus 1002. The communication bus 1002 is used to establish communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0080] Those skilled in the art will understand that Figure 1 The device structure shown does not constitute a limitation on the device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0081] like Figure 1 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a log synchronization program.

[0082] exist Figure 1 In the device shown, network interface 1004 is mainly used to connect to the backend server and communicate data with it; user interface 1003 is mainly used to connect to the client (programmer's end) and communicate data with it; while processor 1001 can be used to call the log synchronization program stored in memory 1005 and execute the operations in the following log synchronization method:

[0083] Identify log synchronization trigger events with a synchronization type of date-based table partitioning, wherein the date-based table partitioning type determines the synchronization type of the log synchronization task based on the date to which the log data belongs;

[0084] Based on the log synchronization trigger event, create and execute a target log synchronization task corresponding to the target date of the log synchronization trigger event, so as to synchronize the log data to be synchronized in the target log synchronization task to the target data source.

[0085] Furthermore, the target log synchronization task includes a daily log synchronization task and / or a cross-day log synchronization task.

[0086] The step of creating and executing a target log synchronization task corresponding to the target date of the log synchronization trigger event, in order to synchronize the log data to be synchronized in the target log synchronization task to the target data source, includes:

[0087] The pre-configured synchronization metadata is determined based on the log synchronization trigger event, and the test log data for the current day and / or the test log data for the next day are obtained, wherein the test log data for the next day is log data with a time span of multiple days;

[0088] Based on the synchronized metadata and the test log data of the day, and / or the test log data of the day, create a log synchronization task for the day and / or a log synchronization task for the day corresponding to the date of the target;

[0089] Execute the same-day log synchronization task and / or cross-day log synchronization task to synchronize the log data to be synchronized in the same-day log synchronization task and / or cross-day log synchronization task to the target data source.

[0090] Further, the step of creating a log synchronization task for the current day and / or a log synchronization task for the day corresponding to the target date based on the synchronization metadata and the test log data of the current day, and / or the test log data of the next day includes:

[0091] The data table prefix is ​​obtained based on the synchronized metadata, and the date to which the target belongs is formatted to obtain built-in variables;

[0092] Based on the test log data of the day, the data table prefix, and the built-in variables, a log synchronization data table for the day is generated to create the log synchronization task for the day; and / or,

[0093] Based on the synchronization metadata, traverse the historical log synchronization record queue before the date to which the target belongs, and obtain the synchronization date table and the latest synchronization tracking column value from the historical log synchronization record queue;

[0094] The cross-day log synchronization task is created based on the cross-day test log data, the synchronization date table, and the latest synchronization tracking column value.

[0095] Furthermore, prior to the step of identifying a log synchronization trigger event of date-based table type, the processor 1001 may invoke the log synchronization program stored in the memory 1005 and perform the operations in the following log synchronization method:

[0096] Receive initial configuration information sent by the user, wherein the initial configuration information includes initial source data source configuration information, initial target data source configuration information, and initial synchronization task configuration information;

[0097] The initial configuration information is used to initialize the synchronization metadata so that the configured synchronization metadata can be reused in the next log synchronization task.

[0098] Further, the step of executing the same-day log synchronization task and / or the cross-day log synchronization task to synchronize the log data to be synchronized in the same-day log synchronization task and / or the cross-day log synchronization task to the target data source includes:

[0099] To acquire the first distributed lock and / or the second distributed lock for the same day's log synchronization task and / or the cross-day log synchronization task;

[0100] The test log data for the current day and / or the test log data for the next day are segmented according to the data slicing method, so as to push the test log data for the current day and / or the test log data for the next day to the target data source in batches;

[0101] When the daily test log data and / or cross-day test log data are detected to be pushed out, the first distributed lock and / or the second distributed lock are released.

[0102] Furthermore, after the step of releasing the first distributed lock and / or the second distributed lock when the daily test log data and / or cross-day test log data have been pushed out, the processor 1001 can be used to call the log synchronization program stored in the memory 1005 and perform the operations in the following log synchronization method:

[0103] Iterate through the synchronization success record table obtained after the test log data of the day and / or the test log data of the cross-day are pushed, and obtain the tracking column value of the last synchronization success in the synchronization success record table;

[0104] Based on the source data table and target data table in the synchronization metadata, obtain the source tracking column value and target tracking column value respectively, and determine whether the source tracking column value and target tracking column value are both greater than the tracking column value of the last successful synchronization.

[0105] If so, the log data in the target data source is updated synchronously to complete the verification of the daily log synchronization task and / or cross-day log synchronization task.

[0106] Furthermore, the step of identifying log synchronization trigger events of the date-based table partitioning type includes:

[0107] When new node information for a synchronization task configuration is detected, the log synchronization trigger event is determined based on the new node information. The triggering conditions for the log synchronization trigger event include manual triggering and / or automatic triggering based on preset rules.

[0108] Obtain the synchronization type field of the log synchronization trigger event, identify the synchronization type of the log synchronization trigger event to determine that the synchronization type is a date table partitioning type.

[0109] Furthermore, after the step of creating and executing a target log synchronization task corresponding to the target date of the log synchronization trigger event, the processor 1001 can call the log synchronization program stored in the memory 1005 and perform the operations in the following log synchronization method:

[0110] When a log data synchronization failure is detected, retrieve the log data of the failed synchronization.

[0111] Log data that fails to synchronize is pushed to the retry data synchronization queue for the corresponding date, so that the log data in the retry data synchronization queue is resynchronized to the target data source in turn.

[0112] Based on the above hardware structure, an embodiment of the log synchronization method of the present invention is proposed.

[0113] Continuous regression automated testing generates a large number of automated test logs every day. These logs are generated by the automated testing platform and stored in a MySQL database. Since the automated test logs need to be retrieved and analyzed quickly, it is necessary to synchronize the MySQL log data to the target data source. However, the existing technical solution uses logstash-input-jdbc (a plugin) to achieve real-time synchronization of MySQL log data, but this solution cannot solve the following two problems: (1) It cannot achieve real-time synchronization of dynamic date partitioning based on the data time field; (2) For log data with future dates that do not exist, manual intervention is required to modify the synchronization configuration.

[0114] To address the aforementioned issues, this invention provides a log synchronization method. By introducing the concept of date segmentation during the synchronization process, log data to be synchronized is batched according to its date, and a corresponding date-based table type is defined. Log synchronization tasks are then executed using this type, enabling dynamic synchronization of log data over time and thus improving the intelligence of the log data synchronization operation. Furthermore, by automatically identifying log data synchronization operation trigger events, the system can automatically create and execute corresponding log data synchronization tasks based on preset rules (e.g., automatic triggering based on date updates), eliminating the manual operation required in existing log synchronization processes. This further enhances the intelligence level of log data synchronization operations and solves the technical problem of low intelligence in log data synchronization operations in the field of automated testing.

[0115] Reference Figure 2 , Figure 2 This is a flowchart illustrating a first embodiment of the log synchronization method of the present invention. The log synchronization method includes:

[0116] Step S10: Identify log synchronization trigger events with a synchronization type of date-based table partitioning, wherein the date-based table partitioning type determines the synchronization type of the log synchronization task based on the date to which the log data belongs;

[0117] In this embodiment, the method is applied to a terminal device. Log data refers to the automated test logs generated during automated testing. The automated test logs are stored in date-separated tables, named item_execute_status_[date]. For example, the automated test log for February 7, 2021, is stored in the table item_execute_status_[20210207]. This application uses a predefined date format to convert the time information in the log data into a specific date format, thereby treating the synchronization task corresponding to this log data with the specific date format as a date-separated table type synchronization task. The date to which the log data belongs refers to the date to which the log data belongs in business logic. The log synchronization trigger event refers to the event used to trigger the log synchronization task.

[0118] The terminal first needs to listen for log synchronization trigger events. These events can be initiated by the user or by the terminal itself according to preset rules. For example, the user can initiate the event initially and then set corresponding timed synchronization frequency parameters. Afterward, the terminal can automatically initiate and execute log data synchronization tasks at the appropriate times based on these parameters to achieve dynamic synchronization updates. When the terminal detects a log synchronization trigger event, it needs to determine whether the corresponding synchronization task type is a date-based table partitioning type. If so, it follows the subsequent steps in this application; otherwise, it executes the synchronization task using the conventional log synchronization method.

[0119] Step S20: Based on the log synchronization trigger event, create and execute a target log synchronization task corresponding to the target date of the log synchronization trigger event, so as to synchronize the log data to be synchronized in the target log synchronization task to the target data source.

[0120] In this embodiment, the target date refers to the current date of the log synchronization trigger task event, and the target log synchronization task may include one or more different types of synchronization tasks. Because in actual testing, in addition to the regular same-day log data that starts and ends on the same day, there may also be cross-day log data with inconsistent start and end dates. Therefore, the target synchronization task may include the same-day log synchronization task corresponding to the current date, as well as cross-day log synchronization tasks preceding the current date.

[0121] The terminal determines the target date of the currently monitored log synchronization trigger event and creates one or more different types of target log synchronization tasks based on the event. These tasks can be created using pre-configured parameters corresponding to the event. The terminal obtains the log data to be synchronized and executes the created target log synchronization tasks, pushing the log data from the source data source (e.g., MySQL) to the target data source (e.g., Elasticsearch) to complete the target synchronization task.

[0122] In addition, during the execution of synchronization tasks, corresponding failure retry mechanisms can be set up. For example, whenever a log data synchronization failure is detected, the failed log data can be added to the corresponding queue so that the log data in the queue can be resynchronized to the target data source. Alternatively, a verification mechanism can be set up for cross-day log synchronization tasks. After the cross-day synchronization task is completed, relevant parameters are compared to determine whether all cross-day log data has been pushed to the target data source so that the data that has not been synchronized can be synchronized again, thereby ensuring the accuracy of the synchronization task.

[0123] This embodiment provides a log synchronization method. The log synchronization method identifies log synchronization trigger events with a synchronization type of date-based table partitioning, where the date-based table partitioning type determines the synchronization type of the log synchronization task based on the date to which the log data belongs. Based on the log synchronization trigger event, a target log synchronization task corresponding to the target date of the log synchronization trigger event is created and executed to synchronize the log data to be synchronized in the target log synchronization task to the target data source. This embodiment introduces the concept of date segmentation during the synchronization process, dividing the log data to be synchronized into batches according to their respective dates, and then defining corresponding date-based table types. Log synchronization tasks are executed using these types, enabling dynamic synchronization of log data over time, thereby improving the intelligence of log data synchronization operations. By automatically identifying log data synchronization operation trigger events, users only need to perform a trigger operation, or even no manual triggering is required. The system can automatically start creating and executing corresponding log data synchronization tasks based on preset rules (e.g., automatic triggering based on date updates), eliminating the manual operation required in the log synchronization process of existing technologies. This further improves the intelligence level of log data synchronization operations and solves the technical problem of low intelligence in log data synchronization operations in the field of automated testing.

[0124] Furthermore, based on the above Figure 2 The first embodiment shown illustrates a second embodiment of the log synchronization method of the present invention. In this embodiment, the target log synchronization task includes a daily log synchronization task and / or a cross-day log synchronization task, and step S20 includes:

[0125] The pre-configured synchronization metadata is determined based on the log synchronization trigger event, and the test log data for the current day and / or the test log data for the next day are obtained, wherein the test log data for the next day is log data with a time span of multiple days;

[0126] Based on the synchronized metadata and the test log data of the day, and / or the test log data of the day, create a log synchronization task for the day and / or a log synchronization task for the day corresponding to the date of the target;

[0127] Execute the same-day log synchronization task and / or cross-day log synchronization task to synchronize the log data to be synchronized in the same-day log synchronization task and / or cross-day log synchronization task to the target data source.

[0128] In this embodiment, the test log data for the current day refers to the log data generated on the current date when the synchronization program is running. Therefore, the synchronization task for the current day only processes the test log data for the current date.

[0129] Cross-day test log data refers to log data generated on the current date of the synchronous program's execution but belonging to the period before that date. The reasons for the generation of this type of test log data are as follows: (1) Test log data generated across days, for example: execution started at 23:59:59 on 2021-02-07 and ended on 2021-02-08. The execution ends at 00:00:01. During this period, the test log data generated spans two days (2021-02-07 and 2021-02-08), but in terms of business logic, it belongs to the test log data table of 2021-02-07; (2) Test log data generated by cross-day retry runs, for example: automated test record A executed on 2021-02-07, and a retry run initiated on record A on 2021-02-08. Although the test log data generated at this time is dated 2021-02-08, in terms of business logic, it belongs to 2021-02-07. Therefore, this part of the test log data is stored in the test log data table of 2021-02-07. For the above two situations, the same-day synchronization task cannot be processed. Therefore, it is necessary to create a separate cross-day synchronization task to process it. The cross-day log synchronization task was introduced to ensure the completeness of date-based log data synchronization solutions. Firstly, from a programming design perspective, cross-day log data and current-day test data do not belong to the same date batch, which is the reason for date-based table partitioning. The cross-day test log data synchronization program embodies the idea of ​​"division of labor and cooperation." Secondly, compared to current-day test log data, cross-day test log data is less abundant because most of it has already been synchronized within its own "day." Therefore, the current-day log data synchronization program handles the synchronization of log data for that day's batch, while the cross-day log data synchronization program handles the synchronization of log data from previous days (including multiple date batches), achieving the highest efficiency in resource utilization.

[0130] Further, the step of creating a log synchronization task for the current day and / or a log synchronization task for the day corresponding to the target date based on the synchronization metadata and the test log data of the current day, and / or the test log data of the next day includes:

[0131] The data table prefix is ​​obtained based on the synchronized metadata, and the date to which the target belongs is formatted to obtain built-in variables;

[0132] Based on the test log data of the day, the data table prefix, and the built-in variables, a log synchronization data table for the day is generated to create the log synchronization task for the day; and / or,

[0133] Based on the synchronization metadata, traverse the historical log synchronization record queue before the date to which the target belongs, and obtain the synchronization date table and the latest synchronization tracking column value from the historical log synchronization record queue;

[0134] The cross-day log synchronization task is created based on the cross-day test log data, the synchronization date table, and the latest synchronization tracking column value.

[0135] In this embodiment, for the log synchronization task of the current day, it is necessary to parse the statement parameter in the synchronization task configuration parameters of the synchronization metadata to obtain the data table prefix, such as item_execute_status_, and then obtain the date_format parameter of the synchronization task configuration table in the synchronization task configuration parameters. The current time is converted into date_format format and assigned to the built-in variable date_format in the synchronization task configuration parameters (the time and date format of the data date table, such as the time and date format corresponding to 20210317 is yyyyMMdd). Then, the JobKey of the synchronization task of the current day can be generated. The format can be: uppercase English letters of the data table prefix + ":DATE_INDEX:" + the date_format format of the current time, such as ITEM_EXECUTE_STATUS_:DATE_INDEX:20210208, to represent the currently processed synchronization task.

[0136] For cross-day log synchronization tasks, the JobKey format can be: uppercase English letters for the data table prefix + ":DATE_INDEX:" + the date_format for the cross-day time, for example: ITEM_EXECUTE_STATUS_:DATE_INDEX:20210208, to indicate the currently processed cross-day test log data synchronization task. It iterates through the historical test log data synchronization record queue for each historical date batch, retrieving the synchronization date table and the latest current synchronization tracking column value (tracking_column_type + 1, used to record the starting point of the next synchronization and avoid duplicate synchronization) current_sync_tracking_column_value. It compares this latest synchronization tracking column value with the latest tracking column value in the source data date table. If it's not the latest tracking column value, a cross-day log data synchronization task is generated using the jobKey, the latest synchronization tracking column value, and the date table information, and submitted to the thread pool for processing. The process can be abstracted as follows: Figure 3 As shown.

[0137] Furthermore, prior to step S10, the following steps are also included:

[0138] Receive initial configuration information sent by the user, wherein the initial configuration information includes initial source data source configuration information, initial target data source configuration information, and initial synchronization task configuration information;

[0139] The initial configuration information is used to initialize the synchronization metadata so that the configured synchronization metadata can be reused in the next log synchronization task.

[0140] In this embodiment, the initial configuration can be divided into three parts: source data source configuration, target data source configuration, and synchronization task configuration. When using date-based table partitioning for real-time synchronization of test log data, users need to configure synchronization metadata first. Subsequent additions of synchronization tasks can reuse the metadata configuration, reducing synchronization task maintenance costs and improving synchronization task configuration efficiency.

[0141] Specifically, taking MySQL as the source data source as an example, the source data source configuration involves configuring the MySQL data source connection information. Data needs to be retrieved from the source data source when processing test log data synchronization. The automated test log data source storage source is MySQL. The source data source (MySQL) configuration parameters are as follows: Figure 4 As shown, the first time you need to configure parameters such as the primary key ID of the source data source, connection address, username, and connection password.

[0142] Taking ElasticSearch as the target data source as an example. The target data source configuration involves configuring the ElasticSearch data source connection information. When processing test log data synchronization, data needs to be pushed to the target data source. The target storage source for automated test log data is ElasticSearch. The target data source configuration parameters are as follows: Figure 5 As shown, the first time you need to configure the primary key ID, connection address, username, and password of the target data source.

[0143] For synchronization task configuration, each test log data table synchronization corresponds to one synchronization task configuration. When adding a synchronization task configuration, it checks whether the synchronization task in the configuration is based on date-partitioned tables. If so, it automatically marks the synchronization type as date-partitioned table synchronization in the synchronization task configuration. The synchronization task configuration parameters are as follows: Figure 6 As shown, the initial configuration requires the following: primary key ID of the synchronization task, primary key ID of the source data source, primary key ID of the target data source, statement parameter (the SQL to be executed when retrieval of data from the source data source is the tracking column of the last log data from the previous synchronization + 1), schedule parameter, and index of the target data source for synchronization.

[0144] Further, the step of executing the same-day log synchronization task and / or the cross-day log synchronization task to synchronize the log data to be synchronized in the same-day log synchronization task and / or the cross-day log synchronization task to the target data source includes:

[0145] To acquire the first distributed lock and / or the second distributed lock for the same day's log synchronization task and / or the cross-day log synchronization task;

[0146] The test log data for the current day and / or the test log data for the next day are segmented according to the data slicing method, so as to push the test log data for the current day and / or the test log data for the next day to the target data source in batches;

[0147] When the daily test log data and / or cross-day test log data are detected to be pushed out, the first distributed lock and / or the second distributed lock are released.

[0148] In this embodiment, for the log synchronization task of the current day, the corresponding distributed lock (first distributed lock) needs to be acquired first, with the log synchronization task of the current day as the target. Then, based on the synchronization target data source index parameter target_index (e.g., item_execute_status_${current_date_value}, where ${current_date_value} is the current synchronization date) in the synchronization metadata, it is determined whether the target data source index exists. If target_index does not exist, the target data source index is created. Then, the value of the built-in variable current_sync_tracking_column_value is obtained. To achieve real-time synchronization of the test log data of the current day, the data slicing concept can be adopted. The test log data of the current day is divided into data segments according to the pre-configured tracking column values, and synchronized in real time multiple times. Assuming that the size of the data slice in one synchronization is 1000, the tracking column range corresponding to the data slice in this synchronization is [current_sync_tracking_column_value, current_sync_tracking_column_value+1000]. The built-in variable is the largest tracking column value of the previous synchronization data slice plus 1. For the first synchronization, the built-in variable is initialized to 1. When synchronizing test log data for the same day based on data slicing, the change process of built-in variables is as follows: Figure 7 As shown.

[0149] The upper section of the diagram represents the start of the first synchronization, with the built-in variable initialized to 1, and the data not yet synchronized. The middle section represents the completion of the first synchronization, with the built-in variable updated to 1001, i.e., (1000+1). Light color indicates no synchronization, and dark color indicates synchronization. The lower section represents the completion of the second synchronization, with the built-in variable updated to 2001, i.e., (2000+1). Light color indicates no synchronization, and dark color indicates synchronization, and so on.

[0150] After the data sharding synchronization is completed, the statement parameter is converted into the actual SQL to be executed, and then the SQL is executed to obtain the log data to be synchronized. The log data is then pushed to the target data source in sequence. The maximum tracking column value, tracking_column_value, in the synchronized test log data is obtained. After updating the built-in variable to the maximum tracking column value + 1, the first distributed lock is released.

[0151] Finally, it is determined whether the date date1 of the day's synchronization process matches the current date date2. If date1 is less than date2, the source data table, target index, and synchronization tracking column value of the day's synchronization process are bound into a date partition information and placed into the historical test log data synchronization record queue. Then, the built-in variable of the day's synchronization process is reset to 1.

[0152] For cross-day log synchronization tasks, a second distributed lock is acquired targeting the cross-day log synchronization task JobKey. Then, based on the JobKey, the table information for previously successfully synchronized dates is retrieved (source data table, target data index, and the tracking column values ​​to be synchronized next). Similar to same-day synchronization tasks, a data slicing approach can be used (assuming a data slice size of 1000) to process multiple date synchronization tasks in batches. SQL is executed to retrieve the log data to be synchronized, and the log data is pushed sequentially to the target data source. The latest tracking column value of the historical log data synchronization record is updated to the largest tracking column value of the current synchronization data, tracking_column + 1. Finally, the second distributed lock is released. It should be noted that a cross-day log data synchronization task only completes one round of synchronization when the generated cross-day test log data synchronization task is submitted to the thread pool for execution and all tasks are completed.

[0153] Furthermore, after the step of releasing the first distributed lock and / or the second distributed lock upon detecting that the test log data for the current day and / or the test log data for the next day has been pushed out, the method further includes:

[0154] Iterate through the synchronization success record table obtained after the test log data of the day and / or the test log data of the cross-day are pushed, and obtain the tracking column value of the last synchronization success in the synchronization success record table;

[0155] Based on the source data table and target data table in the synchronization metadata, obtain the source tracking column value and target tracking column value respectively, and determine whether the source tracking column value and target tracking column value are both greater than the tracking column value of the last successful synchronization.

[0156] If so, the log data in the target data source is updated synchronously to complete the verification of the daily log synchronization task and / or cross-day log synchronization task.

[0157] In this embodiment, the application also includes a verification mechanism. When test log data has been successfully synchronized (on the same day and / or across days), but at some point in the future, this part of the successfully synchronized test log data in the source data source is updated, then the latest form of this part in the target test log data also needs to be synchronized and updated. Therefore, this part of the synchronization update task is handed over to the test log data synchronization progress verification mechanism. The verification mechanism will scan the test log data that has been synchronized in the source data source and the target data source. If the test log data is inconsistent in the source data source and the target data source, it will be synchronized again.

[0158] Specifically, the process checks if a successful synchronization record already exists in the current data table. If it does, it iterates through the date data tables that have already been synchronized. For the date test log data table with a synchronization record, it retrieves the tracking_column value of the last successful synchronization. It then progressively compares data in the source and target data tables whose tracking_column values ​​are less than or equal to the tracking_column value of the last successful synchronization. If an inconsistency is found, it synchronizes and updates the corresponding data in the target data source.

[0159] This embodiment further synchronizes different types of log data by creating separate parallel daily and cross-day synchronization tasks, ensuring the integrity of the date-based log data synchronization scheme and maximizing resource utilization. When users perform real-time synchronization of test log data based on date partitioning, they first configure synchronization metadata, then reuse this metadata configuration when adding new synchronization tasks, reducing maintenance costs and improving configuration efficiency. By employing data slicing to divide the log data to be synchronized into segments and synchronizing them in stages, real-time synchronization of test log data for the same day is achieved. A verification mechanism scans the source and target data sources for synchronized test log data, and performs re-synchronization if the test log data is inconsistent between the source and target data sources, ensuring the accuracy of the synchronization operation.

[0160] Furthermore, based on the above Figure 2 The first embodiment shown illustrates a third embodiment of the log synchronization method of the present invention. In this embodiment, step S10 includes:

[0161] When new node information for a synchronization task configuration is detected, the log synchronization trigger event is determined based on the new node information. The triggering conditions for the log synchronization trigger event include manual triggering and / or automatic triggering based on preset rules.

[0162] Obtain the synchronization type field of the log synchronization trigger event, identify the synchronization type of the log synchronization trigger event to determine that the synchronization type is a date table partitioning type.

[0163] In this embodiment, the log synchronization trigger event can be initiated by the user or automatically initiated by the terminal according to a preset program (e.g., according to a preset timed synchronization frequency parameter). The following explanation uses the user's first initiation as an example. The user adds a synchronization task configuration in the front-end synchronization task management, calls the back-end interface, and the back-end system inserts a synchronization task configuration into the database and adds a node to the ZooKeeper node directory. The synchronization task monitor monitors the node directory, and when new node information is added, it retrieves the newly added synchronization configuration parameters based on the configuration ID. The Quartz job scheduling framework can be used to dynamically add and register synchronization tasks, enabling automatic timed synchronization of daily tasks at a frequency specified by the `schedule` parameter in the configuration table. Then, the synchronization type field in the synchronization configuration parameters is retrieved, and the synchronization type with the `sync_type` field is identified as a date-based table partitioning type.

[0164] Furthermore, after the step of creating and executing a target log synchronization task corresponding to the target date of the log synchronization trigger event, the method further includes:

[0165] When a log data synchronization failure is detected, retrieve the log data of the failed synchronization.

[0166] Log data that fails to synchronize is pushed to the retry data synchronization queue for the corresponding date, so that the log data in the retry data synchronization queue is resynchronized to the target data source in turn.

[0167] In this embodiment, the application also includes a retry mechanism. When the test log data synchronization program fails to synchronize (on the same day and / or across days), the failed test log data will be automatically transferred to the retry mechanism processing program.

[0168] Specifically, the system checks whether a failure retry data synchronization queue has been created for the current date's synchronization data table for the same-day or cross-day log synchronization task. If it's the first synchronization attempt, a failure retry data synchronization queue is created; otherwise, it's not necessary. When synchronization of test log data fails on the same day or across days, the failed log data is pushed to the corresponding date's failure retry data synchronization queue. The system scans the failure retry data synchronization queues for the current date and earlier. If data exists in the queue, the log data in the queue is pushed sequentially to the target data source. If the attempt still fails, the data is added to the end of the failure retry data synchronization queue.

[0169] As a specific embodiment, the overall process of real-time synchronization of test log data based on date-based table partitioning is as follows: Figure 8 As shown, the method for automated real-time synchronization of test log data for date-partitioned tables is generally divided into three parts: synchronization metadata configuration, synchronization task processing, and synchronization task failure handling mechanism. Synchronization metadata configuration is further divided into three parts: source data source configuration, target data source configuration, and synchronization task configuration. The synchronization task processing program includes a daily test log data synchronization program and a cross-day test log data synchronization program. The synchronization task failure handling mechanism includes a test log data synchronization failure retry mechanism and a test log data synchronization progress verification mechanism. It should also be noted that the daily test log data synchronization program, the cross-day test log data synchronization program, the retry mechanism, and the verification mechanism can all be started simultaneously when a new synchronization trigger event is detected, and then executed accordingly based on the actual situation.

[0170] This embodiment further ensures the accuracy of the synchronization operation by setting a retry mechanism, which allows failed log data to be resynchronized, thus avoiding the situation where failed test log data does not exist in the target data source. By listening to log synchronization trigger events in real time, it can respond to and execute the corresponding log synchronization tasks in real time, thereby achieving the purpose of real-time synchronization.

[0171] like Figure 9 As shown, the present invention also provides a log synchronization device, the log synchronization device comprising:

[0172] The synchronization trigger identification module 10 is used to identify log synchronization trigger events with a synchronization type of date table partitioning, wherein the date table partitioning type is used to determine the synchronization type of the log synchronization task based on the date to which the log data belongs;

[0173] The synchronization task execution module 20 is used to create and execute a target log synchronization task corresponding to the target date of the log synchronization trigger event, so as to synchronize the log data to be synchronized in the target log synchronization task to the target data source.

[0174] The methods executed by the above-mentioned program modules can be referred to in the various embodiments of the log synchronization method of the present invention, and will not be repeated here.

[0175] The present invention also provides a log synchronization device.

[0176] The log synchronization device includes a processor, a memory, and a log synchronization program stored in the memory and executable on the processor, wherein when the log synchronization program is executed by the processor, it implements the steps of the log synchronization method as described above.

[0177] The method implemented when the log synchronization program is executed can be referred to in various embodiments of the log synchronization method of the present invention, and will not be repeated here.

[0178] The present invention also provides a computer-readable storage medium.

[0179] The present invention provides a computer-readable storage medium storing a log synchronization program, which, when executed by a processor, implements the steps of the log synchronization method described above.

[0180] The method implemented when the log synchronization program is executed can be referred to in various embodiments of the log synchronization method of the present invention, and will not be repeated here.

[0181] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0182] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0183] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0184] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A log synchronization method, characterized in that, The log synchronization method includes: Identify log synchronization trigger events with a synchronization type of date-based table partitioning, wherein the date-based table partitioning type determines the synchronization type of the log synchronization task based on the date to which the log data belongs; The pre-configured synchronization metadata is determined based on the log synchronization trigger event, and the test log data for the current day and / or the test log data across days is obtained, wherein the test log data across days is log data with a time span of multiple days; The data table prefix is ​​obtained based on the synchronization metadata, and the target date of the log synchronization trigger event is formatted to obtain the built-in variable; Based on the test log data of the day, the data table prefix, and the built-in variables, a log synchronization data table for the day is generated to create a log synchronization task for the day; and / or, Based on the synchronization metadata, traverse the historical log synchronization record queue before the date to which the target belongs, and obtain the synchronization date table and the latest synchronization tracking column value from the historical log synchronization record queue; A cross-day log synchronization task is created based on the cross-day test log data, the synchronization date table, and the latest synchronization tracking column value. Execute the same-day log synchronization task and / or cross-day log synchronization task to synchronize the log data to be synchronized in the same-day log synchronization task and / or cross-day log synchronization task to the target data source.

2. The log synchronization method as described in claim 1, characterized in that, Before the step of identifying log synchronization trigger events of the date-partitioned type, the method further includes: Receive initial configuration information sent by the user, wherein the initial configuration information includes initial source data source configuration information, initial target data source configuration information, and initial synchronization task configuration information; The initial configuration information is used to initialize the synchronization metadata so that the configured synchronization metadata can be reused in the next log synchronization task.

3. The log synchronization method as described in claim 1, characterized in that, The step of executing the same-day log synchronization task and / or the cross-day log synchronization task to synchronize the log data to be synchronized in the same-day log synchronization task and / or the cross-day log synchronization task to the target data source includes: To acquire the first distributed lock and / or the second distributed lock for the same day's log synchronization task and / or the cross-day log synchronization task; The test log data for the current day and / or the test log data for the next day are segmented according to the data slicing method, so as to push the test log data for the current day and / or the test log data for the next day to the target data source in batches; When the daily test log data and / or cross-day test log data are detected to be pushed out, the first distributed lock and / or the second distributed lock are released.

4. The log synchronization method as described in claim 3, characterized in that, After the step of releasing the first distributed lock and / or the second distributed lock when the daily test log data and / or cross-day test log data have been pushed out, the method further includes: Iterate through the synchronization success record table obtained after the test log data of the day and / or the test log data of the cross-day are pushed, and obtain the tracking column value of the last synchronization success in the synchronization success record table; Based on the source data table and target data table in the synchronization metadata, obtain the source tracking column value and target tracking column value respectively, and determine whether the source tracking column value and target tracking column value are both greater than the tracking column value of the last successful synchronization. If so, the log data in the target data source is updated synchronously to complete the verification of the daily log synchronization task and / or cross-day log synchronization task.

5. The log synchronization method as described in claim 1, characterized in that, The steps for identifying log synchronization trigger events of the date-partitioned table type include: When new node information for a synchronization task configuration is detected, the log synchronization trigger event is determined based on the new node information. The triggering conditions for the log synchronization trigger event include manual triggering and / or automatic triggering based on preset rules. Obtain the synchronization type field of the log synchronization trigger event, identify the synchronization type of the log synchronization trigger event to determine that the synchronization type is a date table partitioning type.

6. The log synchronization method as described in any one of claims 1-5, characterized in that, After the step of creating and executing a target log synchronization task corresponding to the target date of the log synchronization trigger event based on the log synchronization trigger event, the method further includes: When a log data synchronization failure is detected, retrieve the log data of the failed synchronization. Log data that fails to synchronize is pushed to the retry data synchronization queue for the corresponding date, so that the log data in the retry data synchronization queue is resynchronized to the target data source in turn.

7. A log synchronization device, characterized in that, The log synchronization device includes: a memory, a processor, and a log synchronization program stored in the memory and executable on the processor, wherein the log synchronization program, when executed by the processor, implements the steps of the log synchronization method as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a log synchronization program, which, when executed by a processor, implements the steps of the log synchronization method as described in any one of claims 1 to 6.

Images