Log processing method and device, equipment, medium and product

Abstract

The invention provides a log processing method and device, equipment, a medium and a product, and can be applied to the field of distributed technologies, the field of artificial intelligence technologies and the field of financial science and technology. The method comprises the following steps: for a to-be-synchronized log set, determining to-be-synchronized log subsets based on log categories; different logs in any to-be-synchronized log subset belong to the same log category; for any to-be-synchronized log subset, determining a corresponding reference log, and performing differential coding on logs in the log subset based on the determined reference log to obtain a log differential coding set corresponding to the log subset; synchronizing the log differential code set corresponding to any log subset to be synchronized to a target end; and the target end is used for determining a corresponding reference log aiming at any received log differential coding set, and determining a corresponding to-be-synchronized log subset based on the determined reference log and the aiming log differential coding set.

Description

Technical Field

[0001] This application relates to the fields of distributed technology, artificial intelligence technology, and financial technology, and specifically to a log processing method, apparatus, device, medium, and product. Background Technology

[0002] Logs play a crucial role in many business operations, specifically for backing up or auditing data, and for storing and retaining operational data. Currently, there are many log synchronization operations available; for example, log synchronization can be performed when backing up log data, allowing logs from one device to be backed up and transmitted to another.

[0003] However, the current method of log synchronization is to directly send log data, which has low synchronization efficiency. Summary of the Invention

[0004] In view of the above problems, this application provides a log processing method, apparatus, device, medium and product for improving log synchronization efficiency.

[0005] According to a first aspect of this application, a log processing method is provided, comprising: determining a subset of logs to be synchronized based on log categories for a set of logs to be synchronized; different logs in any subset of logs to be synchronized belonging to the same log category; determining a corresponding base log for any subset of logs to be synchronized, and differentially encoding the logs in the target subset based on the determined base log to obtain a log differential encoding set corresponding to the target subset of logs; synchronizing the log differential encoding set corresponding to any subset of logs to be synchronized to a target end; wherein the target end is configured to: determine a corresponding base log for any received log differential encoding set, and determine a corresponding subset of logs to be synchronized based on the determined base log and the target log differential encoding set.

[0006] Optionally, determining the corresponding baseline log for any subset of logs to be synchronized includes at least one of the following: for any subset of logs to be synchronized, determining the preset baseline log corresponding to the log category to which it belongs as the baseline log for the target log subset; different log categories correspond to different preset baseline logs; for any subset of logs to be synchronized, determining the current corresponding baseline log by comprehensively considering the logs in the target log subset.

[0007] Optionally, the method further includes: for any historically synchronized log subset, determining the compression level of the corresponding baseline log based on the data volume of the target log subset and the data volume of the corresponding log differential encoding set; determining the corresponding baseline log for any log subset to be synchronized includes: for any log subset to be synchronized, if the compression level of the preset baseline log corresponding to the log category is determined to be higher than the preset compression level threshold, the preset baseline log corresponding to the log category is determined as the baseline log corresponding to the target log subset; different log categories correspond to different preset baseline logs; for any log subset to be synchronized, if the compression level of the preset baseline log corresponding to the log category is determined to be lower than the preset compression level threshold, the current corresponding baseline log is determined by comprehensively considering the logs in the target log subset, and the determined baseline log is determined as the current preset baseline log corresponding to the log category.

[0008] Optionally, determining the corresponding baseline log for any subset of logs to be synchronized includes: for any subset of logs to be synchronized, if it is determined that the current log synchronization frequency is higher than a preset frequency threshold, determining the preset baseline log corresponding to the log category to which it belongs as the baseline log for the target log subset; different log categories correspond to different preset baseline logs; for any subset of logs to be synchronized, if it is determined that the current log synchronization frequency is lower than a preset frequency threshold, determining the current corresponding baseline log by comprehensively considering the logs in the target log subset.

[0009] Optionally, determining the current baseline log by integrating the logs in the target log subset includes: determining the corresponding baseline field value based on different field values ​​of the same log field among different logs in the target log subset; and determining the current baseline log corresponding to the target log subset by integrating the baseline field values ​​of different log fields.

[0010] Optionally, the method further includes: synchronizing the determined baseline log to the target end; synchronizing the log differential encoding set corresponding to any subset of logs to be synchronized to the target end includes: synchronizing the log differential encoding set corresponding to any subset of logs to be synchronized, and the identification information of the corresponding baseline log to the target end; the target end is used to: for any received log differential encoding set, determine the corresponding baseline log based on the identification information of the corresponding baseline log.

[0011] Optionally, synchronizing the log differential encoding set corresponding to any log subset to be synchronized to the target end includes: for any log subset to be synchronized, synchronizing the corresponding log differential encoding set and the integrity verification information corresponding to the target log subset to the target end; the target end is used to: for any received log differential encoding set, determine the corresponding base log, and based on the determined base log and the target log differential encoding set, determine the log subset to be verified; and perform integrity verification on the determined log subset to be verified based on the corresponding integrity verification information.

[0012] Optionally, synchronizing the log differential encoding set corresponding to any log subset to be synchronized to the target end includes: encrypting and synchronizing the log differential encoding set corresponding to any log subset to be synchronized, and the integrity verification information corresponding to the log differential encoding set to the target end; the target end is used to: for any received log differential encoding set, if the verification is passed based on the corresponding integrity verification information, determine the corresponding base log, and determine the corresponding log subset to be synchronized based on the determined base log and the corresponding log differential encoding set.

[0013] Optionally, determining the subset of logs to be synchronized based on log categories for the set of logs to be synchronized includes any one of the following: clustering the logs in the set of logs to be synchronized, and determining the log clusters in the clustering results as the subset of logs to be synchronized; determining the corresponding log categories based on log features for the logs in the set of logs to be synchronized, and determining the subset of logs to be synchronized based on logs belonging to the same log category.

[0014] Optionally, determining the corresponding baseline log for any subset of logs to be synchronized includes: determining multiple candidate baseline logs for any subset of logs to be synchronized; and determining the baseline log corresponding to the specified log subset from the determined candidate baseline logs based on the compression degree of the candidate baseline logs.

[0015] A second aspect of this application provides a log processing apparatus, comprising: a classification module, configured to determine a subset of logs to be synchronized based on log categories for a set of logs to be synchronized; different logs in any subset of logs to be synchronized belong to the same log category; a reference module, configured to determine a corresponding reference log for any subset of logs to be synchronized, and to perform differential encoding on the logs in the specified subset of logs based on the determined reference log to obtain a log differential encoding set corresponding to the specified subset of logs; and a synchronization module, configured to synchronize the log differential encoding set corresponding to any subset of logs to be synchronized to a target end; wherein the target end is configured to: determine a corresponding reference log for any received log differential encoding set, and to determine a corresponding subset of logs to be synchronized based on the determined reference log and the specified log differential encoding set.

[0016] A third aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0017] A fourth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0018] The fifth aspect of this application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method. Attached Figure Description

[0019] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0020] Figure 1 This illustration schematically depicts an application scenario of a log processing method according to an embodiment of this application.

[0021] Figure 2 A flowchart illustrating a log processing method according to an embodiment of this application is shown schematically.

[0022] Figure 3 This schematic diagram illustrates a structural block diagram of a log processing apparatus according to an embodiment of the present application;

[0023] Figure 4 A block diagram schematically illustrates an electronic device suitable for implementing a log processing method according to an embodiment of this application. Detailed Implementation

[0024] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0025] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0026] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0027] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0028] Logs play a crucial role in many business processes, specifically for backing up or auditing data, and for storing and retaining operational or business data. Currently, there are many log synchronization operations, such as backing up log data from one device to another. However, current log synchronization methods involve directly sending log data, resulting in low synchronization efficiency.

[0029] To address the aforementioned technical problems, embodiments of this application provide a log processing method.

[0030] In this method, for the log set that needs to be synchronized, a baseline log can be determined as a reference, making it easy to perform differential encoding on the logs in the log set based on the baseline log. For example, a log field value that is the same as the baseline log can be encoded as 0, which confirms that the log field value is the same as the corresponding log field value in the baseline log, thereby reducing the amount of log data through differential encoding.

[0031] Subsequently, the obtained log differential encoding results can be synchronized. Differential encoding can reduce the amount of log data, thereby reducing the amount of data transmitted during log synchronization and further improving log synchronization efficiency.

[0032] Furthermore, log differential encoding can significantly improve log security. For other business units that cannot determine the baseline log, the differential encoding result may not accurately reflect the original log information. It can also reduce information loss during log synchronization, facilitating subsequent recovery based on the baseline log.

[0033] The above method can be used to synchronize logs in a distributed system. For example, a distributed system can contain primary and backup nodes, and log synchronization can be performed between the primary and backup nodes.

[0034] In the above method, the log set to be synchronized can be further divided according to log categories to obtain different log subsets. This facilitates the determination of corresponding baseline logs for differential encoding based on log categories, allowing for the use of different baseline logs for differential encoding across different log categories. Using baseline logs more closely aligned with each log category improves the compression effect of differential encoding, further reducing the amount of log data and thus reducing the amount of data transmitted during log synchronization, thereby improving log synchronization efficiency. Furthermore, artificial intelligence or machine learning methods can be combined to predict and determine log categories.

[0035] In the above method, in order to achieve log synchronization, the synchronized log differential encoding results can be further recovered based on the corresponding baseline log, which can improve the consistency of log synchronization.

[0036] Furthermore, the above methods can also provide different ways to determine the baseline log. Specifically, for example, a suitable baseline log can be determined in real time for a log subset. This can be done by determining the suitable baseline log based on the specific log information distribution within the log subset. Alternatively, a previously used baseline log can be reused. Different log categories can correspond to different baseline logs, allowing for pre-setting corresponding baseline logs for different log categories, facilitating subsequent differential coding. Therefore, after determining the log category, the pre-determined corresponding baseline log can be directly used for differential coding.

[0037] Understandably, determining the appropriate baseline log in real time can improve the compatibility and compression effect of differential coding, reduce the amount of log data and the amount of data transmitted during log synchronization, and improve the efficiency of log synchronization. On the other hand, using the baseline log corresponding to the pre-set log category can eliminate the need to determine the baseline log in real time, thereby improving the efficiency of differential coding and log synchronization.

[0038] Therefore, in the above methods, the appropriate baseline log determination method can be selected according to actual needs and circumstances. For example, when the number of logs in the log subset is small, the appropriate baseline log can be determined in real time; when there are many logs to be synchronized, or the log synchronization frequency is high, or the timeliness requirement for log synchronization is high, the pre-set baseline log can be used to improve the efficiency of differential coding.

[0039] By selecting an appropriate baseline log determination method, the adaptability of log synchronization to actual needs can be improved, and the flexibility of log synchronization can be increased.

[0040] It should be noted that the log processing method and apparatus provided in the embodiments of this application can be applied to the fields of distributed technology, artificial intelligence technology, and fintech. For example, in the financial or banking sectors, the log processing method provided in the embodiments of this application can be used for log differential encoding and log synchronization during log synchronization. The embodiments of this application can also be applied to any field other than fintech. For example, for log synchronization in the audio / video technology or communication fields, the log processing method provided in the embodiments of this application can be used for log differential encoding and log synchronization. The application fields of the log processing method and apparatus provided in the embodiments of this application are not limited.

[0041] In the technical solution of this application, the user information (including but not limited to user personal information, user image information, user device information, such as location information, log information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure and application of related data all comply with relevant laws, regulations and standards, necessary confidentiality measures have been taken, they do not violate public order and good morals, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0042] In scenarios involving automated decision-making using personal information, the methods, devices, and systems provided in this application all offer users corresponding entry points for choosing to agree to or reject the automated decision-making results. If the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0043] Figure 1 The illustration shows an application scenario diagram of a log processing method according to an embodiment of this application.

[0044] like Figure 1 As shown, application scenario 100 according to this embodiment may include: a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0045] Users can use the first terminal device 101, the second terminal device 102, or the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0046] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0047] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, or the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0048] It should be noted that the log processing method provided in this application embodiment can generally be executed by server 105. Correspondingly, the log processing device provided in this application embodiment can generally be located in server 105. The log processing method provided in this application embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the log processing device provided in this application embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0049] It is understood that server 105 can synchronize logs to various terminal devices using the log processing method provided in this application embodiment. Each terminal device can also execute the log processing method provided in this application embodiment to synchronize logs to other terminal devices or server 105. Furthermore, server 105 can also synchronize logs to other servers or other devices.

[0050] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0051] Figure 2 A flowchart illustrating a log processing method according to an embodiment of this application is shown schematically.

[0052] like Figure 2 As shown, the log processing method provided in this embodiment may include operations S210 to S230. The embodiments of this application do not limit the executing entity of the provided log processing method. Optionally, it can be applied to any device or any application, specifically a server, a source, a terminal, etc. For ease of description, regarding the log synchronization process, the sender of the log synchronization can be called the source, and the receiver of the log synchronization can be called the target. Thus, the source can be identified as the executing entity, executing a log processing method to synchronize the logs to the target.

[0053] In operation S210, for the set of logs to be synchronized, a subset of logs to be synchronized is determined based on the log category; different logs in any subset of logs to be synchronized belong to the same log category.

[0054] In operation S220, for any subset of logs to be synchronized, a corresponding base log is determined, and differential encoding is performed on the logs in the target log subset based on the determined base log to obtain the log differential encoding set corresponding to the target log subset.

[0055] In operation S230, the log differential encoding set corresponding to any log subset to be synchronized is synchronized to the target end; the target end is used to: determine the corresponding base log for any received log differential encoding set, and determine the corresponding log subset to be synchronized based on the determined base log and the corresponding log differential encoding set.

[0056] This method can reduce the amount of log data by differentially encoding the logs that need to be synchronized. By transmitting differentially encoded logs during synchronization, the amount of data transmitted during log synchronization can be reduced, thereby further improving log synchronization efficiency.

[0057] Furthermore, this method can improve the security of log information during log synchronization and reduce the loss of log information during the synchronization process by using differential encoding of the synchronously transmitted logs.

[0058] The embodiments of this application are not limited to a set of logs to be synchronized. Optionally, the set of logs to be synchronized may include logs that need to be synchronized to the target end. Specifically, these may be logs collected from the source end or logs generated within a preset time period. The preset time period may be the time period between the current moment and a historical moment of a preset duration prior to the current moment.

[0059] In a specific example, when synchronizing logs across different databases, the database logs in the source database, such as logs of operations like adding, deleting, modifying, and querying data, can be identified as the set of logs to be synchronized. Alternatively, database logs generated within a preset time period (such as newly generated database logs) can be identified as the set of logs to be synchronized. Similarly, when synchronizing logs between primary and backup nodes, the logs in the primary node can be identified as the set of logs to be synchronized, which can also be logs generated within a preset time period (such as newly generated operation logs).

[0060] The embodiments of this application do not limit the number of determined log subsets to be synchronized. Optionally, one or more log subsets to be synchronized can be determined. Each log in the log subset to be synchronized may belong to the same log category. Accordingly, the above method flow explains the process of differential encoding and synchronization for any log subset to be synchronized. It is understood that for different or multiple log subsets to be synchronized, the operations in the above method flow can be performed separately to achieve differential encoding and synchronization. For a detailed explanation, please refer to other embodiments.

[0061] This application does not limit the order of operations performed between different log subsets to be synchronized; they can be executed in parallel. For example, operations S220 and S230 can be executed in parallel for different log subsets to be synchronized, without affecting each other, and each is synchronized to the target end. Alternatively, the total log encoding set corresponding to the log subsets to be synchronized can be determined by combining the log differential encoding sets corresponding to each log subset to be synchronized, and the total log encoding set can be synchronized to the target end.

[0062] The following sections will explain different aspects of the embodiments of this application.

[0063] 1. In operation S210, for the set of logs to be synchronized, a subset of logs to be synchronized is determined based on the log category; different logs in any subset of logs to be synchronized belong to the same log category.

[0064] The embodiments of this application do not limit the specific method of determining the subset of logs to be synchronized based on log categories. Optionally, different logs belonging to the same log category in the set of logs to be synchronized can be added to the same subset of logs to be synchronized, so that logs of different log categories can be stored separately through different subsets of logs to be synchronized. Different subsets of logs to be synchronized can correspond to different log categories.

[0065] The embodiments of this application do not limit the specific method for determining log categories. Optionally, log categories can be determined directly based on log information, such as the source information of the logs, which can be categorized into database logs, operation logs, transaction logs, etc.; or log categories can be determined based on log content, such as transfer transaction logs, verification transaction logs, and deposit transaction logs, etc. Of course, log categories can also be determined through other methods, such as using a pre-trained log classification model to determine the log category based on the log features of the input logs; or clustering the logs to determine the corresponding clusters as log categories.

[0066] Therefore, optionally, for the set of logs to be synchronized, a subset of logs to be synchronized is determined based on log categories. Specifically, this may include any of the following: (1) clustering the logs in the set of logs to be synchronized, and determining the log clusters in the clustering results as the subset of logs to be synchronized; (2) for the logs in the set of logs to be synchronized, determining the corresponding log categories based on log characteristics, and determining the subset of logs to be synchronized based on logs belonging to the same log category. This embodiment can determine the log categories by clustering or classification, which can improve the flexibility of log category determination and facilitate adaptation to different needs.

[0067] The embodiments of this application do not limit the method of log clustering. Optionally, clustering can be performed based on log features, such as the transaction type, source, and format of the logs. This makes it easy to identify logs with high similarity as a log cluster, thus facilitating subsequent differential encoding based on the high log similarity within the cluster and improving the compression effect of differential encoding. The log cluster itself can be defined as a log category.

[0068] The embodiments of this application do not limit the specific method of determining the corresponding log category based on log features. Optionally, the corresponding log category can be determined directly based on preset rules or a pre-trained log classification model, wherein the log category can be preset or determined in advance through log clustering. In a specific example, the log category can be determined in advance through log clustering, and subsequently, the log cluster to which the log belongs can be determined based on the similarity between the log features and the log features of the log cluster, thereby determining the corresponding log category.

[0069] By classifying logs into categories and determining the subset of logs to be synchronized, it can be confirmed that the logs in the subset belong to the same log category. This facilitates subsequent differential encoding. Based on the same log category, the compression effect of differential encoding can be improved. Furthermore, it can be combined with the subsequent differential encoding steps of log synchronization as a whole technical solution to reduce the amount of data transmitted during log synchronization, improve the efficiency of log synchronization, and reduce the load of log synchronization.

[0070] Second, in operation S220, for any subset of logs to be synchronized, the corresponding base log is determined, and differential encoding is performed on the logs in the target log subset based on the determined base log to obtain the log differential encoding set corresponding to the target log subset.

[0071] The embodiments of this application do not limit the specific method of determining the baseline log. Optionally, the corresponding baseline log can be determined in real time for the subset of logs to be synchronized, or the baseline log can be determined by summarizing the distribution of log information in the subset of logs to be synchronized; alternatively, the baseline log can be determined in advance for the log category based on the log category corresponding to the subset of logs to be synchronized.

[0072] Therefore, optionally, determining the corresponding baseline log for any subset of logs to be synchronized may include at least one of the following: (1) for any subset of logs to be synchronized, determining the preset baseline log corresponding to the log category to which it belongs as the baseline log for the target log subset; different log categories correspond to different preset baseline logs; (2) for any subset of logs to be synchronized, determining the current corresponding baseline log by comprehensively considering the logs in the target log subset. This embodiment can provide two different methods for determining the baseline log. It can use the baseline log corresponding to the log category in advance, or determine the baseline log in real time. It is convenient to choose the method for determining the baseline log according to actual needs, thereby improving the flexibility of the baseline log determination and adapting to different needs.

[0073] The embodiments of this application do not limit the specific method for determining the preset baseline log corresponding to the log category.

[0074] Optionally, pre-defined baseline logs can be generated based on the characteristics of different log categories. For example, logs for transfer transactions can have a fixed template, allowing corresponding pre-defined baseline logs to be generated based on that template.

[0075] In addition, optionally, the preset baseline log corresponding to the log category can be updated, specifically based on actual conditions. For example, if there are significant changes in the logs of a log category, the corresponding preset baseline log can be updated. In a specific embodiment, the logs of the transaction log category can contain date information, and the date information can contain year information. Typically, within the same year, the year information in the preset baseline log can be determined as that year to facilitate differential encoding, performing differential encoding on the month and day. As the year increases, the year information in the preset baseline log can be updated accordingly.

[0076] The embodiments of this application do not limit the specific method of determining the baseline log through comprehensive log analysis.

[0077] Optionally, a log can be randomly selected from the target log subset as the base log. Since the logs in the subset to be synchronized have a high degree of similarity, differential encoding can be performed based on any log as the base log. Alternatively, the distribution of log information in the target log subset can be analyzed. For example, analyzing the frequently repeated log field values ​​in the target log subset allows the frequently repeated log field values ​​to be used as the corresponding log field values ​​in the base log, facilitating differential encoding. Alternatively, the distribution range or value range of log information in the target log subset can be analyzed to easily determine a base value for log information, which can then be added to the base log for convenient differential encoding.

[0078] Therefore, optionally, the current baseline log is determined by comprehensively considering the logs in the target log subset. Specifically, this may include: determining the corresponding baseline field value based on different field values ​​of the same log field among different logs in the target log subset; and determining the current baseline log corresponding to the target log subset by combining the baseline field values ​​of different log fields. This embodiment can determine the baseline field value based on different field values ​​of log fields to determine the baseline log, which can improve the accuracy and efficiency of determining the baseline log.

[0079] The embodiments of this application do not limit the specific method for determining the baseline field value. Optionally, the baseline field value can be determined based on statistical values ​​such as the mode, average, or weighted average among different field values ​​of the same log field. It is understood that statistical values ​​such as the mode can facilitate subsequent differential coding.

[0080] The embodiments of this application do not limit the specific method of determining the baseline log by combining the baseline field values. Optionally, the baseline log corresponding to the current log subset can be obtained by combining the baseline field values ​​of various log fields; alternatively, the baseline log corresponding to the current log subset can be obtained by combining different log fields and their corresponding baseline field values.

[0081] Optionally, the current baseline log determined for the subset of logs to be synchronized can be further determined as a preset baseline log corresponding to the log category to which the subset of logs belongs; specifically, it can be a new preset baseline log. It is understood that one embodiment of determining the preset baseline log can be based on the above embodiments.

[0082] The embodiments of this application do not limit the number of preset benchmark logs corresponding to the same log category. Optionally, the same log category may correspond to one or more preset benchmark logs. Multiple preset benchmark logs corresponding to the log category of any subset of logs to be synchronized can be filtered to select the preset benchmark log that is suitable for the target log subset. The embodiments of this application do not limit the specific filtering method or basis. Optionally, a portion of the logs can be sampled from the target log subset, and the compression effect can be determined using multiple preset benchmark logs, thereby selecting a preset benchmark log with a compression effect superior to other preset benchmark logs.

[0083] The embodiments of this application are not limited to a specific process for selecting the baseline log determination method according to requirements.

[0084] Optionally, the selection can be made based on the number of logs in the log subset. When the number of logs in the log subset is small, the appropriate baseline log can be determined in real time. When the number of logs in the log subset is large, the preset baseline log corresponding to the log category can be used. When there are many logs to be synchronized, or the log synchronization frequency is high, or the timeliness requirement for log synchronization is high, the preset baseline log can be used to improve the efficiency of differential coding.

[0085] In one alternative embodiment, the baseline log can be used for differential coding, thereby determining whether an update is needed based on the differential coding compression effect of the baseline log, and thus selecting an appropriate baseline log determination method.

[0086] The embodiments of this application do not limit the specific method and process for determining the compression effect. Optionally, the compression effect can be determined based on a preset baseline log for the current subset of logs to be synchronized. Specifically, the compression effect can be determined by differential encoding based on the preset baseline log for the entire subset of logs to be synchronized or a sampled portion of the logs. Alternatively, the compression effect can be determined based on a preset baseline log for a subset of historically synchronized logs. Since the historically synchronized log subset does not need to be synchronized in real time, it can be executed in parallel with the log synchronization process to determine the compression effect of the preset baseline log. Specifically, for any log category, the compression effect can be determined for the corresponding historically synchronized log subset using the corresponding preset baseline log, where the log category to which the historically synchronized log subset belongs can be the target log category.

[0087] Understandably, for each log category, a preset baseline log can be used to determine the corresponding compression effect based on a subset of historically synchronized logs. Furthermore, the compression effect can be used to determine whether the preset baseline log needs to be updated. Optionally, the compression effect can be updated accordingly as the subset of historically synchronized logs is updated. In a specific example, as log synchronization continues, a preset baseline log that previously had a good compression effect may gradually become incompatible with new logs, resulting in a decrease in compression effect or degree. Of course, for a subset of historically synchronized logs, a baseline log determined by other methods can be used to determine the compression effect or degree.

[0088] The embodiments of this application do not limit the specific method for determining the compression effect or degree of compression. Optionally, the compression effect or degree of compression can be determined based on the amount of data before and after differential encoding based on the baseline log. Specifically, the compression effect or degree of compression can be determined by the ratio between the amount of data after differential encoding based on the baseline log and the amount of log data before differential encoding based on the baseline log (that is, the amount of data in the original log subset).

[0089] Therefore, optionally, the above method flow may further include: for any historically synchronized log subset, determining the compression level of the corresponding baseline log based on the data volume of the target log subset and the data volume of the corresponding log differential coding set. Specifically, determining the corresponding baseline log for any log subset to be synchronized may include: for any log subset to be synchronized, if the compression level of the preset baseline log corresponding to the log category is determined to be higher than a preset compression level threshold, the preset baseline log corresponding to the log category is determined as the baseline log for the target log subset; different log categories correspond to different preset baseline logs; for any log subset to be synchronized, if the compression level of the preset baseline log corresponding to the log category is determined to be lower than a preset compression level threshold, the current corresponding baseline log is determined by comprehensively considering the logs in the target log subset, and the determined baseline log is set as the current preset baseline log corresponding to the log category. This embodiment can select the method for determining the baseline log based on its compression level, and can choose to generate the baseline log in real time or directly use the preset baseline log based on its compression level, thereby improving the flexibility of baseline log determination and facilitating the improvement of the compression effect of log differential coding.

[0090] Of course, the embodiments of this application do not limit the specific method and process for determining the baseline log compression level. Optionally, the compression level of the baseline log can be determined based on historical logs or a subset of historically synchronized logs, or the corresponding baseline log compression level can be determined directly for the subset of logs to be synchronized, facilitating the selection of the baseline log determination method. Specifically, a portion of the logs can be sampled from the subset of logs to be synchronized, and the compression level can be determined based on the baseline log (a preset baseline log or a currently determined baseline log). This allows for comparison of the compression levels of different baseline logs (the compression levels of the preset baseline log and the currently determined baseline log can be compared), facilitating the selection of a baseline log with a higher compression level than other baseline logs.

[0091] Therefore, optionally, for any subset of logs to be synchronized, a corresponding baseline log is determined. Specifically, this may include: determining multiple candidate baseline logs for any subset of logs to be synchronized; and determining the baseline log corresponding to the specific log subset from the determined candidate baseline logs based on their compression levels. This embodiment can select the baseline log based on its compression level, which can improve the compression effect of differential coding based on the baseline log.

[0092] The embodiments of this application do not limit the method for determining the compression level of candidate benchmark logs. Specifically, the compression level can be determined for historical logs or for a subset of logs to be synchronized. For a detailed explanation, please refer to other embodiments. Based on the compression level of candidate benchmark logs, the corresponding benchmark log is determined. Specifically, a candidate benchmark log with a compression level higher than other candidate benchmark logs can be determined as the benchmark log corresponding to the targeted log subset.

[0093] The embodiments of this application do not limit the method of determining candidate benchmark logs. For details, please refer to the embodiments of various benchmark log determination methods. Multiple candidate benchmark logs can be determined in a variety of different ways, or multiple preset benchmark logs corresponding to the log category can be directly determined as candidate benchmark logs.

[0094] In another alternative embodiment, an appropriate baseline log determination method can be selected based on the actual situation of log synchronization. Specifically, this could be based on the timeliness requirements of log synchronization, the quantity or frequency of log synchronization, or the needs of log synchronization, etc.

[0095] Therefore, optionally, for any subset of logs to be synchronized, a corresponding baseline log is determined. Specifically, this may include: for any subset of logs to be synchronized, if the current log synchronization frequency is higher than a preset frequency threshold, determining the preset baseline log corresponding to the log category as the baseline log for the target log subset; different log categories correspond to different preset baseline logs; for any subset of logs to be synchronized, if the current log synchronization frequency is lower than a preset frequency threshold, determining the current corresponding baseline log by comprehensively considering the logs in the target log subset. This embodiment can select the method for determining the baseline log based on the current log synchronization frequency, adapting to the current log synchronization frequency requirements, improving the flexibility and accuracy of baseline log determination, and conveniently meeting the needs of log synchronization.

[0096] Third, in operation S230, the log differential encoding set corresponding to any log subset to be synchronized is synchronized to the target end; the target end is used to: determine the corresponding base log for any received log differential encoding set, and determine the corresponding log subset to be synchronized based on the determined base log and the corresponding log differential encoding set.

[0097] The embodiments of this application do not limit the specific process of synchronizing the log differential encoding set to the target end.

[0098] Optionally, the log differential encoding set can be sent directly to the target end, or the log differential encoding set can be encrypted and sent to the target end. Specifically, the target end's public key can be used for encryption. Alternatively, the integrity verification information of the log differential encoding set can be sent to the target end to facilitate the target end's verification of the integrity of the log differential encoding set.

[0099] In one optional embodiment, since the target end needs to perform recovery based on the baseline log corresponding to the log differential coding set, it also needs to synchronize the corresponding baseline log to the target end. The embodiments of this application do not limit the specific baseline log synchronization process.

[0100] Optionally, the corresponding baseline log can be sent directly to the target, or it can be sent in encrypted form, or the integrity verification information of the baseline log can be added, etc.

[0101] It is understood that the synchronization process of the log differential encoding set and the synchronization process of the corresponding baseline log can be executed in parallel or sequentially, and the embodiments of this application are not limited thereto.

[0102] In one optional embodiment, each baseline log can be pre-synchronized to the target end, thereby facilitating the subsequent direct determination of the baseline log corresponding to any log differential encoding set based on the identification information of the baseline log.

[0103] Therefore, optionally, the above method flow may further include: synchronizing the determined baseline log to the target end. Correspondingly, synchronizing the log differential encoding set corresponding to any subset of logs to be synchronized to the target end may specifically include: synchronizing the log differential encoding set corresponding to any subset of logs to be synchronized, and the identification information of the corresponding baseline log, to the target end; the target end can then: for any received log differential encoding set, determine the corresponding baseline log based on the identification information of the corresponding baseline log. This embodiment can conveniently determine the baseline log based on its identification information by synchronizing the baseline log, without needing to carry the corresponding baseline log during the synchronization of the log differential encoding set, thus improving the information security of log synchronization. Furthermore, by synchronizing the baseline log once, the target end can use the same baseline log multiple times for log recovery without repeatedly transmitting the same baseline log, reducing the amount of data transmitted during log synchronization and improving the efficiency of log synchronization.

[0104] In another optional embodiment, an integrity verification step can be added to the log synchronization process to improve the integrity and accuracy of log information. Optionally, the integrity verification can be performed on the log differential encoding set or on the log subset obtained after recovery to determine whether it is the same as the corresponding log subset to be synchronized.

[0105] Therefore, optionally, the log differential encoding set corresponding to any log subset to be synchronized is synchronized to the target end. Specifically, this may include: for any log subset to be synchronized, synchronizing the corresponding log differential encoding set and the integrity verification information corresponding to the target log subset to the target end. Correspondingly, the target end can be used to: for any received log differential encoding set, determine the corresponding base log, and based on the determined base log and the target log differential encoding set, determine the log subset to be verified; and based on the corresponding integrity verification information, perform integrity verification on the determined log subset to be verified. This embodiment can improve the data consistency, integrity, and accuracy of log synchronization by providing integrity verification information for the log subset itself, facilitating integrity verification by the target end after restoring the log subset.

[0106] The log subset to be verified can be obtained by the target end based on the log differential encoding set and the corresponding baseline log. It may be the same as or different from the original log subset to be synchronized. Therefore, the integrity of the log subset to be verified can be verified through the integrity verification information corresponding to the log subset.

[0107] The embodiments of this application do not limit the specific integrity verification process and integrity verification information. Optionally, the integrity verification information may be a hash value or a digest, etc. Optionally, a hash value can be determined for the subset of logs to be verified, so that it can be compared with the integrity verification information corresponding to the target log subset to determine the integrity of the subset of logs to be verified, that is, to determine whether the subset of logs to be verified is the same as the target subset of logs to be synchronized.

[0108] Optionally, the log differential encoding set corresponding to any subset of logs to be synchronized can be synchronized to the target end. Specifically, this can include encrypting and synchronizing the log differential encoding set corresponding to any subset of logs to be synchronized, along with the integrity verification information corresponding to the log differential encoding set, to the target end. Correspondingly, the target end can: for any received log differential encoding set, if the integrity verification information passes, determine the corresponding base log, and based on the determined base log and the targeted log differential encoding set, determine the corresponding subset of logs to be synchronized. This embodiment, by providing integrity verification information for the log differential encoding set, enables the target end to perform integrity verification on the synchronized log differential encoding set, determining whether the log differential encoding set has been tampered with during log synchronization, thereby improving the security, data consistency, accuracy, and integrity of log synchronization. Furthermore, the encrypted synchronization method also enhances the information security of log synchronization.

[0109] The embodiments of this application do not limit the specific integrity verification process and integrity verification information. Optionally, the integrity verification information may be a hash value or digest, etc. Optionally, a hash value can be determined for the received log differential encoding set, so that it can be compared with the integrity verification information corresponding to the log differential encoding set to determine the integrity of the received log differential encoding set, that is, to determine whether the received log differential encoding set is the same as the sent log differential encoding set.

[0110] It is understood that the above-mentioned integrity verification implementations can be combined with each other, specifically, two consecutive integrity verifications can be performed to improve the integrity and accuracy of log information.

[0111] Embodiments of this application may also provide another log processing method, which can be applied to a distributed system. Optionally, the distributed system may include a source and a target end that need to perform log synchronization.

[0112] The method flow of this embodiment may include: At the source end, for the set of logs to be synchronized, a subset of logs to be synchronized is determined based on log categories; different logs within any subset of logs to be synchronized belong to the same log category. At the source end, for any subset of logs to be synchronized, a corresponding base log is determined, and differential encoding is performed on the logs in the target subset based on the determined base log to obtain a log differential encoding set corresponding to the target subset. The source end synchronizes the log differential encoding set corresponding to any subset of logs to be synchronized to the target end.

[0113] Accordingly, the target end can determine the corresponding base log for any received log differential encoding set, and based on the determined base log and the corresponding log differential encoding set, determine the corresponding subset of logs to be synchronized. Therefore, the target end can obtain the subset of logs to be synchronized from the source end. Other subsets of logs to be synchronized can be synchronized to the target end in the same way, thus enabling the target end to obtain the entire set of logs to be synchronized from the source end.

[0114] For a detailed explanation of this method embodiment, please refer to the explanations of other embodiments. The target device can execute the corresponding steps to restore the logs and complete log synchronization.

[0115] For ease of understanding, this application also provides an application embodiment.

[0116] In banking operations, log synchronization is a key step in ensuring data consistency and system reliability. This includes different types of logs such as database logs, business logs, operation logs, and transaction logs.

[0117] This embodiment provides a method to improve the efficiency of bank log synchronization, which can effectively reduce network bandwidth resource consumption, reduce costs, improve log synchronization efficiency, and enhance the reliability and security of log data synchronization.

[0118] This embodiment provides a method to improve the efficiency of bank log synchronization. It may involve an intelligent semantic parsing module, an intelligent classification module, a log collection and preprocessing module, a dynamic differential encoding module, a database log compression module, and a database log recovery module.

[0119] By introducing an intelligent semantic parsing module and employing a hybrid parsing model based on a business rule engine and deep learning, semantic features such as transaction type (e.g., transfer, deposit / withdrawal), account identifier, transaction amount, and timestamp are extracted from logs to generate structured semantic tags. The hybrid parsing model comprises a rule engine module and a semantic fine-tuning model. The rule engine module predefines multiple transaction log syntax rules for parsing fixed fields such as account type and transaction code. The semantic fine-tuning model performs semantic extraction on nested transaction logs from banks, and its training samples can include multiple historical transaction logs.

[0120] By introducing a dynamic configuration management module, and based on the semantic features output by the intelligent semantic parsing module, and building a dynamic classification model based on the characteristics of banking business cycles (such as weekdays / holidays, peak / off-peak periods), logs are divided into semantic clusters such as real-time transaction type, batch clearing type, and reconciliation type.

[0121] By introducing a log collection and preprocessing module, log information is collected, and redundant information, error information and other invalid content are removed, while retaining complete log data and encrypted verification segments.

[0122] A dynamic differential encoding module is introduced to employ different encoding strategies for different semantic clusters. An intelligent compression decision-maker is configured to dynamically adjust the encoding strategy by monitoring parameters such as transaction peak coefficients and field volatility indices in real time. For example, when the transaction volume exceeds 5000 transactions per second, it automatically switches to a fast compression mode, reducing the update frequency of the baseline log; for foreign exchange transaction logs during periods of exchange rate fluctuation, it automatically improves the precision of the differential encoding of the amount field. Furthermore, it receives compression ratio evaluations from the log compression module and adjusts the baseline log accordingly to improve the compression ratio.

[0123] By introducing a log compression module, an intelligent header is generated, containing a semantic cluster identifier, a baseline log version number, compression strategy parameters, sender log summary information, and a compression ratio. The intelligent header includes an encrypted strategy identifier and a log integrity checksum, used by the receiver to verify that the compressed package has not been tampered with and is consistent with the sender's log. It also supports cross-system strategy compatibility identification. The compression ratio is used to evaluate the compression effect of the baseline log; if the compression ratio is low, it can be gradually increased by using other baseline logs. The intelligent header and compressed data segments are combined to form an encrypted compressed package. If a fast compression mode is used, encryption is not used to reduce computational overhead.

[0124] By introducing a log recovery module, the encrypted compressed package transmitted from the log compression module is verified for integrity based on the checksum in the smart header. Then, the log digest information from the sending end is extracted from the smart header. At the same time, the received compressed data segment is decompressed, and the hash value and key information digest of the decompressed log are calculated. Then, the digest information calculated by the receiving end is compared with that of the sending end, and then it is sent to each backup node through the network.

[0125] The method for improving the efficiency of bank log synchronization provided in this embodiment may include the following steps.

[0126] Step 1: The log collection and preprocessing module collects transaction logs through a dedicated bank interface, removes redundant information, error messages and other invalid content, and retains complete log data and encrypted verification segments.

[0127] Step 2: The intelligent semantic parsing module adopts a hybrid parsing model based on business rule engine and deep learning to extract semantic features such as transaction type (e.g., transfer, deposit and withdrawal), account identifier, transaction amount, timestamp, etc. from the logs and generate structured semantic tags.

[0128] Step 3: The intelligent classification module constructs a dynamic classification model based on the characteristics of banking business cycles (such as weekdays / holidays, peak / off-peak periods), dividing logs into semantic clusters such as real-time transaction type, batch clearing type, and reconciliation type.

[0129] Step 4: The dynamic differential encoding module employs intelligent differential encoding strategies for different semantic clusters. An intelligent compression decision-maker is set up to dynamically adjust the encoding strategy by monitoring parameters such as transaction peak coefficients and field volatility indices in real time. For example, when the transaction volume exceeds 5000 transactions per second, it automatically switches to fast compression mode, reducing the update frequency of the baseline log; for foreign exchange transaction logs during periods of exchange rate fluctuation, it automatically improves the differential encoding accuracy of the amount field. Furthermore, it receives compression ratio evaluations from the log compression module and adjusts the baseline log accordingly to improve the compression ratio.

[0130] The specific strategies are as follows: (1) Real-time transaction type: adopt the sliding window benchmark log mechanism, automatically update the benchmark log every 500 transactions, and use variable length difference encoding for fluctuation fields such as amount and last four digits of account; (2) Batch settlement type: generate benchmark log based on preset settlement template, and use incremental encoding for aggregate fields such as number of transactions and total amount; (3) Reconciliation and verification type: retain complete key fields, and only use hash difference encoding for duplicate verification information.

[0131] Step 5: The log compression module generates a smart header containing a semantic cluster identifier, a baseline log version number, compression strategy parameters, and sender log summary information. The smart header includes an encrypted strategy identifier and a log integrity checksum, used by the receiver to verify that the compressed package has not been tampered with and is consistent with the sender's logs. It also supports cross-bank system strategy compatibility identification. The smart header is combined with the compressed data segment to form an encrypted compressed package. If a fast compression mode is used, encryption is not used to reduce computational overhead.

[0132] Step Six: The relevant operations of the log recovery module include: Before sending the compressed package, the sending end calculates the hash value and key information digest of the log and embeds them into the smart header; after receiving the compressed package, the receiving end first verifies the integrity of the compressed package according to the checksum in the smart header. If it is incomplete, it requests retransmission; after the integrity verification is passed, it extracts the log digest information of the sending end from the smart header, and decompresses the received compressed data segment, calculates the hash value and key information digest of the decompressed log; it compares the digest information calculated by the receiving end with that of the sending end. If they match, it means that the log has not changed during transmission, and the consistency is guaranteed. Then, it completes the log reconstruction in conjunction with the dynamic benchmark log library and writes it into the synchronization log; if they do not match, it triggers the prompt mechanism and starts the data repair process.

[0133] This embodiment can compress logs by using differential encoding based on baseline logs, reducing the amount of data synchronized by logs. This can effectively reduce network bandwidth resource consumption, lower costs, improve log synchronization efficiency, and enhance data reliability and security. It can also achieve at least the following beneficial effects: (1) Improved business adaptability: The compression strategy optimized for the banking business cycle can maintain a high compression rate even during peak transaction periods, effectively adapting to fluctuations in banking operations. (2) Improved consistency checking capability: Through dual digest comparison and integrity verification between the sender and receiver, the consistency of logs from sender to receiver is ensured, greatly reducing the risk of data transmission errors and tampering, and improving the reliability of log data. (3) Security and reliability: Algorithm encryption improves the security of bank data. (4) The dynamic adjustment mechanism ensures that the compression performance of the system is less affected by fluctuations in log volume, exhibiting strong stability and adaptability. (5) High compatibility.

[0134] Corresponding to the above method embodiments, this application also provides a log processing apparatus. The following will be combined with... Figure 3 The device is described in detail.

[0135] Figure 3 The diagram illustrates a structural block diagram of a log processing apparatus according to an embodiment of this application.

[0136] like Figure 3 As shown, the log processing device 300 provided in this embodiment includes: a category module 310, a baseline module 320, and a synchronization module 330.

[0137] The category module 310 is used to determine a subset of logs to be synchronized based on log categories for the set of logs to be synchronized; different logs in any subset of logs to be synchronized belong to the same log category. In one embodiment, the category module 310 can be used to perform the operation S210 and related steps described above, which will not be repeated here.

[0138] The baseline module 320 is used to determine the corresponding baseline log for any subset of logs to be synchronized, and to perform differential encoding on the logs in the target log subset based on the determined baseline log, thereby obtaining the log differential encoding set corresponding to the target log subset. In one embodiment, the baseline module 320 can be used to execute the operation S220 and related steps described above, which will not be repeated here.

[0139] Synchronization module 330 is used to synchronize the log differential encoding set corresponding to any log subset to be synchronized to the target end; the target end is used to: for any received log differential encoding set, determine the corresponding base log, and based on the determined base log and the corresponding log differential encoding set, determine the corresponding log subset to be synchronized. In one embodiment, synchronization module 330 can be used to perform the operation S230 and related steps described above, which will not be repeated here.

[0140] Optionally, the baseline module 320 is used to perform at least one of the following: (1) for any log subset to be synchronized, the preset baseline log corresponding to the log category to be synchronized is determined as the baseline log corresponding to the log subset to be synchronized; different log categories correspond to different preset baseline logs; (2) for any log subset to be synchronized, the current corresponding baseline log is determined by combining the logs in the log subset to be synchronized.

[0141] Optionally, the benchmark module 320 is further configured to: for any historically synchronized log subset, determine the compression level of the corresponding benchmark log based on the data volume of the targeted log subset and the data volume of the corresponding log differential encoding set. The benchmark module 320 is configured to: for any log subset to be synchronized, if it is determined that the compression level of the preset benchmark log corresponding to the log category is higher than a preset compression level threshold, determine the preset benchmark log corresponding to the log category as the benchmark log corresponding to the targeted log subset; different log categories correspond to different preset benchmark logs; for any log subset to be synchronized, if it is determined that the compression level of the preset benchmark log corresponding to the log category is lower than a preset compression level threshold, comprehensively analyze the logs in the targeted log subset to determine the current corresponding benchmark log, and set the determined benchmark log as the current preset benchmark log corresponding to the log category.

[0142] Optionally, the reference module 320 is used to: for any subset of logs to be synchronized, if it is determined that the current log synchronization frequency is higher than a preset frequency threshold, determine the preset reference log corresponding to the log category as the reference log corresponding to the target log subset; different log categories correspond to different preset reference logs; for any subset of logs to be synchronized, if it is determined that the current log synchronization frequency is lower than a preset frequency threshold, determine the current corresponding reference log by comprehensively considering the logs in the target log subset.

[0143] Optionally, the baseline module 320 is used to: determine the corresponding baseline field value among different logs in the targeted log subset based on different field values ​​of the same log field; and determine the current baseline log corresponding to the targeted log subset by combining the baseline field values ​​of different log fields.

[0144] Optionally, the synchronization module 330 is further configured to: synchronize the determined baseline log to the target end; the synchronization module 330 is configured to: synchronize the log differential encoding set corresponding to any subset of logs to be synchronized, and the identification information of the corresponding baseline log to the target end; the target end is configured to: for any received log differential encoding set, determine the corresponding baseline log based on the identification information of the corresponding baseline log.

[0145] Optionally, the synchronization module 330 is used to: for any log subset to be synchronized, synchronize the corresponding log differential encoding set and the integrity verification information corresponding to the log subset to the target end; the target end is used to: for any received log differential encoding set, determine the corresponding base log, and based on the determined base log and the corresponding log differential encoding set, determine the log subset to be verified; and based on the corresponding integrity verification information, perform integrity verification on the determined log subset to be verified.

[0146] Optionally, the synchronization module 330 is used to: encrypt and synchronize the log differential encoding set corresponding to any log subset to be synchronized, and the integrity verification information corresponding to the log differential encoding set, to the target end; the target end is used to: for any received log differential encoding set, if the verification is passed based on the corresponding integrity verification information, determine the corresponding base log, and based on the determined base log and the targeted log differential encoding set, determine the corresponding log subset to be synchronized.

[0147] Scientifically, the category module 310 is used to perform any of the following: (1) clustering the logs in the log set to be synchronized, and determining the log clusters in the clustering results as the log subset to be synchronized; (2) for the logs in the log set to be synchronized, determining the corresponding log category based on log characteristics, and determining the log subset to be synchronized based on logs belonging to the same log category.

[0148] Optionally, the benchmark module 320 is used to: determine multiple candidate benchmark logs for any subset of logs to be synchronized; and determine the benchmark log corresponding to the specified log subset from the determined candidate benchmark logs based on the compression degree of the candidate benchmark logs.

[0149] According to embodiments of this application, any plurality of modules among category module 310, reference module 320, and synchronization module 330 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules can be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of this application, at least one of category module 310, reference module 320, and synchronization module 330 can be at least partially implemented as hardware circuitry, such as field-programmable gate array (FPGA), programmable logic array (PLA), system-on-a-chip, system-on-a-substrate, system-on-package, application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in any one of software, hardware, and firmware, or in a suitable combination of any of these. Alternatively, at least one of category module 310, reference module 320, and synchronization module 330 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0150] For an explanation of this device embodiment, please refer to other embodiments. Each embodiment in the above method embodiment can be executed by the corresponding module in this device embodiment.

[0151] Figure 4 A block diagram schematically illustrates an electronic device suitable for implementing a log processing method according to an embodiment of this application.

[0152] like Figure 4 As shown, an electronic device 900 according to an embodiment of this application includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage portion 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0153] RAM 903 stores various programs and data required for the operation of electronic device 900. Processor 901, ROM 902, and RAM 903 are interconnected via bus 904. Processor 901 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 902 and / or RAM 903. It should be noted that the programs may also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0154] According to embodiments of this application, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to a bus 904. The electronic device 900 may also include one or more of the following components connected to the input / output (I / O) interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output (I / O) interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 910 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0155] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0156] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 902 and / or RAM 903 and / or one or more memories other than ROM 902 and RAM 903 described above.

[0157] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement any of the method embodiments provided in the embodiments of this application.

[0158] When the computer program is executed by the processor 901, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0159] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 909, and / or installed from a removable medium 911. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0160] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0161] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0162] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0163] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

Claims

1. A log processing method, characterized in that, The method includes: For a set of logs to be synchronized, a subset of logs to be synchronized is determined based on the log category; different logs in any subset of logs to be synchronized belong to the same log category. For any subset of logs to be synchronized, determine the corresponding baseline log, and perform differential encoding on the logs in the target log subset based on the determined baseline log to obtain the log differential encoding set corresponding to the target log subset. Synchronize the log differential encoding set corresponding to any log subset to be synchronized to the target end; the target end is used to: determine the corresponding base log for any received log differential encoding set, and determine the corresponding log subset to be synchronized based on the determined base log and the corresponding log differential encoding set.

2. The method according to claim 1, characterized in that, For any subset of logs to be synchronized, determining the corresponding baseline log includes at least one of the following: For any subset of logs to be synchronized, the preset baseline log corresponding to the log category is determined as the baseline log for the target log subset; different log categories correspond to different preset baseline logs. For any subset of logs to be synchronized, the corresponding baseline log is determined by combining the logs in the subset.

3. The method according to claim 1, characterized in that, The method further includes: for any historical synchronized log subset, determining the compression level of the corresponding baseline log based on the data volume of the target log subset and the data volume of the corresponding log differential coding set; The step of determining the corresponding baseline log for any subset of logs to be synchronized includes: For any subset of logs to be synchronized, if the compression level of the preset baseline log corresponding to the log category is higher than the preset compression level threshold, the preset baseline log corresponding to the log category is determined as the baseline log for the target log subset; different log categories correspond to different preset baseline logs. For any subset of logs to be synchronized, if the compression level of the preset baseline log corresponding to the log category is determined to be lower than the preset compression level threshold, the current baseline log is determined by comprehensively considering the logs in the target log subset, and the determined baseline log is set as the current preset baseline log corresponding to the log category.

4. The method according to claim 1, characterized in that, The step of determining the corresponding baseline log for any subset of logs to be synchronized includes: For any subset of logs to be synchronized, if it is determined that the current log synchronization frequency is higher than a preset frequency threshold, the preset baseline log corresponding to the log category is determined as the baseline log for the target log subset; different log categories correspond to different preset baseline logs. For any subset of logs to be synchronized, if the current log synchronization frequency is determined to be lower than a preset frequency threshold, the corresponding baseline log is determined by combining the logs in the target subset.

5. The method according to claim 2, characterized in that, The synthesis of logs within the specified log subset determines the current baseline log, including: Among different logs in the target log subset, the corresponding baseline field value is determined based on different field values ​​of the same log field; by combining the baseline field values ​​of different log fields, the current baseline log corresponding to the target log subset is determined.

6. The method according to claim 1, characterized in that, The method further includes: synchronizing the determined baseline log to the target terminal; The step of synchronizing the log differential encoding set corresponding to any subset of logs to be synchronized to the target end includes: synchronizing the log differential encoding set corresponding to any subset of logs to be synchronized, as well as the identification information of the corresponding base log, to the target end; The target end is used to: for any received log differential encoding set, determine the corresponding base log based on the identification information of the corresponding base log.

7. The method according to claim 1, characterized in that, The step of synchronizing the log differential encoding set corresponding to any subset of logs to be synchronized to the target end includes: For any subset of logs to be synchronized, the corresponding log differential encoding set and the integrity verification information corresponding to the log subset are synchronized to the target end. The target end is used to: determine the corresponding baseline log for any received log differential encoding set, and determine the log subset to be verified based on the determined baseline log and the corresponding log differential encoding set; and perform integrity verification on the determined log subset to be verified based on the corresponding integrity verification information.

8. The method according to claim 1, characterized in that, The step of synchronizing the log differential encoding set corresponding to any subset of logs to be synchronized to the target end includes: The log differential encoding set corresponding to any subset of logs to be synchronized, and the integrity verification information corresponding to the log differential encoding set, are encrypted and synchronized to the target end. The target end is used to: for any received log differential encoding set, if the verification is passed based on the corresponding integrity verification information, determine the corresponding base log, and based on the determined base log and the targeted log differential encoding set, determine the corresponding log subset to be synchronized.

9. The method according to claim 1, characterized in that, The step of determining a subset of logs to be synchronized based on log categories for the set of logs to be synchronized includes any one of the following: Cluster the logs in the set of logs to be synchronized, and determine the log clusters in the clustering results as the subset of logs to be synchronized; For logs in the log set to be synchronized, the corresponding log category is determined based on log characteristics, and a subset of logs to be synchronized is determined based on logs belonging to the same log category.

10. The method according to claim 1, characterized in that, The step of determining the corresponding baseline log for any subset of logs to be synchronized includes: For any subset of logs to be synchronized, determine multiple candidate baseline logs; Based on the compression level of the candidate benchmark logs, the benchmark logs corresponding to the target subset are determined from the identified candidate benchmark logs.

11. A log processing device, characterized in that, The device includes: The category module is used to determine the subset of logs to be synchronized based on the log category for a set of logs to be synchronized; different logs in any subset of logs to be synchronized belong to the same log category. The baseline module is used to determine the corresponding baseline log for any subset of logs to be synchronized, and to perform differential encoding on the logs in the target log subset based on the determined baseline log, so as to obtain the log differential encoding set corresponding to the target log subset. The synchronization module is used to synchronize the log differential encoding set corresponding to any log subset to be synchronized to the target end; the target end is used to: determine the corresponding base log for any received log differential encoding set, and determine the corresponding log subset to be synchronized based on the determined base log and the corresponding log differential encoding set.

12. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 10.

14. A computer program product comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 10.

Images