Log processing method, electronic device, storage medium, and program product

Abstract

Embodiments of the present application provide a log processing method, an electronic device, a storage medium and a program product, relating to the technical field of computer, and the method comprises: in the case that a log file of a controller is greater than or equal to a predetermined capacity threshold, obtaining a hash file corresponding to the log file; any hash metadata in the hash file is used to represent log content generated by one event processing logic of the controller triggered by an event on one resource object; grouping the hash metadata, the log content corresponding to the hash metadata in the same group belongs to the same event processing logic of the same resource object; judging the repetition of each hash metadata group to determine the repeated log content; and deduplicating the log file according to the repeated log content. In this scheme, the log content can be distinguished according to the resource object and the event processing logic, realizing log isolation in different dimensions, which is conducive to avoiding mutual influence during log deduplication and reducing the probability of deduplication failure.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a log processing method, electronic device, storage medium, and program product. Background Technology

[0002] In container cluster management systems, controllers can drive resource objects from their current state to a desired state through custom control loops. This pattern involves repeatedly executing the same service logic within the loop, generating a large amount of redundant logs. This is especially problematic in complex service scenarios with a large number of controllers and subscribed resources, leading to a continuous accumulation of redundant logs and wasted resources. Log deduplication is therefore necessary. However, currently, there is a lack of dedicated log deduplication solutions for the controller pattern. A log deduplication solution for the controller pattern is needed to reduce redundant logs and optimize resource utilization. Summary of the Invention

[0003] This application provides a log processing method, electronic device, storage medium, and program product to alleviate or solve one or more technical problems existing in the related art.

[0004] In a first aspect, embodiments of this application provide a log deduplication method, comprising: obtaining a hash file corresponding to the log file when the log file of the controller is greater than or equal to a predetermined capacity threshold; any hash metadata in the hash file is used to characterize the log content generated by an event processing logic executed by the controller for an event triggered by a resource object; grouping the hash metadata in the hash file to obtain each hash metadata group, wherein the log content corresponding to the hash metadata in the same group belongs to the same event processing logic of the same resource object; performing a duplicate judgment on each hash metadata group, determining the duplicate log content based on the determined duplicate hash metadata group; and performing deduplication processing on the log file based on the duplicate log content.

[0005] Secondly, embodiments of this application provide a controller log processing method for a container cluster management system, comprising: responding to a resource configuration change request of at least one resource object in the container cluster management system, executing event processing logic corresponding to the resource configuration change request, and generating corresponding log content based on the execution of the event processing logic; generating hash metadata corresponding to each log content, the hash metadata being used to characterize the log content generated by the controller executing the event processing logic once on the resource object; writing each log content and the corresponding hash metadata into a log file and a hash file respectively; and processing the log file and the hash file using the log processing method of the first aspect to obtain a deduplicated log file.

[0006] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the methods of embodiments of this application when executing the computer program.

[0007] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of any one of the embodiments of this application.

[0008] Fifthly, embodiments of this application provide a computer program product, including a computer program, which, when executed by a processor, implements any of the methods described in the embodiments of this application.

[0009] According to the log processing method of this application embodiment, the log content represented by the hash metadata in the hash file corresponds to an event processing logic executed by the controller on a resource object. After grouping the hash metadata, the log content of the same event processing logic belonging to the same resource object is divided into a hash metadata group. This allows the corresponding log content to be distinguished according to the resource object and the event processing logic, realizing log isolation of different dimensions.

[0010] In this embodiment, each hash metadata group can distinguish the log content of different resource objects, ensuring that the log content of different resource objects does not interfere with each other. This helps avoid mutual interference during log deduplication, thereby reducing the probability of deduplication failure. Furthermore, the log content is differentiated for different resource objects, so even if different resource objects trigger the same event, the log content of each resource object will not interfere with each other, reducing the probability of log incompleteness due to erroneous deletion of log content from other resource objects. Moreover, the grouping mechanism based on resource objects and event processing logic in this embodiment can group the hash metadata of the log content of the same event processing logic for the same resource object into the same group. This means that a single log entry can accurately locate an event processing logic for a specific resource object, helping to maintain the temporal sequence and integrity of the log content in terms of event processing logic. Due to its temporal sequence and integrity in event processing logic, the deduplicated logs not only reduce redundant logs and optimize resource usage, but also accurately locate the log of a specific event processing logic for a single resource object, without missing or redundant logs, facilitating rapid problem troubleshooting.

[0011] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0012] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0013] Figure 1 A flowchart of a log processing method according to an embodiment of this application is shown.

[0014] Figure 2 A detailed flowchart of a log processing method according to an exemplary embodiment of this application is shown.

[0015] Figure 3 A flowchart illustrating the controller log processing method of a container cluster management system according to an embodiment of this application is shown.

[0016] Figure 4 A schematic diagram of the log processing apparatus according to an embodiment of this application is shown.

[0017] Figure 5 A schematic diagram of the controller log processing device of the container cluster management system according to an embodiment of this application is shown.

[0018] Figure 6 A block diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0019] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0020] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.

[0021] In practical applications, Kubernetes (K8s) is an open-source container orchestration system used to automatically deploy, scale, and manage containerized applications. An application is the actual software running within the Kubernetes management system, such as a page server, database, or any other type of service. Resource objects are various entities that can be managed and manipulated by the Kubernetes management system, including but not limited to pods, services, and deployments. A pod is the basic deployment unit in the Kubernetes management system and can contain one or more containers. A service defines how a group of containers are accessed, such as accessing containers via network addresses inside or outside the Kubernetes management system. A deployment defines how an application is deployed, and may include, for example, the number of replicas, container image versions, etc. Considering that the Kubernetes management system is a cluster system for managing containerized applications, in the following descriptions, for ease of description, the Kubernetes management system may be simply referred to as a cluster or cluster system.

[0022] In this scenario, resource objects can be understood as tools used to describe and manage the deployment and runtime status of applications within a cluster. Container cluster management systems can declaratively configure and track resource objects, providing applications with elastic scaling and fault recovery capabilities, and offering rich service discovery, networking, and storage functionalities to support cloud-native application development and deployment.

[0023] In container cluster management systems, the controller is a core component responsible for driving the transformation of resource objects from their current state to the desired state. The controller pattern is a programming pattern used in container cluster management systems, implementing a set of operational mechanisms for resource management and coordination. The controller pattern is widely used in container cluster management systems. Specifically, relying on the underlying resource monitoring mechanism (List & Watch), the controller monitors resource state changes and responds accordingly. Specifically, the controller can obtain a list of resource objects through the Application Programming Interface Server (APIServer), subscribe to change events of specific resource objects on the APIServer, and monitor changes to resource objects in real time by establishing long-lived connections (Watch). When the resource configuration of a resource object changes, such as when a resource object is added, updated, or deleted, the APIServer can send an event notification to the controllers that have subscribed to the relevant resource objects. After receiving the event notification, the controller parses the message in the notification, triggers the controller's control loop based on the event type (e.g., add, update, or delete), and executes the corresponding operation.

[0024] In summary, when the current state and desired state of a resource object managed by the container cluster management system are inconsistent, the system sends an event to the corresponding controller. The controller then executes the appropriate event handling logic (i.e., controller logic), driving the resource object to change from its current state to the desired state. This processing model effectively responds to state changes of resource objects in the cluster without frequently polling the application programming interface server, effectively reducing system resource consumption and network traffic.

[0025] In normal operating mode, the controller continuously compares the current state and desired state of resource objects in the cluster through a continuously running control loop, and attempts to change the current state of resource objects towards the desired state by executing specific event handling logic. This control loop is the core of the controller, ensuring the self-healing and automatic adjustment capabilities of the cluster system. Logs are generated within the controller's control loop, which help operations personnel maintain and debug the container cluster management system.

[0026] If an error occurs while the controller is executing event handling logic, it may repeatedly execute the same logic to attempt to correct the problem multiple times until the resource object reaches the desired state. During this process, the controller generates a large amount of duplicate content in its logs, resulting in a large amount of redundant logs. This is especially problematic in complex service scenarios where there are many controllers and a large number of subscribed resource objects. The continuous accumulation of redundant logs wastes system resources, necessitating log deduplication. Log deduplication primarily involves simplifying some duplicate log entries to reduce log size and improve troubleshooting efficiency.

[0027] In related technologies, a unified log management system is provided through a log collector (Fluentd). Its log deduplication is based on configured conditions to identify duplicate log content, performs hash calculations on the log content to obtain the hash value of the log content, and then allows users to choose to remove log content with the same hash value within a specified time window.

[0028] However, in container cluster management systems, event processing cycles may span different time points, making the start and end times of resource object logs uncertain. Log collectors need to select log content within a specified time interval for deduplication. If this time interval doesn't match the container cluster management system's event processing cycle, it may truncate the logs for that cycle, causing deduplication to fail. Furthermore, in controller mode, the same event processing logic may be triggered by multiple different resource objects, resulting in logs from different objects being recorded simultaneously within the same time period. Since log collectors lack resource object-based log isolation mechanisms, direct deduplication may mistakenly delete logs from other objects, leading to incomplete logs. Additionally, in controller mode, the same object may trigger event processing logic multiple times, and the log content corresponding to these multiple events is not isolated or differentiated, making it impossible to guarantee the log's temporal order after deduplication and accurately pinpoint the time of the problem.

[0029] In related technologies, the open-source server-side data processing pipeline (Logstash) is also used to collect log content from multiple sources, calculate the hash value of the log content, and use the hash value as the data identifier of the log content. When storing log content, log content with the same data identifier will overwrite each other, thereby achieving log deduplication.

[0030] However, the order and timing of logs are crucial for understanding the causal relationships of events and troubleshooting. This log deduplication scheme is equivalent to storing only one copy of the log content with the same hash value in the log file. In scenarios where it is necessary to investigate and analyze the cause of errors, this may not be effective for problem investigation. Furthermore, data from multiple sources may have the same hash value, making it impossible to locate the source after the log content is stored. This will increase the difficulty of subsequent error investigation based on the log content.

[0031] It should be noted that the application scenarios or examples provided in the embodiments of this application are for ease of understanding, and the embodiments of this application do not specifically limit the application of the technical solutions. In addition, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0032] The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0033] Figure 1 A flowchart illustrating the method of an embodiment of this application is shown, as follows: Figure 1 As shown, the method may include steps S101, S102, S103 and S104.

[0034] Step S101: If the controller's log file is greater than or equal to a predetermined capacity threshold, obtain the hash file corresponding to the log file; any hash metadata in the hash file is used to characterize the log content generated by the controller executing an event processing logic triggered by an event of a resource object.

[0035] Step S102: Group the hash metadata in the hash file to obtain each hash metadata group. The log content corresponding to the hash metadata in the same group belongs to the same event processing logic of the same resource object.

[0036] Step S103: Perform a duplicate check on each hash metadata group, and determine the duplicate log content based on the identified duplicate hash metadata groups.

[0037] Step S104: Deduplicate the log files based on duplicate log content.

[0038] In the log processing method of this application embodiment, the log content represented by the hash metadata in the hash file corresponds to an event processing logic executed by the controller on a resource object. After grouping the hash metadata, the log content of the same event processing logic belonging to the same resource object is divided into a hash metadata group. This allows the corresponding log content to be distinguished according to the resource object and the event processing logic, realizing log isolation of different dimensions.

[0039] Compared to log collectors in related technologies, which lack a resource object-based log isolation mechanism and may mistakenly delete logs of other resource objects during deduplication, leading to incomplete logs, the hash metadata groups in this embodiment can distinguish the log content of different resource objects, preventing interference between their log contents. This helps avoid mutual interference during log deduplication, thus reducing the probability of deduplication failure. Furthermore, the log content is differentiated for different resource objects, so even if different resource objects trigger the same event, their log contents will not interfere with each other, further reducing the probability of incomplete logs due to the mistaken deletion of log content from other resource objects.

[0040] Compared to related technologies where log collectors deduplicate log content within a specified time interval, applying this time-interval-based deduplication method to container cluster management systems can lead to truncation of log content within the event processing cycle. The event processing cycle is designed to support and execute event processing logic. This embodiment employs a grouping mechanism based on resource objects and event processing logic. This allows the hash metadata of log content for the same event processing logic of the same resource object to be grouped into the same group. This means that a single log entry can accurately pinpoint an event processing logic instance for a specific resource object. Log deduplication is based on the event processing logic of the resource object, not just time. This helps avoid the truncation of log content within the event processing cycle caused by deduplication within a specified time interval, and helps maintain the temporal sequence and integrity of log content in the event processing logic. Furthermore, due to the temporal sequence and integrity of the event processing logic, the deduplicated logs not only reduce redundant logs and optimize resource usage, but also accurately pinpoint the log of a specific event processing logic instance for a single resource object, avoiding missing or redundant logs, thus facilitating rapid problem troubleshooting.

[0041] Compared to related technologies that use server-side data processing pipelines to collect log content from multiple sources and store only one copy of the log content with the same hash value, thus avoiding duplicate log content in the log file, the log processing method in this application removes duplicate log content stored in the log file. This results in higher log file integrity and facilitates troubleshooting. Furthermore, based on the grouping of each hash metadata, it is possible to accurately distinguish which resource object and which event processing logic each hash metadata in the hash file originates from, thereby accurately distinguishing log content from multiple sources and preventing mutual interference and the inability to locate the source.

[0042] In some embodiments, any hash metadata includes: ownership information of the corresponding log content, the ownership information including object identifier and execution identifier, the object identifier being used to identify the resource object, and the execution identifier being used to identify one execution of the event processing logic; the above step S102 of grouping the hash metadata in the hash file to obtain each hash metadata group may specifically include the following steps: grouping the hash metadata in the hash file using the object identifier to obtain hash metadata groups belonging to each resource object; for any hash metadata group of a resource object, grouping the hash metadata group of the resource object using the execution identifier to obtain hash metadata groups of each event processing logic belonging to the resource object.

[0043] For example, attribution information is used to indicate which resource object in the container cluster management system and in which controller loop's event handling logic the corresponding log content belongs to. The attribution information for log content generated by the same resource object in a single controller loop's event handling logic is consistent.

[0044] For example, the object identifier includes at least one of the following information items: the application programming interface group (Group) to which the resource object belongs, the version (Version) of the resource object, the type (Kind) of the resource object, the namespace (Namespace) to which the resource object belongs, and the name (Name) of the resource object. These information items can collectively define the complete identity of a resource object.

[0045] For example, the execution identifier includes at least a timestamp of the event processing logic being executed. In other examples, the execution identifier may also include a random number. In this example, the timestamp is used to record the precise point in time, and the random number is used to provide an additional unique identifier. The timestamp and the random number uniquely identify which event processing logic the corresponding log content belongs to. The combination of timestamps and random numbers can reduce conflicts between records generated by executing the same event processing logic at the same time, further improving the uniqueness of data identification.

[0046] It should be understood that the number of digits in the random number can be customized according to actual needs, such as a 3-digit random number, a 6-digit random number, etc. This application does not impose specific limitations on the embodiments.

[0047] In this embodiment, the attribution information includes object identifiers and execution identifiers. When grouping hash metadata, firstly, the object identifiers in the hash metadata are used for grouping to determine which resource object each piece of hash metadata corresponds to. Specifically, if the resource object identifiers are the same, the log content corresponding to the hash metadata belongs to the same resource object. Next, each piece of hash metadata corresponding to each resource object identifier is grouped according to its execution identifier, resulting in hash metadata groups. Each hash metadata group contains hash metadata for one event processing logic of a certain resource object. Specifically, if the execution identifiers are the same, the log content corresponding to the hash metadata belongs to the same event processing logic. Grouping the hash metadata in the hash file according to the object identifier and execution identifier provides a data foundation for subsequent deduplication processing of the log file based on each hash metadata group.

[0048] In some embodiments, any hash metadata may further include: content hash information for indicating the corresponding log content; performing a duplicate determination on each hash metadata group, and determining duplicate log content based on the determined duplicate hash metadata groups, including the following steps: obtaining hash metadata groups of each event processing logic belonging to the same resource object, to obtain each target group belonging to each resource object; for any resource object, determining the duplicate content hash information in each target group of the resource object; determining the duplicate log content based on the duplicate content hash information.

[0049] For example, for any resource object, each content hash information within any target group of the resource object corresponds to the same execution identifier, that is, to the same event processing logic. For different event processing logics of the same resource object, if the content hash information corresponding to any two adjacent event processing logics of the resource object is the same, it can be determined that the content hash information corresponding to the two event processing logics is duplicate content hash information.

[0050] In this embodiment, the group-based deduplication strategy can determine the hash information of duplicate content based on each hash metadata group, thereby identifying duplicate log content. Then, multiple log entries in the log file are deduplicated as a whole based on the duplicate log content, which helps maintain the temporal order and integrity of the log content in terms of event processing logic. Due to its temporal order and integrity in event processing logic, the deduplicated logs not only reduce redundant logs and optimize resource usage, but also accurately locate the log of a specific event processing logic for a single resource object, making it more suitable for the controller use case of a container cluster management system.

[0051] In some embodiments, for any resource object, determining the duplicate content hash information in each target group of the resource object includes: for any resource object, sequentially obtaining two hash metadata groups that are adjacent in execution time from the target group according to the order of execution time contained in the attribution information; if the content hash information in each row of the two hash metadata groups is the same, the two hash metadata groups are determined to be duplicate groups; if the number of hash metadata groups determined to be duplicate groups is greater than or equal to 2, the content hash information in the hash metadata groups other than the first group and the last group in the determined duplicate groups is taken as duplicate content hash information.

[0052] For example, the timestamp of the execution event processing logic included in the attribution information can be used to represent the execution time of the corresponding event processing logic. Assuming the target group includes three hash metadata groups in order of execution time, the hash information of each row of content in the first and second hash metadata groups can be compared first, then the hash information of each row of content in the second and third hash metadata groups can be compared again. If the hash information of each row of content in these three hash metadata groups is the same, then the hash information in the second hash metadata group can be considered as duplicate content hash information. In other words, the hash information in the first and third hash metadata groups is retained.

[0053] For example, the logic for determining the duplication of content hash information in two adjacent hash metadata groups may include: First, determining whether the number of rows in the two adjacent hash metadata groups is the same; if they are not the same, they are determined to be non-duplicate. If the number of rows in the two hash metadata groups is the same, then comparing the content hash information of each row is compared; if all rows are the same, the content hash information of the two groups is considered to be duplicated. If any row is inconsistent, the content hash information of the two groups is considered not to be duplicated. If duplicate groups appear more than twice, that is, the number of hash metadata groups determined to be duplicate groups is greater than 2, for example, assuming that in 10 hash metadata groups of a resource object, from the first hash metadata group to the 10th hash metadata group, the content hash information in each row of every two adjacent hash metadata groups is the same, then only the first hash metadata group and the last hash metadata group can be retained, and the content hash information in the second to ninth hash metadata groups in between can be considered as duplicate content hash information.

[0054] In this embodiment, for multiple duplicate hash metadata groups, the content hash information from the first and last hash metadata groups is retained. This provides a data foundation for subsequently retaining the log content corresponding to the content hash information from the first and last hash metadata groups. Retaining the log content corresponding to the content hash information from the first and last hash metadata groups ensures the relative integrity and order of the log content. Furthermore, during troubleshooting, the execution time of the event processing logic contained in the retained log content can provide the start and end times of the troubleshooting process. This facilitates quickly locating the specific moment the problem occurred, accelerating fault diagnosis and problem resolution, and improving the efficiency and accuracy of log analysis.

[0055] In some embodiments, the log processing method further includes the following steps: for any resource object, obtain the line number of the duplicate content hash information in the hash file to obtain the duplicate line number; and record the duplicate line number in memory.

[0056] For example, for hash metadata groups that are determined to be duplicate groups, the line numbers of the hash metadata in these hash metadata groups are recorded in memory. For instance, in the example above, if the content hash information in the 2nd to 9th hash metadata groups is duplicate content hash information, then the line numbers of the content hash information in the 2nd to 9th hash metadata groups can be recorded in memory.

[0057] In this embodiment, the line number of the repeated content hash information is recorded in memory instead of being recorded in the file, which can avoid frequent disk read and write operations, improve data processing speed, and save storage resources.

[0058] In some embodiments, the log content in the log file corresponds line by line to the hash metadata in the hash file; the step of determining the duplicate log content based on the hash information of the duplicate content may specifically include: obtaining the duplicate line number from memory; and taking the log content in the log file with the same line number as the duplicate line number as the duplicate log content.

[0059] As an example, based on the duplicate line numbers retrieved from memory, it can be determined which lines of hash metadata in the hash file are duplicated. Since the log content and hash metadata correspond line by line, this can be used to reverse map back to the log file to determine which log content is duplicated, that is, to determine the log content that needs to be deduplicated.

[0060] In this embodiment, log deduplication is performed based on the duplicate line numbers read from the log file. This allows for precise deduplication by line number, preventing erroneous deletion or omission of duplicate log content. Furthermore, since the duplicate line numbers are stored in memory rather than on disk, memory operations are significantly faster than disk operations, thus improving data retrieval speed and reducing processing time. Additionally, because the hash metadata corresponds to each line of log content, this mapping relationship allows for rapid deduplication without repeatedly scanning the entire log file, thereby improving deduplication efficiency.

[0061] In some embodiments, the above-described deduplication process for log files based on duplicate log content may specifically include the following steps: determining the non-duplicate log content in the log file based on the duplicate log content; generating a deduplicated log file based on the non-duplicate log content; deleting the log file containing duplicate log content; and deleting the hash file.

[0062] For example, the log content in the log file is traversed line by line according to the duplicate line number. If the line number of a certain line is included in the duplicate line number, it means that the log content of this line is duplicated, and the log content of this line can be skipped. Otherwise, it means that the log content of this line is not duplicated, and the log content of this line is written to a new deduplication log file until the log traversal is completed.

[0063] For ease of description, a log file containing duplicate log content can be called a redundant log file. For example, the filename of the deduplicated log file can be different from the filename of the redundant log file. After generating the deduplicated log file, the hash file corresponding to the redundant log file has also completed its task of identifying duplicate log content. Therefore, the redundant log file and its corresponding hash file can be deleted to save storage space.

[0064] In some application scenarios, container cluster management systems employ a log rotation mechanism. Log rotation is a management strategy for log files in container cluster management systems and an essential feature for production environments. For example, when the total storage capacity of log files across all controllers in a container cluster management system reaches a predetermined upper limit, the oldest log file (hereinafter referred to as the old log file) can be deleted. Alternatively, when the storage capacity of a controller's log files reaches a predetermined capacity threshold or the time elapsed since its creation reaches a predetermined duration threshold, the log file can be deleted, renamed, or compressed. This prevents log files from growing indefinitely and consuming excessive disk space, ensuring that new log content is effectively recorded and that old log files are managed and cleared in a continuous cycle. It should be understood that the total storage capacity limit, predetermined capacity threshold, and duration threshold can be customized according to actual needs, and this embodiment does not impose specific limitations.

[0065] In some application scenarios, when deleting log files according to the log rotation mechanism, if the deduplication operation of the log file is being performed synchronously, the log deduplication logic may fail due to the inability to correctly obtain the log file. This application provides an asynchronous deduplication method for log files. Deduplication is triggered when the controller's log file is greater than or equal to a predetermined capacity threshold. The deduplication process does not directly delete duplicate log content from the log file. Instead, without changing the log content contained in the log file, a deduplicated log file is generated based on the non-duplicate log content, and then the original log file and its corresponding hash file are deleted. This allows log deduplication to be achieved without affecting the main process of the controller (e.g., the process that drives resource objects to change from their current state to the desired state), and it also balances log rotation by deleting redundant log files and their corresponding hash files. As can be seen from the above embodiments, the log rotation implemented in this application does not cause the log deduplication logic to fail. Furthermore, before performing log deduplication, the controller can write the newly generated log content and corresponding hash metadata into a new log file and a new hash file, ensuring that the new log content can be effectively recorded.

[0066] In some embodiments, before obtaining the hash file corresponding to the log file, the following steps are further included: for any log content in the log file, obtaining the ownership information of the log content; performing a hash calculation on the log content to obtain content hash information; generating hash metadata of the log content based on the ownership information and the content hash information; and writing the log content and the corresponding hash metadata into the log file and the hash file, respectively.

[0067] For example, after generating log content, the controller generates hash metadata of the log content based on the obtained ownership information of the log content and the calculated content hash information of the log content, and then writes the log content to the log file and writes the hash metadata of the log content to the hash file corresponding to the log file.

[0068] In this embodiment, after generating log content, corresponding hash metadata is generated based on the log content. The hash metadata includes ownership information and content hash information, providing a data basis for subsequent grouping of hash metadata and determination of duplicate log content.

[0069] In some embodiments, the log content includes operation data, which records operations performed on resource objects. The content hash information includes first hash data corresponding to the operation data, and the first hash data has a predetermined length. In this embodiment, the step of performing hash calculation on the log content to obtain content hash information may specifically include: generating first hash data based on each character of the operation data and predetermined padding characters when the length of the operation data is less than the predetermined length; generating first hash data based on the operation data when the length of the operation data is equal to the predetermined length; and sampling characters in the operation data when the length of the operation data is greater than the predetermined length, generating first hash data based on the sampled characters, wherein the number of sampled characters is equal to the predetermined length, and operation data of the same length corresponds to the same character sampling position between different log contents.

[0070] For example, if the operation data in the log content includes uncertain element information, this uncertain element information needs to be removed first. Uncertain element information includes, but is not limited to, timestamps and random numbers. In log deduplication scenarios, generating first hash data based on the operation data after removing uncertain element information can ignore the influence of uncertain factors such as timestamps and random numbers on the first hash data, thereby more accurately identifying and removing duplicate content.

[0071] For example, when the length of the operation data is less than the predetermined length of the first hash data, generating the first hash data based on each character of the operation data and the predetermined padding characters can include several methods. One method is to directly concatenate each character of the operation data with the predetermined padding characters until the length of the concatenated characters equals the predetermined length, thus obtaining the first hash data. Another method is to calculate the hash value of the operation data or the hash value of each character in the operation data through hash operations, and then concatenate the calculated hash value with the predetermined padding characters until the length of the concatenated characters equals the predetermined length, thus obtaining the first hash data.

[0072] For example, when the length of the operation data is equal to the predetermined length of the first hash data, generating the first hash data based on the operation data can include several methods. One method is to directly use the operation data as the first hash data. Another method is to calculate the hash value of the operation data or the hash value of each character in the operation data through hash operations, and use the calculated hash value as the first hash data.

[0073] For example, when the length of the operation data is greater than the predetermined length of the first hash data, generating the first hash data based on the sampled characters can include several methods. One method is to directly arrange the sampled characters in a predetermined order to obtain the first hash data. The predetermined order can be the positional order of the characters in the operation data or the character sampling order. Another method is to calculate the hash value of the sampled character sequence or the hash value of each sampled character through hash operation, and use the calculated hash value as the first hash data.

[0074] In the first method of generating the first hash data, the first hash data contains the plaintext of each character of the operation data. This plaintext is directly readable, facilitating subsequent intuitive difference comparisons and simplifying processing steps. In the second method of generating the first hash data, for applications with high data sensitivity, comparison based on hash values ​​can reduce the risk of data leakage and is more conducive to maintaining data security. Users can choose the specific method for generating the first hash data according to their actual needs; this application embodiment does not impose specific limitations.

[0075] For example, the hash algorithm corresponding to the hash operation may include, but is not limited to, any of the following: Message-Digest Algorithm 5 (MD5) algorithm, Secure Hash Algorithm 1 (SHA-1). The specific algorithm can be selected according to actual needs, and this application embodiment does not impose specific limitations.

[0076] For example, the pre-defined filler characters can be any symbolic characters and / or special characters that do not represent a specific literal meaning, such as including but not limited to the asterisk "*", the hash symbol "#", etc., and this application embodiment does not make specific limitations.

[0077] In this embodiment, the log content includes at least operation data. Based on the length of the operation data, a method for generating the corresponding hash data is flexibly selected to provide a data foundation for the generation of hash metadata.

[0078] In some embodiments, the step of sampling characters in the operation data described above may specifically include: taking the remainder of the index value of each character in the operation data with respect to a predetermined length to obtain the remainder result of each character, where the index value is used to indicate the position of the corresponding character in the operation data; grouping the index values ​​of each character according to the remainder result of each character, where each index value in the same group corresponds to the same remainder result; determining the grouping order of each group according to a predetermined sorting of the remainder results; sequentially selecting an index value from each group according to the grouping order, wherein the position of the selected index value in the corresponding group corresponds to the grouping order of the corresponding group and a first value, where the first value is the maximum number of index values ​​contained in each group; and using the characters corresponding to the selected index values ​​as sampled characters.

[0079] For example, taking a 100-bit character set in the operation data and a predetermined length of 42 bits for the first hash data, the sampled character is equal to the predetermined length, therefore the sampled character also needs to be 42 bits. The index value of each character is from 1 to 100, and the index value of each character is modulo 42. Index values ​​with the same modulo result are placed in the same group.

[0080] Specifically, if the remainder is 1, the index values ​​in the corresponding group include 1, 43, and 85;

[0081] If the remainder is 2, the index values ​​in the corresponding group include: 2, 44, and 86;

[0082] ...;

[0083] If the remainder result is 41, then the index values ​​in the corresponding group include: 41 and 83;

[0084] If the remainder is 0, the index values ​​in the corresponding group include 42 and 84.

[0085] Assuming the predetermined order of the remainder results is 1, 2, ..., 41, 0, then the grouping order of each group will remain consistent with this predetermined order. Specifically, for remainder results 1 to 41, if the remainder result is i, then the index value in the corresponding group is the index value of the i-th group. i is an integer greater than or equal to 1 and less than or equal to 41. For remainder result 0, the index value in the corresponding group is the index value of the 42nd group.

[0086] As an example, if we denote the position of the selected index value in the corresponding group as k, the grouping order of the corresponding group as i, and the first value as m, the correspondence between the three can be, for example, k = (i-1) mod m + 1. Here, k and m are both integers greater than or equal to 1.

[0087] In this example, the maximum number of index values ​​contained in each group is 3, i.e., m = 3. The first index value is selected from the first group (k = 1 when i = 1 and m = 3), the second index value is selected from the second group (k = 2 when i = 2 and m = 3), the third index value is selected from the third group (k = 3 when i = 3 and m = 3), the first index value is selected from the fourth group (k = 1 when i = 4 and m = 3), and so on, until the third index value is selected from the 42nd group. If the number of index values ​​in any group is less than 3, for example, only 2, when the third index value of that group is needed, the first index value of that group can be used as a substitute, i.e., the last index value of the group is padded to the first index value of that group.

[0088] It should be understood that there may be other correspondences between the selected index value's position in the corresponding group, the grouping order of the corresponding group, and the first value. Specific settings can be customized according to actual needs, and this application embodiment does not impose specific limitations.

[0089] It should be noted that character sampling of the operation data can also be performed in other ways in this embodiment. For example, a fixed sampling interval can be predefined based on the length of the operation data and the number of characters to be sampled. For instance, if the characters in the operation data are 100 bits and the predetermined length of the first hash data in the hash metadata is 42 bits, meaning that 42 bits of characters are needed for sampling, then sampling can be set to occur every n characters, where n is an integer greater than or equal to 1 and less than or equal to 2. When sampling characters, characters can be extracted from the beginning of the data according to the defined sampling interval to obtain the sampled characters. Alternatively, there can be no fixed sampling interval; sampling can be performed directly based on experience at each predetermined character position in the operation data to obtain the sampled characters. The number of predetermined character positions is equal to the predetermined length. In other words, users can choose the specific method of character sampling according to their actual needs, as long as the number of sampled characters is equal to the predetermined length and the same length of operation data in different log contents corresponds to the same character sampling positions.

[0090] This embodiment provides a specific implementation method for character sampling of operational data. By taking the remainder of the index value of each character in the operational data with respect to a predetermined length, it is beneficial to ensure that the index values ​​of each character are evenly divided into multiple groups, with each group corresponding to a remainder result, facilitating a uniform distribution of index values ​​within each group. Furthermore, by selecting index values ​​that correspond to the grouping words and the first value, it is beneficial to ensure that the sampled index values ​​remain balanced across the groups, reducing the probability of oversampling or undersampling in individual groups. In practical applications, the modulus value selected for the remainder, the position of the selected index value in the corresponding group, the grouping order of the corresponding groups, and the specific value of the first value can be adjusted according to specific data characteristics and requirements.

[0091] In some embodiments, the step of performing hash calculation on the log content to obtain content hash information may further include at least one of the following: obtaining a predetermined number corresponding to the log level included in the log content, generating second hash data included in the content hash information based on the predetermined number, wherein the log level is used to characterize the type of log content; obtaining a predetermined first number of characters from a first predetermined position in the stack information included in the log content, generating third hash data included in the content hash information based on the predetermined first number of characters, wherein the stack information is used to locate the code position corresponding to the generation of the log content; obtaining a predetermined second number of characters from a second predetermined position in the call information included in the log content, generating fourth hash data included in the content hash information based on the predetermined second number of characters, wherein the call information is used to characterize the calling function corresponding to the generation of the log content.

[0092] For example, log levels include, but are not limited to, at least one of the following: INFO level, DEBUG level, WARN level, and ERROR level. INFO level logs can be used to record normal controller operation information and resource status information, such as controller startup and resource status changes. DEBUG level logs provide detailed debugging information. WARN level logs are used to record information about potential problems and abnormal situations, such as, but not limited to, warnings about approaching resource limits (e.g., memory, disk space). ERROR level logs are used to record information that affects the functionality or operation of the controller, such as, but not limited to, critical resources (e.g., database, message queue) being unavailable, data inconsistency, or data integrity issues.

[0093] As an example, in the hash metadata, the information level is pre-set to have 1 bit. For instance, the pre-defined number for the information level could be 1, the pre-defined number for the debug level could be 2, the pre-defined number for the warning level could be 3, and the pre-defined number for the error level could be 4. It should be understood that the pre-defined number for the information level is not limited to the specific number in the example; in actual application scenarios, users can customize the setting as needed, and this application embodiment does not impose specific limitations.

[0094] For example, in the hash metadata, the number of bits for the stack information is pre-set to a first number. The first predetermined position can be the end position of the most recent stack-up information in the stack information. A predetermined first number of characters are retrieved from the first predetermined position, for example, 10 characters can be retrieved from the end position of the most recent stack-up information.

[0095] For example, in the hash metadata, the number of characters for the call information is pre-set to a second number. The second predetermined position can be the tail position of the most recent call information in the call information. Retrieving a predetermined second number of characters from the second predetermined position could, for example, be taking 10 characters from the end position of the most recent call information.

[0096] In the above example, the number of bits for the information level, the stack information, and the call information in the hash metadata can be preset according to actual needs. The first predetermined position and the second predetermined position can also be, for example, a predetermined middle position of the corresponding information. Specifically, they can be preset according to actual needs. This application embodiment does not make specific limitations.

[0097] For example, the method of generating the second hash data based on a predetermined number, the method of generating the third hash data based on a predetermined first number of characters, and the method of generating the fourth hash data based on a predetermined second number of characters can refer to at least one of the various methods of generating the first hash data based on the operation data described above. Further details will not be elaborated here.

[0098] In this embodiment, the hash algorithm provided by the first method for generating hash data is more efficient, capable of quickly calculating the hash information of structured log content, thereby rapidly representing the log content, with efficiency far exceeding traditional hash calculation methods. In practical applications, it can be flexibly selected according to actual needs.

[0099] In this embodiment, the log content may also include at least one of log level, stack information and call information. Based on the above information contained in the log content, corresponding hash data is generated to provide a data foundation for the generation of hash metadata.

[0100] In some embodiments, the step of writing log content and corresponding hash metadata to a log file and a hash file respectively may specifically include: performing a first write operation on the hash file, the first write operation being used to write hash metadata into the hash file; and, if the first write operation is successful, performing a second write operation on the log file, the second write operation being used to write log content corresponding to the successfully written hash metadata into the log file.

[0101] For example, for any log entry and the corresponding hash metadata generated synchronously, the hash metadata corresponding to the log entry is first written to the hash file. If the writing is successful, the log entry is then written to the log file to ensure that the log entries and their corresponding hash metadata are written successfully one by one.

[0102] For example, to ensure that log content and hash metadata are not lost after a system restart or power outage, the log content can be written to a log file, the hash metadata to a hash file, and then the log file and hash file can be written to a persistent storage device, such as a hard drive, solid-state drive, or network storage device. This process of writing the file is also called writing to disk or persisting to disk.

[0103] In this embodiment, before a single log entry is written to disk, corresponding hash metadata is generated synchronously. To avoid information redundancy and accumulation caused by recording hash metadata information in the log file, the hash metadata needs to be saved separately to a hash file. Since the log content and hash metadata correspond one-to-one, such as corresponding to each entry or line, the log content and hash metadata need to be written successfully at the same time to achieve atomic writing of the log content and hash metadata to disk, ensuring the consistency and integrity of the information stored in the log file and hash file.

[0104] In some embodiments, the log processing method further includes: rolling back the first write operation if the first write operation fails; rolling back the first write operation and the second write operation if the second write operation fails; and, if the second write operation is terminated and restarted, determining the first line number of the log content in the log file and the second line number of the hash metadata in the hash file, and, if the first line number and the second line number are inconsistent, rolling back the hash metadata in the hash file according to the difference between the first line number and the second line number, and re-executing the hash metadata write operation after rolling back the hash metadata.

[0105] For example, for any log entry and its corresponding hash metadata, the hash metadata is written to the hash file first. If writing fails, the process returns and does not continue writing. After successfully writing the hash metadata to the hash file, the log entry is written to the log file. If writing fails, the latest write to the hash file is rolled back. If the log entry is successfully written to the log file, the process returns correctly. If the program terminates abnormally while writing the log entry to the log file, the program will check whether the number of lines in the log file and the hash file are consistent when it restarts. If they are inconsistent, and the number of lines in the hash file is greater than or equal to the number of lines in the log file, the excess lines in the hash file are rolled back, and the program first tries to write the hash metadata corresponding to the log entry to the hash file again, and then tries to write the log entry to the log file again.

[0106] In this embodiment, by prioritizing the writing of hash metadata and ensuring its successful persistence to disk before writing the log content, data consistency between the log file and the hash file can be guaranteed. After an abnormal program termination and restart, checking the line counts in the log file and hash file and performing necessary rollback operations can reduce the probability of data loss or inconsistency after a system restart, thereby improving system reliability.

[0107] In some embodiments, after the step of deduplicating the log file, the log processing method further includes: if the size of the deduplicated log file reaches a predetermined size threshold, performing file compression on the deduplicated log file to obtain a compressed deduplicated log file.

[0108] In this embodiment, the size of the deduplicated log file itself also has an upper limit. When the file size reaches the predetermined capacity threshold, log compression is performed to further reduce storage space.

[0109] According to the log processing method in the embodiments of this application, a log deduplication scheme is proposed in the controller mode of a container cluster management system. In this scheme, the log content represented by the hash metadata in the hash file corresponds to an event processing logic executed by the controller on a resource object. After grouping the hash metadata, the log content belonging to the same resource object and the same event processing logic is divided into one hash metadata group. This allows the corresponding log content to be distinguished according to resource objects and event processing logic, thereby isolating the logs for two types and different dimensions, ensuring that the log content does not interfere with each other and preventing erroneous deduplication. In addition, the group-based deduplication strategy can perform overall deduplication of multiple log contents in the log file, rather than deduplication based on a single log, which is more suitable for the controller use case of the container cluster management system. Furthermore, for the disk writing of log content and hash metadata, an atomic disk writing strategy based on log files and hash files is designed to achieve a cross-file atomic disk writing strategy, ensuring that the log content and hash metadata are written successfully or fail simultaneously, preventing log and hash mismatch, and ensuring data consistency between log files and hash files. Furthermore, after generating deduplicated log files based on non-duplicate log content, log files containing duplicate log content and their corresponding hash files can be deleted, which is beneficial for providing a controller-mode storage resource optimization solution for large-scale scenarios.

[0110] Figure 2 A detailed flowchart of a log processing method according to an exemplary embodiment of this application is shown. In some embodiments, the log processing method includes the following steps.

[0111] As shown in "S201, Generate Log Content", when the controller executes event handling logic on a resource object, each log entry corresponds to one event handling logic for a resource object.

[0112] As shown in "S202, Hash Generation", generate hash metadata corresponding to the log content.

[0113] In this step, the controller generates hash metadata corresponding to each log entry before it is written to disk. The hash metadata includes log ownership information and content hash information.

[0114] As an example, log attribution information is used to characterize which resource object and which controller's event handling logic generated the log entry. The attribution information for log entries generated by the same resource object within a single event handling logic of the controller is consistent. The log attribution information structure could be, for example, as shown in: {Application Programming Interface Group-Version-Type-Namespace-Name}_{Timestamp-6-digit random number}, where {Application Programming Interface Group-Version-Type-Namespace-Name} is the object identifier, used to uniquely identify a resource object in a container cluster management system, and {Timestamp-6-digit random number} is the execution identifier, used to uniquely identify which event handling logic the log entry belongs to.

[0115] As an example, content hash information is used to represent log content. Content hash information can include four parts: log level, stack information, call information, and operation data. The operation data, after removing uncertain elements, is, for example, 64 bits in total. The value logic of each bit is as follows:

[0116] The first digit represents the log level: 1 for informational level, 2 for debug level, 3 for warning level, and 4 for error level.

[0117] The 2nd to 11th positions represent stack information, taking the last 10 characters of the most recent stack push information;

[0118] The 12th to 21st characters represent the call information, taken from the last 10 characters of the most recent call information;

[0119] Bits 22-64: Represent the operation data, requiring 43 bits of character sampling. The strategy is as follows: If the operation data length is 43, the operation data is directly used as the corresponding hash data; if the operation data length is ≤43, the operation data is padded with asterisks (*) until it reaches 43 bits to obtain the corresponding hash data; if the operation data length is >43, the elements with the same remainder value are grouped according to the index value % 42. Assuming each group has 3 elements, the first element of the first group, the second element of the second group, the third element of the third group, the first element of the fourth group, the second element of the fifth group, and so on, are taken sequentially until 43 bits are reached; if the number of elements in each group is less than 3, the last element can be padded to become the first element in that group.

[0120] As shown in "S203, Atomicity Disk Writing", the log content and the corresponding hash metadata are written to the log file and hash file respectively.

[0121] In this step, before each log entry is written to disk, corresponding hash metadata is generated synchronously. The log content and the hash metadata are in a one-to-one correspondence, for example... Figure 2 The logs 1-4 shown represent 4 log entries, and hashes 1-4 are hash metadata that correspond one-to-one with logs 1-4.

[0122] For example, log content and hash metadata correspond one by one or line by line. Log content is written to log file, hash metadata is written to hash file, and the writing of log content and hash metadata is guaranteed to be successful. This achieves atomic disk write of log content and hash metadata, ensuring the consistency and integrity of the information stored in log file and hash file.

[0123] In some scenarios, as the event processing logic of the controller continues to execute, the corresponding log files and hash files will accumulate until the log files reach a predetermined capacity threshold. The newly generated log content and the corresponding hash metadata will be written to the new log files and the new hash files, while the current log files and the corresponding hash files will perform the following asynchronous deduplication logic.

[0124] As shown in "S204, Object Grouping", hash metadata is grouped according to object identifier.

[0125] In this step, the hash metadata in the hash file is grouped according to the object identifier to obtain each object group. The log content corresponding to the hash metadata in the same object group belongs to the same resource object.

[0126] exist Figure 2 The diagram shows two object groups: hashes 1-1 to hashes 1-3 represent the content hash information of each hash metadata in one object group, and hashes 2-1 to hashes 2-3 represent the content hash information of each hash metadata in the other object group. The object identifiers of the hash metadata differ between the two object groups.

[0127] As shown in "S205, Event Grouping", events are grouped according to the event handling logic.

[0128] In this step, each object group is grouped according to the execution identifier to obtain event groups. Each event group contains hash metadata of an event processing logic for a specific resource object.

[0129] exist Figure 2 In this process, it is determined which event processing logic corresponds to the log content data for each hash metadata entry. Entities with the same execution identifier belong to the same event processing logic. Taking the first object group as an example, the hash metadata within that object group can be grouped according to the execution identifier to obtain different event groups. Figure 2Although the hash content contained in the first and second event groups shown can both be represented as "hash1-1 and hash1-2", the execution identifier of each hash metadata in the first event group is different from the execution identifier of each hash metadata in the second event group.

[0130] As shown in "S206, Repeated Calculation", repeatability is checked for event groups in each object group.

[0131] In this step, several event groups within each object group are arranged chronologically. A duplicate check is performed on any two adjacent event groups within the same object group. The check determines whether the content hash information in the two adjacent event groups is completely identical. If they are identical, it indicates a duplicate; otherwise, it is a non-duplicate. The hash metadata associated with duplicate content hash information can be deleted, while the hash metadata associated with non-duplicate content hash information can be retained.

[0132] The logic for determining duplicates is as follows: For two adjacent event groups obtained sequentially, determine whether the number of rows in the two groups is the same. If they are not the same, they are considered non-duplicates. If the number of rows in the two groups is the same, compare the content hash information of each row. If each row is the same, they are considered duplicates. If any row is different, they are considered non-duplicates. If a duplicate group appears more than twice, only the first and last groups are retained, and the row number of the intermediate groups is recorded as a duplicate row, and the row number of the duplicate row is recorded in memory.

[0133] As shown in "S207, Log Deduplication", the log files are deduplicated based on each hash metadata group.

[0134] In this step, after the above steps, it can be determined which lines in the hash file have duplicate hash metadata. Since the log content and hash metadata correspond line by line, this can be used to reverse map back to the log file to determine the log content that needs to be deduplicated, thereby performing log deduplication. The detailed steps are as follows: obtain the line number of the line that needs to be duplicated from memory; traverse the log file line by line. If the line number of the log content is included in the line number of the duplicate line, it means that the log content of the line is duplicated, so skip it; otherwise, it means that the log content of the line is not duplicated, and write the non-duplicate log content into the deduplication log file until the log traversal is completed. At this time, the log content contained in the deduplicated log file is the deduplicated log content.

[0135] As shown in "S208, Log Compression", file compression is performed on the deduplicated log file.

[0136] In this step, if the size of the deduplicated log file reaches a predetermined capacity threshold, file compression is performed on the deduplicated log file to obtain a compressed deduplicated log file.

[0137] Through the above steps S201-S208, a complete log deduplication process is completed in the container cluster management system. Subsequent new log files will continue to undergo the same deduplication process, thereby greatly reducing the size of the logs.

[0138] Figure 3 A flowchart illustrating a controller log processing method of a container cluster management system according to an embodiment of this application is shown. Figure 3 As shown, in some embodiments, the method includes the following steps.

[0139] S301, in response to a resource configuration change request from at least one resource object in the container cluster management system, executes the event handling logic corresponding to the resource configuration change request, and generates corresponding log content based on the execution of the event handling logic.

[0140] S302, Generate hash metadata corresponding to each log content. The hash metadata is used to represent the log content generated when the controller executes an event processing logic on a resource object.

[0141] S303, write each log entry and its corresponding hash metadata to the log file and hash file respectively.

[0142] S304, the log file and hash file are processed using the log processing method described in the above embodiment to obtain a deduplicated log file.

[0143] In this embodiment, a resource configuration change request is a request to change resource configuration. The controller can respond to a resource configuration change request from any resource object in the container cluster management system, execute the event handling logic corresponding to the request, generate corresponding log content and hash metadata, write the log content to a log file, and write the hash metadata to a hash file. Based on the log processing method described in the previous embodiment, log deduplication is performed using the log file and hash file to obtain a deduplicated log file. According to this method, a log deduplication scheme is proposed in the controller mode of the container cluster management system, performing overall deduplication of multiple log entries in the log file, significantly reducing the size of the log files generated by the controller in the system.

[0144] In some embodiments, before the step of processing log files and hash files in step S304, the method further includes: obtaining the controller's configuration file from the container cluster management system, the configuration file including the controller's deduplication switch parameter, the parameter value of the deduplication switch parameter being used to indicate the on / off state of the log deduplication function; and determining that the controller has enabled the log deduplication function when the switch state is on.

[0145] For example, the deduplication function of the controller can be controlled by a deduplication switch parameter in the configuration file. For instance, if the value of the deduplication switch parameter is 0 or OFF, it means that the log deduplication function is off; if the value of the deduplication switch parameter is 1 or ON, it means that the deduplication function is on. The parameter value can be customized by the user, and this embodiment does not impose specific limitations.

[0146] In this embodiment, by using the deduplication switch parameter in the configuration file, users can flexibly enable or disable the log deduplication function according to actual needs, so that the log management strategy can better adapt to different operating environments and requirements.

[0147] For the specific implementation methods of each step in the embodiments of this application, please refer to the above-mentioned combination. Figure 1 The corresponding descriptions in the log processing methods described, and their corresponding beneficial effects, will not be elaborated further here.

[0148] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide a log processing device.

[0149] Figure 4 This is a schematic diagram of a log processing device provided in an embodiment of this application. Figure 4 As shown, the log processing device includes the following modules.

[0150] The acquisition module 410 is used to acquire the hash file corresponding to the log file when the controller's log file is greater than or equal to a predetermined capacity threshold; any hash metadata in the hash file is used to characterize the log content generated by the controller executing an event processing logic triggered by an event of a resource object.

[0151] Grouping module 420 is used to group the hash metadata in the hash file to obtain each hash metadata group. The log content corresponding to the hash metadata in the same group belongs to the same event processing logic of the same resource object.

[0152] The determination module 430 performs a duplicate determination on each hash metadata group and determines the duplicate log content based on the determined duplicate hash metadata groups.

[0153] The deduplication module 440 is used to deduplicat log files based on duplicate log content.

[0154] In some embodiments, any hash metadata includes: ownership information of the corresponding log content, the ownership information including object identifier and execution identifier, the object identifier being used to identify the resource object, and the execution identifier being used to identify one execution of the event processing logic; the grouping module 420 is specifically used to include: grouping the hash metadata in the hash file using the object identifier to obtain hash metadata groups belonging to each resource object; for any hash metadata group of a resource object, grouping the hash metadata group of the resource object using the execution identifier to obtain hash metadata groups of each event processing logic belonging to the resource object.

[0155] In some embodiments, any hash metadata may further include content hash information for indicating the corresponding log content; the determination module 430 is specifically configured to: obtain hash metadata groups of each event processing logic belonging to the same resource object, and obtain each target group belonging to each resource object; for any resource object, determine the duplicate content hash information in each target group of the resource object; and determine the duplicate log content based on the duplicate content hash information.

[0156] In some embodiments, when determining duplicate content hash information in each target group of a resource object for any resource object, the determination module 430 is specifically used to: for any resource object, sequentially obtain two hash metadata groups that are adjacent in execution time from the target group according to the order of execution time contained in the attribution information; if the content hash information in each row of the two hash metadata groups is the same, determine the two hash metadata groups as duplicate groups; if the number of hash metadata groups determined to be duplicate groups is greater than or equal to 2, take the content hash information in the hash metadata groups other than the first group and the last group in the determined duplicate groups as duplicate content hash information.

[0157] In some embodiments, the log processing apparatus further includes: a recording module, configured to, for any resource object, obtain the line number of the duplicate content hash information in the hash file to obtain the duplicate line number; and record the duplicate line number in memory.

[0158] In some embodiments, the log content in the log file corresponds line by line to the hash metadata in the hash file; when the deduplication module 440 determines the duplicate log content based on the hash information of the duplicate content, it is specifically used to: obtain the duplicate line number from memory; and take the log content in the log file whose line number is the same as the duplicate line number as the duplicate log content.

[0159] In some embodiments, when the deduplication module 440 is used to deduplicatize log files based on duplicate log content, it is specifically used to: determine the non-duplicate log content in the log file based on the duplicate log content; generate a deduplicated log file based on the non-duplicate log content; delete the log file containing duplicate log content; and delete the hash file.

[0160] In some embodiments, the log processing apparatus further includes: a file writing module, configured to, before obtaining the hash file corresponding to the log file, obtain the ownership information of any log content in the log file; perform hash calculation on the log content to obtain content hash information; generate hash metadata of the log content based on the ownership information and the content hash information; and write the log content and the corresponding hash metadata into the log file and the hash file respectively.

[0161] In some embodiments, the log content includes operation data, which is used to record operations performed on resource objects. The content hash information includes first hash data corresponding to the operation data, and the first hash data has a predetermined length. When the file writing module performs hash calculation on the log content to obtain the content hash information, it specifically performs the following steps: if the length of the operation data is less than the predetermined length, it generates first hash data based on each character of the operation data and a predetermined padding character; if the length of the operation data is equal to the predetermined length, it generates first hash data based on the operation data; if the length of the operation data is greater than the predetermined length, it samples the operation data and generates first hash data based on the sampled characters, wherein the number of sampled characters is equal to the predetermined length, and operation data of the same length corresponds to the same character sampling position between different log contents.

[0162] In some embodiments, when the file writing module is used to sample characters in the operation data, it specifically performs the following steps: taking the remainder of the index value of each character in the operation data with respect to a predetermined length to obtain the remainder result of each character, where the index value is used to indicate the position of the corresponding character in the operation data; grouping the index values ​​of each character according to the remainder result of each character, where each index value in the same group corresponds to the same remainder result; determining the grouping order of each group according to a predetermined sorting of the remainder results; sequentially selecting an index value from each group according to the grouping order, wherein the position of the selected index value in the corresponding group corresponds to the grouping order of the corresponding group and a first value, where the first value is the maximum number of index values ​​contained in each group; and using the characters corresponding to the selected index values ​​as sampled characters.

[0163] In some embodiments, when the file writing module performs hash calculation on the log content to obtain content hash information, it specifically performs at least one of the following: obtaining a predetermined number corresponding to the log level included in the log content, generating second hash data included in the content hash information based on the predetermined number, wherein the log level is used to characterize the type of the log content; obtaining a predetermined first number of characters from a first predetermined position in the stack information included in the log content, generating third hash data included in the content hash information based on the predetermined first number of characters, wherein the stack information is used to locate the code position corresponding to the generation of the log content; obtaining a predetermined second number of characters from a second predetermined position in the call information included in the log content, generating fourth hash data included in the content hash information based on the predetermined second number of characters, wherein the call information is used to characterize the calling function corresponding to the generation of the log content.

[0164] In some embodiments, when the file writing module is used to write log content and corresponding hash metadata to a log file and a hash file respectively, it is specifically used to: perform a first write operation on the hash file, the first write operation being used to write hash metadata in the hash file; and if the first write operation is successful, perform a second write operation on the log file, the second write operation being used to write log content corresponding to the successfully written hash metadata in the log file.

[0165] In some embodiments, the file writing module is further configured to: roll back the first write operation if the first write operation fails; roll back the first write operation and the second write operation if the second write operation fails; and, if the second write operation is terminated and restarted, determine the first line number of the log content in the log file and the second line number of the hash metadata in the hash file, and if the first line number and the second line number are inconsistent, roll back the hash metadata in the hash file according to the difference between the first line number and the second line number, and re-execute the hash metadata writing operation after rolling back the hash metadata.

[0166] In some embodiments, the log processing module further includes a compression module, which is used to perform file compression on the deduplicated log file after deduplication processing, when the size of the deduplicated log file reaches a predetermined capacity threshold, to obtain a compressed deduplicated log file.

[0167] The functions of each module in each device in the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0168] Figure 5 This is a schematic diagram of the structure of a controller log processing device for a container cluster management system provided in an embodiment of this application. Figure 5As shown, the log processing device includes the following modules.

[0169] The log generation module 510 is used to respond to resource configuration change requests of at least one resource object in the container cluster management system, execute event processing logic corresponding to the resource configuration change request, and generate corresponding log content based on the execution of the event processing logic.

[0170] The hash generation module 520 is used to generate hash metadata corresponding to each log content. The hash metadata is used to represent the log content generated when the controller executes an event processing logic on a resource object.

[0171] The file writing module 530 is used to write the contents of each log and the corresponding hash metadata to the log file and the hash file respectively.

[0172] Deduplication module 540, used for the above combination Figure 1 The described log processing method processes log files and hash files to obtain deduplicated log files.

[0173] In some embodiments, the log processing apparatus further includes: a determination module, configured to obtain a controller configuration file from the container cluster management system before processing log files and hash files, the configuration file including a deduplication switch parameter of the controller, the parameter value of the deduplication switch parameter being used to indicate the on / off state of the log deduplication function; and if the switch state is on, determine that the controller has enabled the log deduplication function.

[0174] The functions of each module in each device in the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0175] Figure 6 This is a block diagram of an electronic device used to implement embodiments of this application. Figure 6 As shown, the electronic device includes a memory 601 and a processor 602. The memory 601 stores a computer program that can run on the processor 602. When the processor 602 executes the computer program, it implements the method described in the above embodiments. The number of memories 601 and processors 602 can be one or more. In a specific implementation, the electronic device may also include a communication interface 603 for communicating with external devices and exchanging data.

[0176] In practical implementation, if the memory 601, processor 602, and communication interface 603 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0177] Optionally, in a specific implementation, if the memory 601, processor 602 and communication interface 603 are integrated on a single chip, the memory 601, processor 602 and communication interface 603 can communicate with each other through an internal interface.

[0178] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0179] This application provides a computer program product, including a computer program that, when executed by a processor, implements the method provided in this application.

[0180] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0181] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0182] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0183] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0184] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0185] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0186] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0187] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0188] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0189] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0190] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0191] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A log processing method, characterized in that, The method includes: If the controller's log file is greater than or equal to a predetermined capacity threshold, obtain the hash file corresponding to the log file; any hash metadata in the hash file is used to characterize the log content generated by the controller executing an event processing logic triggered by an event of a resource object. The hash metadata in the hash file is grouped to obtain each hash metadata group. The log content corresponding to the hash metadata in the same group belongs to the same event processing logic of the same resource object. The duplicate log content is determined based on the identified duplicate hash metadata groups. The log files are deduplicated based on duplicate log content.

2. The method according to claim 1, characterized in that, The hash metadata includes: the ownership information of the corresponding log content, the ownership information including object identifier and execution identifier, the object identifier is used to identify the resource object, and the execution identifier is used to identify one execution of the event processing logic; The process of grouping the hash metadata in the hash file to obtain each hash metadata group includes: The hash metadata in the hash file is grouped using object identifiers to obtain hash metadata groups belonging to each resource object; For any resource object's hash metadata group, the hash metadata group of the resource object is grouped using the execution identifier to obtain hash metadata groups belonging to each event processing logic of the resource object.

3. The method according to claim 2, characterized in that, Each hash metadata element also includes content hash information indicating the corresponding log content; the process of determining duplicate log content based on the determined duplicate hash metadata elements includes: Obtain the hash metadata groups of each event processing logic belonging to the same resource object to obtain each target group belonging to each resource object; For any resource object, determine the hash information of the duplicate content in each target group of the resource object; The duplicate log content is determined based on the hash information of the duplicate content.

4. The method according to claim 3, characterized in that, For any resource object, determining the hash information of duplicate content in each target group of the resource object includes: For any resource object, according to the order of execution time contained in the attribution information, two hash metadata groups that are adjacent in execution time are sequentially obtained from the target group; If the hash information of each row in the two hash metadata groups is the same, the two hash metadata groups are determined to be duplicate groups. If the number of hash metadata packets identified as duplicate groups is greater than or equal to 2, the content hash information of the hash metadata packets in the identified duplicate groups other than the first and last groups shall be used as the duplicate content hash information.

5. The method according to claim 3, characterized in that, The method further includes: For any resource object, obtain the line number of the duplicate content hash information in the hash file to get the duplicate line number; The duplicate row number is recorded in memory.

6. The method according to claim 5, characterized in that, The log content in the log file corresponds line-by-line to the hash metadata in the hash file; determining the duplicate log content based on the hash information of the duplicate content includes: Retrieve the duplicate row number from memory; Log entries with the same line number as the duplicate line number in the log file are considered duplicate log entries.

7. The method according to claim 1, characterized in that, The step of deduplicating the log file based on duplicate log content includes: Based on the repeated log content, determine the non-repeating log content in the log file; Generate a deduplicated log file based on the non-duplicate log content; Delete the log file containing the duplicate log content, and delete the hash file.

8. The method according to claim 1, characterized in that, Before obtaining the hash file corresponding to the log file, the method further includes: For any log content in the log file, obtain the ownership information of the log content; The log content is hashed to obtain the content hash information; Based on the attribution information and the content hash information, generate hash metadata for the log content; The log content and the corresponding hash metadata are written to the log file and the hash file, respectively.

9. The method according to claim 8, characterized in that, The log content includes operation data, which records operations performed on the resource object. The content hash information includes first hash data corresponding to the operation data, and the first hash data has a predetermined length. The step of performing hash calculation on the log content to obtain content hash information includes: If the length of the operation data is less than the predetermined length, the first hash data is generated based on each character of the operation data and the predetermined padding character. If the length of the operation data is equal to the predetermined length, the first hash data is generated based on the operation data; If the length of the operation data is greater than the predetermined length, the operation data is sampled for characters, and the first hash data is generated based on the sampled characters. The number of sampled characters is equal to the predetermined length, and operation data of the same length corresponds to the same character sampling position between different log contents.

10. The method according to claim 9, characterized in that, The character sampling of the operation data includes: The index value of each character in the operation data is moduloed by the predetermined length to obtain the modulo result of each character. The index value is used to indicate the position of the corresponding character in the operation data. The index values ​​of each character are grouped according to the remainder result of each character, and each index value in the same group corresponds to the same remainder result; The grouping order of each group is determined according to the predetermined sorting of the remainder results; An index value is selected sequentially from each group according to the grouping order, wherein the position of the selected index value in the corresponding group corresponds to the grouping order of the corresponding group and a first value, and the first value is the maximum number of index values ​​contained in each group. The characters corresponding to each selected index value are used as sampling characters.

11. The method according to claim 9, characterized in that, The step of performing a hash calculation on the log content to obtain content hash information also includes at least one of the following: Obtain a predetermined number corresponding to the log level included in the log content, and generate second hash data included in the content hash information based on the predetermined number, wherein the log level is used to characterize the type of the log content; From the first predetermined position in the stack information included in the log content, a predetermined first number of characters are obtained, and based on the predetermined first number of characters, third hash data included in the content hash information is generated. The stack information is used to locate the code position corresponding to the generation of the log content. From the second predetermined position in the call information included in the log content, a predetermined second number of characters are obtained, and based on the predetermined second number of characters, fourth hash data included in the content hash information is generated. The call information is used to characterize the call function corresponding to the generation of the log content.

12. The method according to claim 8, characterized in that, The step of writing the log content and the corresponding hash metadata into the log file and the hash file respectively includes: Perform a first write operation on the hash file, wherein the first write operation is used to write the hash metadata into the hash file; If the first write operation is successful, a second write operation is performed on the log file. The second write operation is used to write log content corresponding to the successfully written hash metadata into the log file.

13. The method according to claim 12, characterized in that, The method further includes: If the first write operation fails, the first write operation will be rolled back. If the second write operation fails, roll back both the first write operation and the second write operation. If the second write operation is terminated and restarted, the first line number of the log content in the log file and the second line number of the hash metadata in the hash file are determined. If the first line number and the second line number are inconsistent, the hash metadata in the hash file is rolled back according to the difference between the first line number and the second line number, and the write operation of the hash metadata is re-executed after the hash metadata is rolled back.

14. The method according to claim 1, characterized in that, After deduplication of the log file, the method further includes: If the size of the deduplicated log file reaches a predetermined capacity threshold, file compression is performed on the deduplicated log file to obtain a compressed deduplicated log file.

15. A method for processing controller logs in a container cluster management system, characterized in that, The method includes: In response to a resource configuration change request from at least one resource object in the container cluster management system, the system executes the event handling logic corresponding to the resource configuration change request and generates corresponding log content based on the execution of the event handling logic. Generate hash metadata corresponding to each log content, wherein the hash metadata is used to characterize the log content generated when the controller executes the event processing logic on the resource object once; Write the contents of each log file and the corresponding hash metadata to the log file and hash file respectively; The log file and the hash file are processed using the method described in any one of claims 1 to 14 to obtain a deduplicated log file.

16. The method according to claim 15, characterized in that, Before processing the log file and the hash file, the process further includes: Obtain the controller's configuration file from the container cluster management system. The configuration file includes the controller's deduplication switch parameter. The parameter value of the deduplication switch parameter is used to indicate the on / off state of the log deduplication function. When the switch is in the ON state, it is determined that the controller has enabled the log deduplication function.

17. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1 to 14 or any one of claims 15 to 16.

18. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of any one of claims 1 to 14 or any one of claims 15 to 16.

19. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 14 or any one of claims 15 to 16.

Images