Remote log processing methods, apparatus, devices, and storage media for applications

Abstract

This disclosure provides a method, apparatus, device, and storage medium for remote log processing of an application, relating to the field of distributed storage technology. The method includes: obtaining a log processing request from an application; responding to the data processing request using a data cache in a log file system when the log processing request is a data processing request; responding to the metadata processing request using a metadata cache in a log file system when the log processing request is a metadata processing request; and, if the log data in the data cache changes, or if the metadata in the metadata cache changes, flushing the changed log data or changed metadata to a remote distributed file system via an asynchronous worker thread.

Description

Technical Field

[0001] This disclosure relates to the field of computer storage technology, and more particularly to the field of distributed storage and cloud storage technology. Specifically, it relates to a method, apparatus, device, and storage medium for remote log processing of an application. Background Technology

[0002] Application logs are crucial and important, allowing us to understand and monitor the current state of the system and the events that have occurred. There is considerable research and practical work in the field of log processing. A few frameworks and industrial practices support direct logging to message queues or backend storage. However, achieving a highly available, low-latency, and low-cost remote log service remains a significant challenge. Summary of the Invention

[0003] This disclosure provides a method, apparatus, device, and storage medium for remote log processing of an application.

[0004] According to one aspect of this disclosure, a method for remote log processing of an application is provided, comprising:

[0005] Obtain the application's log processing request;

[0006] When the log processing request is a data processing request, the data processing request is responded to using the data cache in the log file system;

[0007] When the log processing request is a metadata processing request, the metadata processing request is responded to using the metadata cache in the log file system;

[0008] If the log data in the data cache changes, or if the metadata in the metadata cache changes, the changed log data or changed metadata is flushed to a remote distributed file system via an asynchronous worker thread.

[0009] According to one aspect of this disclosure, a remote log processing apparatus for an application is provided, comprising:

[0010] The request retrieval module is used to retrieve the application's log processing requests;

[0011] The data processing module is configured to respond to the data processing request using the data cache in the log file system when the log processing request is a data processing request;

[0012] The metadata processing module is used to respond to the metadata processing request using the metadata cache in the log file system when the log processing request is a metadata processing request.

[0013] The asynchronous change module is used to flush the changed log data or changed metadata into a remote distributed file system via an asynchronous worker thread when the log data in the data cache changes, or when the metadata in the metadata cache changes.

[0014] According to another aspect of this disclosure, an electronic device is provided, the electronic device comprising:

[0015] At least one processor; and

[0016] A memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods provided in any embodiment of this disclosure.

[0018] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods provided in any embodiment of this disclosure.

[0019] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the methods provided according to any embodiment of this disclosure.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0021] Figure 1a This is an architecture diagram of a remote logging system provided according to an embodiment of the present disclosure;

[0022] Figure 1b This is a flowchart of a remote log processing method for an application provided according to an embodiment of the present disclosure;

[0023] Figure 1c This is a schematic diagram of asynchronous log processing according to an embodiment of the present disclosure;

[0024] Figure 2a This is a flowchart of another remote log processing method for an application provided according to an embodiment of the present disclosure;

[0025] Figure 2b This is a schematic diagram of another asynchronous log processing method provided according to an embodiment of this disclosure;

[0026] Figure 2c This is a schematic diagram illustrating a renaming operation according to an embodiment of the present disclosure;

[0027] Figure 2d This is a schematic diagram illustrating another renaming operation provided according to an embodiment of this disclosure;

[0028] Figure 3a This is a flowchart of another remote log processing method for an application provided according to an embodiment of the present disclosure;

[0029] Figure 3b This is a schematic diagram of dynamic traffic scheduling provided according to an embodiment of the present disclosure;

[0030] Figure 3c This is a schematic diagram of hybrid compression of log data according to an embodiment of the present disclosure;

[0031] Figure 4 This is a schematic diagram of the structure of a remote log processing device for an application provided according to an embodiment of the present disclosure;

[0032] Figure 5 This is a block diagram of an electronic device used to implement the remote log processing method of the application in the embodiments of this disclosure. Detailed Implementation

[0033] The architecture of a remote logging system is as follows: Figure 1a As shown, the remote log service mainly includes: remote log control plane, remote log data plane, and underlying cloud storage DFS.

[0034] Log control plane: Primarily comprises the LogMaster and LogDaemon modules. LogMaster is a centralized service module, mainly responsible for control flow processing and user remote log management needs, including log volume lifecycle management. A log volume is mapped to a file directory in DFS. LogDaemon is a standalone module, primarily responsible for the lifecycle management of the data plane module LogFS, and supports fault-tolerant persistence and hot upgrades of LogFS. It is deployed on compute nodes as a DaemonSet.

[0035] The log data plane primarily consists of LogFS, which handles application log read and write requests and asynchronously synchronizes them to DFS. LogFS is a log Fuse module developed based on the user-space Fuse framework. Each Pod (container) is allocated a LogFS process (assigned by LogDaemon) upon creation, and the corresponding log volume for the Pod is integrated into the Pod using a Mount method for application read and write log files. The advantage of the Fuse mounting method is that it requires no modification to the client application.

[0036] Cloud Storage DFS: A high-throughput, low-cost distributed file system that supports HDFS (Hadoop Distributed File System) semantics and append-only, random, and sequential read modes. It is well-suited for customer log read / write scenarios. DFS employs a compute-storage separation architecture, primarily consisting of: Proxy compute nodes, ExtendServer log data storage modules, Master control nodes, and MetaDB underlying metadata storage modules.

[0037] Regarding data storage costs and isolation, in scenarios with millions of tenants, considering the overall storage costs and data isolation and security issues between tenants, we place different tenant logs in different file system directories. This ensures data isolation between multiple tenants and reduces storage costs to some extent by sharing a storage cluster.

[0038] Figure 1b This is a flowchart of a remote log processing method for an application according to an embodiment of this disclosure. The method is applicable to situations where user application logs are printed to cloud storage. The method can be executed by a remote log processing device for an application, which can be implemented in software and / or hardware and integrated into an electronic device. As shown in Figure 1, the remote log processing method for an application in this embodiment may include:

[0039] S101, Obtain the application's log processing request;

[0040] S102, if the log processing request is a data processing request, the data processing request is responded to using the data cache in the log file system;

[0041] S103, if the log processing request is a metadata processing request, the metadata processing request is responded to using the metadata cache in the log file system;

[0042] S104, if the log data in the data cache changes, or if the metadata in the metadata cache changes, the changed log data or changed metadata is flushed to the remote distributed file system via an asynchronous worker thread.

[0043] Applications perform numerous real-time read and write operations on logs. The success rate and latency of these operations directly impact application stability and critical functions such as monitoring and alerting. DFS, as a low-cost remote storage solution, evolved from an offline file system, and its performance and reliability cannot directly meet the real-time read and write requirements of online applications. Furthermore, to further optimize storage costs, all tenants of the log service share a DFS cluster within a single region. Tenants are the owners of the applications, and one tenant can correspond to users on different clients. Taking the storage of online logs for applications A and B in the instant messaging domain as an example, both application A and application B are tenants, and both applications A and B can have multiple users. Storing online logs on performance-constrained offline file storage via an unstable network, sharing the limited performance of a single cluster, poses a significant challenge to service reliability. We conducted an in-depth analysis of the log read and write behavior of online applications and introduced an asynchronous read and write mechanism into LogFS to completely decouple the most important and frequent log processing requests from the backend, eliminating the impact of network latency fluctuations on users.

[0044] like Figure 1c As shown, LogFS includes a data buffer, a local metadata buffer, and an asynchronous worker. The data buffer is used to cache the data written by the user at this mount point within a minute, and the local metadata buffer is used to cache the metadata of this mount point.

[0045] When log processing requests are data processing requests, they can be either write log requests or read log requests. Write log requests are the most important log operations, directly affecting the stability of tenant processes, and can be written directly to the data cache. In this case, if the log data in the data cache changes, asynchronous worker threads periodically flush the data to the backend distributed file system. The asynchronous worker threads can also clear log caches older than a few minutes. For read log requests, the system prioritizes querying the data to be read from the data cache, i.e., tail reads. 95% of user reads are tail reads, used for real-time log analysis and monitoring. If the data cache is insufficient, a catch-up read is used to access the backend DFS to read data. A catch-up read means reading from a position other than the tail of the log, and is generally triggered by a user's temporary query.

[0046] When the log processing request is a metadata processing request, it can be a metadata operation or a metadata read request. For metadata operations, the metadata cache can be modified directly, and the changed metadata can be flushed to the remote distributed file system through an asynchronous worker thread. For metadata read requests, the metadata cache can be used directly to respond.

[0047] By using a local data cache to respond to data processing requests and a metadata cache to respond to metadata processing requests, most user operations are responded to directly after taking effect locally. That is, the local response is followed by asynchronous flushing to the backend, which is not affected by network latency and decouples important log operations from DFS. Moreover, the separation of the data cache and the metadata cache can further improve response efficiency compared to the mixed caching of log data and metadata.

[0048] It should be noted that because log data and metadata operations are first cached locally and then asynchronously flushed to the backend DFS, there is a risk of data loss in extreme cases, such as compute node failure. To address this issue, we offer a synchronous write mode for users with higher data integrity requirements, which can prevent data loss.

[0049] The technical solution provided in this disclosure sets up a data cache, a metadata cache, and an asynchronous worker thread in LogFS, using a local data cache to respond to data processing requests and a metadata cache to respond to metadata processing requests. This allows most user operations to be responded to locally first and then asynchronously flushed to the backend, unaffected by network latency, thus decoupling important log operations from DFS. Moreover, the decoupling of the data cache and the metadata cache can further improve response efficiency compared to mixed caching of log data and metadata.

[0050] In one optional implementation, the method further includes: if the metadata processing request is a metadata operation, writing the metadata operation into a metadata operation queue of a log file system; and flushing the changed metadata into a remote distributed file system via an asynchronous worker thread, including: obtaining the metadata operation from the metadata operation queue via an asynchronous worker thread, and updating the metadata in the distributed file system using the obtained metadata operation.

[0051] Meta-information operations, such as creating, deleting, and renaming log files, are triggered during log splitting and affect log writing. The importance of meta-information operations is second only to log write requests. (See reference) Figure 1c LogFS also includes a Meta OP Queue, used to cache metadata operations such as creation, deletion, and renaming. In response to metadata operations, metadata in the metadata cache can be directly modified, and the operations are pushed into the Meta OP Queue. Asynchronous worker threads periodically retrieve metadata operations from the queue and update the metadata database in the distributed file system using these operations. Meta operations in the queue can be popped out in a first-in, first-out (FIFO) manner. By introducing the Meta OP Queue for asynchronous processing of metadata operations, system stability is further improved.

[0052] Figure 2a This is a flowchart of another remote log processing method for an application provided according to an embodiment of this disclosure. See also... Figure 2a The remote log processing method of the application in this embodiment may include:

[0053] S201, Get the application's log processing request;

[0054] S202, if the log processing request is a log read request, determine whether there is data to be read in the data cache of the log file system; if not, continue to S203; if it exists, obtain the data to be read from the data cache.

[0055] S203, pre-read the distributed file system and write the pre-read log data into the data cache of the log file system;

[0056] S204, determine the data to be read from the data cache;

[0057] S205, if the log processing request is a metadata processing request, the metadata processing request is responded to using the metadata cache in the log file system;

[0058] S206, if the log data in the data cache changes, or if the metadata in the metadata cache changes, the changed log data or changed metadata is flushed to a remote distributed file system via an asynchronous worker thread.

[0059] refer to Figure 2b LogFS (Log File System) can include not only a data cache and a metadata cache, but also a data cache. For user log read requests, the tail of the data cache can be read first to respond to the request. If the data cache is insufficient, a prefetch is performed on the distributed file system, pre-reading a set size (e.g., 4MB) of log data from the distributed file system and storing it in the data cache. The data cache is then used to respond to log read requests. In other words, if the data cache is insufficient, the data cache can be used to respond to log read requests; if the data cache is also insufficient, the distributed file system is pre-read again. For catch-up reads, the prefetch strategy, which pre-reads a set size of log data from the distributed file system and writes it to the data cache for later use, reduces the number of requests to the distributed file system and improves read throughput.

[0060] In one optional implementation, the pre-reading of the distributed file system includes: determining a target order based on the current position and the previous position of the catch-up read of the distributed file system; the target order is either sequential read or reverse read; and using the target order to pre-read the distributed file system.

[0061] For example, the system compares the current start position and the previous end position of the catch-up read. If the current start position is after the previous end position, the target order is determined to be sequential read, meaning the user is currently performing a sequential read; otherwise, the target order is determined to be reverse read. When the target order is sequential read, log data following the current request is pre-read from the distributed file system and stored in a data cache for subsequent reading. When the target order is reverse read, log data preceding the current request is pre-read from the distributed file system and stored in a data cache for subsequent reading. By performing sequential or reverse pre-read based on the current and previous positions of the catch-up read in the distributed file system, the success rate of responding to log read requests is significantly improved, thereby enhancing the overall system performance and response speed.

[0062] In one optional implementation, the metadata cache stores metadata of all files under a specified mount point; responding to the metadata processing request using the metadata cache in the log file system includes: if the metadata processing request is a read metadata request, searching for the metadata to be read from the metadata cache.

[0063] refer to Figure 2b The metadata cache stores metadata for all log files under the log volume integrated by the mount point. For metadata read requests, the local metadata response from the cache is used directly, without accessing the remote distributed file system. Metadata read requests can be either Stat requests or Readdir requests. Stat requests retrieve detailed metadata for a single file or directory, while Readdir requests list all filenames in a directory but do not directly provide metadata for these files. After retrieving the directory, Stat can be used to obtain the metadata for the corresponding files.

[0064] The technical solution provided in this disclosure, by setting up a high-speed data cache in LogFS and designing a special pre-read strategy for catch-up reading in combination with log usage scenarios, can reduce the number of requests to the distributed file system and further improve the read throughput of log data.

[0065] In one optional implementation, the changed metadata is flushed into a remote distributed file system via an asynchronous worker thread, including: when the log processing request is a rename operation, sending the rename operation to the distributed file system via an asynchronous worker thread, causing the target control node in the distributed file system to persist the rename operation, and controlling other control nodes in the distributed system to update the cached path directory.

[0066] For metadata stored in a database schema, finding file inodes via the full path has always been a major problem in the industry. Lookup operations that query files or directories level by level will generate N interactions, causing database I / O to be amplified N times and significantly increasing query latency, thus becoming a system bottleneck.

[0067] To more effectively address this issue, we designed and implemented the following optimization strategy: full-path directory caching. Regardless of whether it's a file or a directory, cached information is queried sequentially until the cache is exhausted, then a query is performed from the MetaDB (metadata database). For the last-level directory or file, the data in the database is always used as the reference. This ensures that the queried data is always reliable. In cache-hit scenarios, the number of I / O operations can be reduced to one, improving query performance while also solving the I / O amplification problem.

[0068] However, we also noticed that renaming (rename dir) operations can cause issues with cache consistency and correctness. To handle this scenario, we implement the following strategy: Upon receiving a rename operation, we persist the rename operation in MetaDB and simultaneously clear all entries related to that directory from the caches of all control nodes. The rename operation is considered successful only after all control node caches have been successfully cleared. In case of an exception, a retry mechanism can be used to ensure that all control nodes clear their caches successfully as much as possible. Even after reaching the required number of retries, the rename operation is still considered successful. Control nodes periodically fetch rename operation information from MetaDB to update their local caches. If a fetch fails, all local caches are cleared.

[0069] refer to Figure 2c Taking the request to rename / a / b / d to / a / c / d as an example, Master1 adds / a / c / d to its own cache and persists the renaming operation to MetaDB; each control node also deletes / a / b / d. Each control node can also periodically pull renaming operation information from MetaDB to update its local cache. (Reference) Figure 2dIf the deletion of / a / b / d fails on control node 3, or if the renaming operation from MetaDB fails, the cache is cleared so that it can be updated from MetaDB again later. Subsequently, control node 3 responds to metadata read requests by updating the cached metadata from MetaDB, thus maintaining the consistency and correctness of the cache in control node 3.

[0070] Figure 3a This is a flowchart of another remote log processing method for an application provided according to an embodiment of this disclosure. See also... Figure 3a The remote log processing method of the application in this embodiment may include:

[0071] S301, Obtain the current traffic quota of the target tenant in the target computing node in the current period from the traffic allocation server; wherein, the current traffic quota is determined based on the historical traffic of the target tenant in each computing node in the historical period;

[0072] S302, Obtain the log processing request of the application, and perform traffic supervision on the log processing request of the target tenant according to the current traffic quota;

[0073] S303, if the log processing request is a data processing request, the data processing request is responded to using the data cache in the log file system;

[0074] S304, if the log processing request is a metadata processing request, the metadata processing request is responded to using the metadata cache in the log file system;

[0075] S305, if the log data in the data cache changes, or if the metadata in the metadata cache changes, the changed log data or changed metadata is flushed to the remote distributed file system via an asynchronous worker thread.

[0076] In our remote logging system, the basic isolated object is the tenant. A single tenant may have multiple mount points accessing multiple stateless compute nodes (proxies) in the cluster. Our actual requirement is a traffic scheduling problem involving multiple clients and multiple servers. The target tenant is any tenant that generates the log processing request, and the target compute node is any compute node used to process the log processing request. (Reference) Figure 3b Tenant 1 can request Proxy 1 and Proxy 2 through client1 and client2 respectively, and Tenant 2 can request Proxy 1 and Proxy 2 through client3...clientN respectively.

[0077] refer to Figure 3bA QoS server is introduced between different compute nodes. Each compute node periodically reports its historical traffic on its own compute nodes to the QoS server, based on tenant and traffic type, regardless of the client used by the tenant. The traffic type can be throughput traffic for log data or metadata traffic. For each tenant, the QoS server determines the tenant's traffic allocation across compute nodes based on the tenant's historical traffic across these nodes and dynamically allocates the current traffic quota allowed for the current period to each compute node according to the traffic allocation. The target compute node obtains the tenant's current traffic quota from the QoS server and uses this current traffic quota to police the log processing requests sent by the target tenant to the target compute node.

[0078] Dynamic traffic allocation is unaffected by the client used by the tenant. (Reference) Figure 3b The target compute node 1 uses tenant 1's current traffic quota on the target compute node 1 to perform traffic policing on log processing requests sent by tenant 1 to the target compute node 1 via client 1 and client 2. The target compute node 1 also uses tenant 2's current traffic quota on the target compute node 1 to perform traffic policing on log processing requests sent by tenant 2 via client 3...client N to the target compute node 1. This process achieves dynamic traffic scheduling for multiple tenants and multiple mounts, ensuring minimal performance loss.

[0079] In one optional implementation, the current traffic quota is determined as follows: based on the historical traffic used by the target tenant through the target compute node in the historical period and the total historical traffic used by the target tenant through each compute node in the historical period, the traffic allocation ratio of the target tenant on the target compute node is determined; the current traffic quota is determined using the traffic allocation ratio and the total current traffic of the target tenant in the current period.

[0080] For example, dynamic traffic allocation can be performed in the following ways:

[0081] Ration i (T n ) = Throughout i (T n ) / Throughout total (T n );

[0082] Quota i (T n+1 Quota total ×Ration i (T n );

[0083] Among them, T n For the nth historical period, T n+1 For the current period, Ration i For the traffic allocation of the target tenant on the i-th compute node, Throughout i (T n ) represents the historical traffic used by the target tenant on the i-th compute node, Throughout total Quota represents the total historical traffic used by the target tenant across all compute nodes. i Quota is the current traffic quota for the target tenant on the i-th compute node. total This represents the total current traffic of the target tenant in the current period. The above processing enables dynamic traffic scheduling for multiple tenants and multiple mounts, reducing tenant costs. Furthermore, we can adjust the traffic scheduling period within an acceptable performance penalty range; the shorter the traffic scheduling period, the more precise the rate limiting effect.

[0084] The technical solution provided in this disclosure addresses the scenario of multi-tenant, multi-mounted remote log storage by periodically allocating quota values ​​for any tenant to each computing node and updating them at regular intervals. This allows computing nodes to independently monitor the traffic of multiple client requests from a tenant based on the allocated traffic quota, thereby achieving dynamic traffic scheduling for multi-tenant, multi-mounted systems and reducing tenant costs.

[0085] In one optional implementation, the changed log data is flushed to a remote distributed file system via an asynchronous worker thread, including: sending the changed log data to the distributed file system via the asynchronous worker thread, causing the distributed system to perform the following: dividing the acquired log data into at least one data block according to a set length; compressing and storing the data blocks that reach the set length; storing the data blocks that do not reach the set length uncompressed, and supplementing the data blocks with newly acquired log data.

[0086] In a distributed file system, the logical unit of data management is the stripe. Stripes can have a fixed length (L) and support both compressed and uncompressed types. Each stripe generates an index, thus resolving the index bloat problem caused by I / O writes. For the read amplification problem caused by fixed compression, data stripe splitting and concatenation are used to address this; complete stripes are compressed, while concatenated stripes are not.

[0087] During data block partitioning, received user log data can be divided into N data blocks according to a set length. The first N-1 data blocks of length L are compressed and stored, while the Nth data block, which is less than L, is stored uncompressed. For data block concatenation, during continuous user log data writing, if the previous data block fails to fill the entire block, the newly written data block is used to pad it to size L, and the remaining data is then partitioned according to size L. This hybrid compression process provides a good data compression ratio and optimizes the index size.

[0088] refer to Figure 3c The system sends changed log data to the distributed file system via asynchronous worker threads. The distributed file system then flushes the updated log data as follows: For user data 0 (userdata0), it is divided into four blocks of a set length L. The first three blocks are all of length L and are compressed. The compressed data is stored on the hard drive as page0, page1, and page2. The last stripe, 3, is shorter than the set length L and is stored uncompressed, corresponding to page3 on the hard drive. For user data 1 (userdata1), it is padded into stripe3 uncompressed. At this point, stripe3 is still shorter than the set length L. For user data 2 (userdata2), some data is padded into stripe3 until stripe3 reaches the fixed length L, and then written to page3 on the hard drive. Page3 is stored uncompressed. The remaining part of user data 2 is then divided according to the fixed length L, and the data blocks that reach L are compressed and padded into stripe4 and stripe5 respectively.

[0089] Figure 4 This is a schematic diagram of a remote log processing device for an application according to an embodiment of this disclosure. This device is suitable for printing user application logs to cloud storage. The device can be implemented in software and / or hardware and can be integrated into an electronic device. Figure 4 As shown, the remote log processing device 400 of the application in this embodiment may include:

[0090] The request acquisition module 410 is used to acquire the application's log processing requests;

[0091] The data processing module 420 is used to respond to the data processing request using the data cache in the log file system when the log processing request is a data processing request;

[0092] Meta-information processing module 430 is used to respond to the meta-information processing request using the meta-information cache in the log file system when the log processing request is a meta-information processing request.

[0093] The asynchronous change module 440 is used to flush the changed log data or changed metadata into a remote distributed file system via an asynchronous worker thread when the log data in the data cache changes, or when the metadata in the metadata cache changes.

[0094] In one optional embodiment, the metadata processing module 430 further includes an operation queue processing unit, which is used to write the metadata operation into the metadata operation queue of the log file system when the metadata processing request is a metadata operation.

[0095] The asynchronous change module 440 is specifically used for:

[0096] The metadata operations are obtained from the metadata operation queue through an asynchronous worker thread, and the obtained metadata operations are used to update the metadata in the distributed file system.

[0097] In one alternative embodiment, the data processing module 420 includes:

[0098] The first reading unit is used to determine whether there is data to be read in the data buffer when the data processing request is a log reading request;

[0099] A prefetch unit is used to prefetch the distributed file system if it does not exist, and write the prefetched log data into the data cache of the log file system;

[0100] The second reading unit is used to determine the data to be read from the data cache.

[0101] In one alternative implementation, the prefetching unit includes:

[0102] The sequence subunit is used to determine the target order based on the current position and the previous position of the catch-up read of the distributed file system; the target order is either sequential read or reverse read.

[0103] The prefetching subunit is used to prefetch the distributed file system according to the target order.

[0104] In one optional implementation, the metadata cache stores metadata of all files under the specified mount point;

[0105] The metadata processing module 430 is specifically used for:

[0106] If the metadata processing request is a metadata read request, the metadata to be read is retrieved from the metadata cache.

[0107] In one optional implementation, the asynchronous change module 440 is specifically used for:

[0108] By sending changed log data to the distributed file system via an asynchronous worker thread, the distributed system performs the following actions: dividing the acquired log data into at least one data block according to a set length; compressing and storing the data blocks that have reached the set length; storing the data blocks that have not reached the set length without compression, and supplementing the data blocks with newly acquired log data.

[0109] In an optional embodiment, the remote log processing device 400 for the above application further includes:

[0110] The traffic quota module is used to obtain the current traffic quota of the target tenant in the target computing node in the current period from the traffic allocation server; wherein, the current traffic quota is determined based on the historical traffic of the target tenant in each computing node in the historical period;

[0111] The traffic monitoring module is used to monitor the traffic of the target tenant based on the current traffic quota.

[0112] In one optional implementation, the current traffic quota is determined as follows: based on the historical traffic used by the target tenant through the target compute node in the historical period and the total historical traffic used by the target tenant through each compute node in the historical period, the traffic allocation ratio of the target tenant on the target compute node is determined; the current traffic quota is determined using the traffic allocation ratio and the total current traffic of the target tenant in the current period.

[0113] In one optional implementation, the asynchronous change module 440 is specifically used for:

[0114] In the case where the log processing request is a rename operation, the rename operation is sent to the distributed file system through an asynchronous worker thread, so that the target control node in the distributed file system persists the rename operation and controls other control nodes in the distributed system to update the cached path directory.

[0115] The remote log processing device for the application provided in the embodiments of the present invention can execute the remote log processing method of the application provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0116] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0117] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0118] Figure 5 This is a block diagram of an electronic device used to implement the remote log processing method of the application in the embodiments of this disclosure. Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0119] like Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. The RAM 503 may also store various programs and data required for the operation of the electronic device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0120] Multiple components in electronic device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows electronic device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0121] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as a remote logging method for an application. For example, in some embodiments, the remote logging method for an application can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the remote logging method for an application described above can be performed. Alternatively, in other embodiments, the computing unit 501 can be configured to perform the remote logging method for an application by any other suitable means (e.g., by means of firmware).

[0122] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0123] like Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. The RAM 503 may also store various programs and data required for the operation of the electronic device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0124] Multiple components in electronic device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows electronic device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0125] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as a remote logging method for an application. For example, in some embodiments, the remote logging method for an application can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the remote logging method for an application described above can be performed. Alternatively, in other embodiments, the computing unit 501 can be configured to perform the remote logging method for an application by any other suitable means (e.g., by means of firmware).

[0126] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0127] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0128] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0129] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, audio input, or tactile input).

[0130] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0131] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0132] Artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies mainly include computer vision, audio recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0133] Cloud computing refers to a technology system that enables access to a shared pool of physical or virtual resources via a network. These resources can include servers, operating systems, networks, software, applications, and storage devices, and can be deployed and managed on demand and in a self-service manner. Cloud computing technology can provide efficient and powerful data processing capabilities for applications such as artificial intelligence and blockchain, as well as for model training.

[0134] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0135] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for remote log processing in an application, comprising: Get the application's log processing request; When the log processing request is a data processing request, the data processing request is responded to using the data cache in the log file system; When the log processing request is a metadata processing request, the metadata processing request is responded to using the metadata cache in the log file system; If the log data in the data cache changes, or if the metadata in the metadata cache changes, the changed log data or changed metadata is flushed to a remote distributed file system via an asynchronous worker thread. The log file system includes a data cache area, a metadata cache area, and asynchronous worker threads; the data cache area is used to cache data written by the mount point user within a minute, and the metadata cache area is used to cache the metadata of the mount point.

2. The method according to claim 1, further comprising: If the metadata processing request is a metadata operation, the metadata operation is written into the metadata operation queue of the log file system. The changed metadata is flushed to a remote distributed file system via asynchronous worker threads, including: The metadata operations are obtained from the metadata operation queue through an asynchronous worker thread, and the obtained metadata operations are used to update the metadata in the distributed file system.

3. The method according to claim 1, wherein, The step of responding to the data processing request using a data cache in the log file system includes: If the data processing request is a log read request, determine whether there is data to be read in the data cache. If it does not exist, the distributed file system is pre-read, and the pre-read log data is written into the data cache of the log file system; The data to be read is determined from the data cache.

4. The method according to claim 3, wherein, The pre-reading of the distributed file system includes: The target order is determined based on the current position and the previous position of the catch-up read in the distributed file system; the target order is either sequential read or reverse read. The distributed file system is pre-read using the target order.

5. The method according to claim 1, wherein, The metadata cache stores metadata for all files under the specified mount point; The step of responding to the metadata processing request using the metadata cache in the log file system includes: If the metadata processing request is a metadata read request, the metadata to be read is retrieved from the metadata cache.

6. The method according to claim 1, wherein, The changed log data is flushed to a remote distributed file system via asynchronous worker threads, including: The distributed file system sends changed log data to the asynchronous worker thread, causing the distributed file system to perform the following actions: dividing the acquired log data into at least one data block according to a set length; compressing and storing the data blocks that reach the set length; storing the data blocks that do not reach the set length uncompressed, and supplementing the data blocks with newly acquired log data.

7. The method according to claim 1, further comprising: Obtain the current traffic quota of the target tenant in the target compute node for the current period from the traffic allocation server; wherein, the current traffic quota is determined based on the historical traffic of the target tenant in each compute node in the historical period; Traffic monitoring is performed on the target tenant based on the current traffic quota.

8. The method according to claim 7, wherein, The current traffic quota is determined as follows: based on the historical traffic used by the target tenant through the target computing node in the historical period and the total historical traffic used by the target tenant through each computing node in the historical period, the traffic allocation ratio of the target tenant on the target computing node is determined; The current traffic quota is determined using the traffic allocation ratio and the total current traffic of the target tenant in the current period.

9. The method according to claim 1, wherein, The changed metadata is flushed to a remote distributed file system via asynchronous worker threads, including: In the case where the log processing request is a rename operation, the rename operation is sent to the distributed file system through an asynchronous worker thread, so that the target control node in the distributed file system persists the rename operation and controls other control nodes in the distributed system to update the cached path directory.

10. A remote log processing device for an application, comprising: The request retrieval module is used to retrieve the application's log processing requests; The data processing module is configured to respond to the data processing request using the data cache in the log file system when the log processing request is a data processing request; The metadata processing module is used to respond to the metadata processing request using the metadata cache in the log file system when the log processing request is a metadata processing request. The asynchronous change module is used to flush the changed log data or changed metadata into a remote distributed file system via an asynchronous worker thread when the log data in the data cache changes or when the metadata in the metadata cache changes. The log file system includes a data cache area, a metadata cache area, and asynchronous worker threads; the data cache area is used to cache data written by the mount point user within a minute, and the metadata cache area is used to cache the metadata of the mount point.

11. The apparatus according to claim 10, wherein the metadata processing module further comprises an operation queue processing unit, configured to write the metadata operation into the metadata operation queue of the log file system when the metadata processing request is a metadata operation; The asynchronous change module is specifically used for: The metadata operations are obtained from the metadata operation queue through an asynchronous worker thread, and the obtained metadata operations are used to update the metadata in the distributed file system.

12. The apparatus according to claim 10, wherein, The data processing module includes: The first reading unit is used to determine whether there is data to be read in the data buffer when the data processing request is a log reading request; A prefetch unit is used to prefetch the distributed file system if it does not exist, and write the prefetched log data into the data cache of the log file system; The second reading unit is used to determine the data to be read from the data cache.

13. The apparatus according to claim 12, wherein, The prefetching unit includes: The sequence subunit is used to determine the target order based on the current position and the previous position of the catch-up read of the distributed file system; the target order is either sequential read or reverse read. The prefetching subunit is used to prefetch the distributed file system according to the target order.

14. The apparatus according to claim 10, wherein, The metadata cache stores metadata for all files under the specified mount point; The metadata processing module is specifically used for: If the metadata processing request is a metadata read request, the metadata to be read is retrieved from the metadata cache.

15. The apparatus according to claim 10, wherein, The asynchronous change module is specifically used for: The distributed file system sends changed log data to the asynchronous worker thread, causing the distributed file system to perform the following actions: dividing the acquired log data into at least one data block according to a set length; compressing and storing the data blocks that reach the set length; storing the data blocks that do not reach the set length uncompressed, and supplementing the data blocks with newly acquired log data.

16. The apparatus of claim 10, further comprising: The traffic quota module is used to obtain the current traffic quota of the target tenant in the target computing node in the current period from the traffic allocation server; wherein, the current traffic quota is determined based on the historical traffic of the target tenant in each computing node in the historical period; The traffic monitoring module is used to monitor the traffic of the target tenant based on the current traffic quota.

17. The apparatus according to claim 16, wherein, The current traffic quota is determined as follows: based on the historical traffic used by the target tenant through the target computing node in the historical period and the total historical traffic used by the target tenant through each computing node in the historical period, the traffic allocation ratio of the target tenant on the target computing node is determined; The current traffic quota is determined using the traffic allocation ratio and the total current traffic of the target tenant in the current period.

18. The apparatus according to claim 10, wherein the asynchronous change module is specifically used for: In the case where the log processing request is a rename operation, the rename operation is sent to the distributed file system through an asynchronous worker thread, so that the target control node in the distributed file system persists the rename operation and controls other control nodes in the distributed system to update the cached path directory.

19. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-9.

21. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-9.

Images