A front-end log collection method, device and system

Abstract

The application discloses a front-end log collection method, device and system, and realizes low-cost and safe collection of front-end logs. The system comprises: a front-end application configured to obtain an encryption token from an enterprise log platform, and push the front-end logs after encapsulation to a cloud log service platform, wherein the encryption token is used for encrypted transmission of the front-end logs in the cloud log service platform; the cloud log service platform is configured to receive the front-end logs pushed by the front-end application and realize temporary storage of the front-end logs, so as to be consumed by the enterprise log platform; the enterprise log platform is configured to refresh the encryption token at a certain refresh frequency, subscribe to a log consumption service of the cloud log service platform to obtain the front-end logs, and store the front-end logs; the front-end application stops pushing the front-end logs after failing to obtain the encryption token from the enterprise log platform, and the enterprise log platform obtains the front-end logs from the cloud log service platform by using a high-speed queue cache mechanism.

Description

Technical Field

[0001] This invention relates to the field of software development, and in particular to a front-end log collection method, apparatus and system. Background Technology

[0002] Log and statistical data from front-end services for apps or websites are crucial information for back-end engineers to analyze program performance and are also important references for business operational decisions. While some open-source software exists for collecting these logs, it also has many limitations.

[0003] Existing technologies collect and store log data in the cloud. However, due to the large volume of log data, persistent cloud storage is costly, and developers have a limited number of cloud storage accounts. Configuring multiple cloud storage accounts for developers would also increase costs.

[0004] Storing logs in the cloud presents the problem of extremely high bandwidth costs for secondary log processing. Furthermore, current technologies for pushing logs to the cloud rely on WebTracking, a completely unencrypted log transmission solution, which carries significant security risks. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a front-end log collection method, apparatus, and system, which achieves low-cost and secure collection of front-end logs by constructing an enterprise's front-end log acquisition solution.

[0006] In a first aspect, the present invention provides a front-end log collection system, comprising: a front-end application, a cloud log service platform, and an enterprise log platform; The front-end application is configured to obtain an encryption token from the enterprise log platform, encrypt the front-end logs generated by the front-end application using the encryption token, and then encapsulate the encrypted front-end logs and push them to the cloud log service platform, wherein the encryption token is used for encrypted transmission of the front-end logs in the cloud log service platform. The cloud log service platform is configured to receive the front-end logs pushed by the front-end application and temporarily store the front-end logs for consumption by the enterprise log platform. The enterprise log platform is configured to refresh the encryption token at a certain refresh frequency, subscribe to log consumption services from the cloud log service platform to obtain the front-end logs, and store the front-end logs; After the front-end application fails to obtain the encryption token from the enterprise log platform, it stops pushing the front-end logs. The enterprise log platform then uses a high-speed queue caching mechanism to obtain the front-end logs from the cloud log service platform.

[0007] In one embodiment, the enterprise log platform includes an encryption service module, which is configured to provide the front-end application with an interface for obtaining the encryption token and to refresh the encryption token when the encryption token is not valid.

[0008] In one embodiment, the encryption service module further includes a whitelist module, which is configured to receive applications for adding whitelisted applications and add the whitelisted applications to the encryption service module so that the whitelisted applications can obtain the encryption token.

[0009] In one embodiment, the enterprise log platform includes a message subscription module, which is configured to provide a high-speed queue caching mechanism to batch retrieve the front-end logs from the cloud log service platform.

[0010] In one embodiment, the enterprise log platform includes a synchronization plugin module, which is configured to listen to the latest front-end logs in the cloud log service platform by comparing the offset of the log storage, and push the latest front-end logs to the message subscription module.

[0011] In one embodiment, the enterprise log platform includes a format conversion module configured to convert the front-end logs into a user-readable format.

[0012] In one embodiment, the cloud log service platform is further configured to receive the front-end logs transmitted by the front-end application through a log transmission method that encapsulates H5 using the Node.js programming language.

[0013] A second aspect of the present invention provides a front-end log collection method, comprising: Provide the front-end application with an encryption token for obtaining encrypted front-end logs, and an interface for obtaining the encryption token, which is used for authentication of front-end log transmission; Subscribe to log consumption service from cloud log service platform, listen to the latest front-end logs in cloud log service platform by comparing the log storage offset, and synchronously obtain the front-end logs; The front-end logs are pushed to the Kafka open-source component, production data is consumed from the Kafka open-source component, and the front-end logs are stored.

[0014] In one embodiment, providing the front-end application with an encryption token for obtaining encrypted front-end logs includes: The front-end application is provided with a temporary encryption token. The validity of the encryption token is determined. If the encryption token is invalid, it is updated.

[0015] A third aspect of the present invention provides a front-end log collection device, comprising: The encryption module is configured to provide the front-end application with an encryption token for obtaining encrypted front-end logs, and an interface for obtaining the encryption token, wherein the encryption token is used for authentication of the transmission of front-end logs; The synchronization module is configured to subscribe to log consumption services from the cloud log service platform, listen to the latest front-end logs in the cloud log service platform by comparing the offset of the log storage, and synchronously obtain the front-end logs. The collection module is configured to push the front-end logs to the Kafka open-source component, consume production data from the Kafka open-source component, and store the front-end logs.

[0016] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a computer, performs the method described in the first aspect of the embodiments of the present invention.

[0017] This invention improves information security by enabling encrypted transmission of front-end logs through information interaction between a self-developed front-end application and an enterprise log platform for long-term storage of front-end logs using encryption tokens. It uses a cloud log service platform as a storage intermediary for front-end logs and ultimately stores the front-end logs persistently in the enterprise log platform. During this process, the cloud log service platform acts as a log transfer platform, requiring less storage space and accounts, thus reducing costs. Furthermore, the enterprise log platform, as local storage, can meet the needs of long-term storage of front-end logs. Attached Figure Description

[0018] Figure 1 This is a timing diagram of a front-end log collection system according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating a front-end log collection method according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the business process of a front-end log collection system according to an embodiment of the present invention; Figure 4 This is a schematic diagram of a front-end log collection device in an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Given the lack of control over front-end logs in enterprises, this invention enables enterprises to achieve a significant leap forward in front-end log collection and analysis. Prior to this invention, enterprises used Alibaba Cloud's SLS platform (Log Service System) to store front-end logs. However, cloud storage made it difficult to effectively manage front-end logs, and the limited number of administrators due to Alibaba Cloud account limitations made log processing inconvenient and untimely. The Alibaba Cloud SLS platform's ability to collect and store front-end logs is increasingly failing to meet enterprise needs; therefore, finding a solution for local management of front-end logs on the enterprise side is essential. Of course, this invention is not limited to Alibaba Cloud; remote services such as Huawei Cloud will also be applicable.

[0021] Please see Figure 1 The present invention provides a front-end log collection system, comprising: a front-end application, a cloud log service platform, and an enterprise log platform. The enterprise log platform further comprises an encryption service module, a whitelist module, a message subscription module, a synchronization plugin module, a format conversion module, and a verification module. The encryption service module, whitelist module, message subscription module, synchronization plugin module, format conversion module, and verification module are all integrated within the enterprise log platform and correspond to different execution functions.

[0022] The front-end application (SDK) is a self-developed component of the enterprise, developed and coded using NodeJS, replacing the original Alibaba Cloud SDK. Based on Alibaba Cloud's interface, the front-end application transmits front-end logs to the cloud log service platform through a log transmission method that encapsulates H5 (Hypertext Markup Language) using the NodeJS development language. The cloud log service platform receives the front-end logs transmitted by the front-end application.

[0023] The enterprise log platform is a backend service written in Golang, built internally within the enterprise using the Graylog open-source component for persistent storage of frontend logs. The enterprise log platform and the frontend application are designed with a Vue architecture to meet H5 log transmission authentication requirements. The enterprise log platform provides a unified method for obtaining temporary encryption tokens and a high-performance backend service for log transfer, connecting the enterprise end with the Alibaba Cloud SLS platform. Ultimately, frontend logs are placed into the enterprise's internal log storage system, thus ensuring that logs, from frontend generation to final enterprise use, provide the most effective way for business developers to promptly and effectively troubleshoot user issues, while also reducing application costs.

[0024] Specifically, according to Figure 1As shown in the sequence diagram, when a user interacts with a webpage through their browser, the site generates front-end logs in a certain way. The self-developed front-end application encrypts and reports this front-end log data, improving system security and log integrity to a certain extent. It should be noted that this encryption refers to the encryption of the front-end log content itself, which differs from existing technologies that send account, password, and device MAC codes to Alibaba Cloud for verification, and also from the encryption of communication between the front-end application and Alibaba Cloud. This encryption distinguishes itself from the unencrypted Webtracking transmission method used in existing technologies. Since the Alibaba Cloud native SDK does not support encrypted log uploads to Alibaba Cloud, encrypted uploads are implemented through a self-developed front-end application to prevent the enterprise's Alibaba Cloud SLS platform from being exposed to the public network without encryption.

[0025] If the front-end service requires key-based authentication when transmitting logs, there is a risk of key leakage due to key configuration on the front-end service, and the cost of changing keys after an attack is high. This invention addresses this by providing a unified interface for generating temporary keys. The front-end application only needs to periodically call the interface to obtain a temporary key for verification, eliminating the need for the front-end to store any data and thus improving security for log transmission authentication.

[0026] The following explanation uses Alibaba Cloud SLS platform as an example of a cloud-based log service platform: The front-end application obtains an encryption token from the encryption service module. The encryption service module of the enterprise log platform provides the front-end application with a unified interface for obtaining the encryption token, which the front-end application can call to return a temporary authentication encryption token. The encryption service module refreshes the encryption token at a certain frequency to ensure its timeliness.

[0027] After obtaining the encryption token, the front-end application encrypts the locally generated front-end logs. If the front-end application fails to obtain the encryption token from the encryption service module, or if the encryption token expires, it stops pushing the front-end logs. Simultaneously, the front-end application checks the validity of the encryption token and refreshes it if the token's validity is insufficient.

[0028] For example, the front-end application checks if the locally cached encryption token is valid (non-existent or expired). If the encryption token is about to expire (within 3-5 minutes), it immediately calls the encryption service module's service interface to obtain the latest encryption token. If the retrieval fails, it retryes after a period of time. If the retrieval is successful, it is cached locally. The encryption service module also verifies the validity of the encryption token. When the encryption token's validity is insufficient, it refreshes the encryption token to ensure that the encryption token obtained by the front-end application is usable. For example, it can be set to refresh every 30 minutes, or other intervals can be set.

[0029] The encrypted front-end logs are encapsulated and pushed to a cloud-based log service platform. The encryption token is used for encrypted transmission of the front-end logs within the cloud-based log service platform. This method of obtaining a unified temporary encryption token ensures that the authentication key is not leaked. The front-end can only obtain a temporary encryption token with a relatively short validity period for log transmission authentication, which meets the minimum access control (MINUTE) requirements. This service can be scaled in parallel to meet high-pressure requests.

[0030] When users interact with a webpage or app, the generated logs are either stored locally in the front-end application or uploaded immediately (this can be achieved through a front-end defined SDK). The front-end application compresses these logs and then pushes them synchronously or asynchronously to the Alibaba Cloud SLS platform using encrypted token authentication. It should be understood that the front-end application uses an encrypted token to encrypt the generated logs before uploading them to the Alibaba Cloud SLS platform. The enterprise log platform then retrieves these logs from the Alibaba Cloud SLS platform. The Alibaba Cloud SLS platform only stores front-end logs for a short period, and this temporary data storage reduces costs.

[0031] The cloud-based log service platform receives the front-end logs pushed by the front-end application, verifies the encryption token (verified by the Alibaba Cloud SLS platform), and temporarily stores the front-end logs after successful verification. This allows for continuous updates of the front-end logs while occupying relatively little storage space on the Alibaba Cloud SLS platform, and also facilitates consumption by the enterprise log platform. Since the enterprise log platform subscribes to and consumes the front-end log streams from the Alibaba Cloud SLS platform, multiple Alibaba Cloud accounts are not required to maintain the front-end logs on the Alibaba Cloud SLS platform, thus reducing costs.

[0032] The enterprise log platform consumes front-end log streams from the Alibaba Cloud SLS platform through a built-in synchronization plugin module. This synchronization plugin module is configured to monitor the latest front-end logs in the cloud-based log service platform by comparing the log storage offset. Specifically, using heartbeat detection, a coroutine sends a heartbeat request to the SLS server (i.e., the Alibaba Cloud SLS platform) every 20 seconds. If the heartbeat fails, the consumption status is set to SHUTDOWN. Each heartbeat checks if the SLShard has changed; if it has, it is updated; otherwise, the consumption connection is removed. Then, the SLShard list (log shards) is obtained, and the Shard information is iterated in an infinite loop (the SHUTDOWN status indicates the end of the infinite loop). Multiple coroutines are started for each Shard to monitor the log retrieval process status. The log retrieval process status includes seven states: INITIALIZING, INITIALIZING_DONE, PULL_PROCESSING, PULL_PROCESSING_DONE, CONSUME_PROCESSING, CONSUME_PROCESSING_DONE, and SHUTDOWN. For each shard, a coroutine is first started for INITIALIZING. This process retrieves the latest consumption point (i.e., offset) of the current shard. If no consumption has occurred, the parameter in the configuration file is used as the latest consumption point, and the status is set to INITIALIZING_DONE. When the PULL (coroutine) detects the INITIALIZING_DONE or CONSUME_PROCESSING_DONE status, the process pulls the SLS log. If the pull process fails, the status is rolled back to INITIALIZING_DONE; if successful, the consumption point is recorded and refreshed. The SLS server records the current state as PULL_PROCESSING_DONE. When the coroutine CONSUME_PROCESSING detects that the current state is PULL_PROCESSING_DONE, it calls a custom consumption process function to consume the logs obtained from the SLS server. If the custom function fails, the consumption point is rolled back. Both success and failure will mark the state as CONSUME_PROCESSING_DONE. When the running program captures a termination signal, it enters the SHUTDOWN state, at which point the consumption point is synchronized to the SLS server for recording.

[0033] By setting up a synchronization plugin to subscribe to a consumer group on the Alibaba Cloud SLS platform, the latest front-end logs can be obtained in batches in a very short time. Tests have shown that data consumption can reach 2 million per minute. Of course, the data consumption capacity can be further expanded by modifying the functional modules of this invention later, such as by extending the synchronization plugin module in parallel.

[0034] After consuming the front-end log reports from the Alibaba Cloud SLS platform, the synchronization plugin module pushes the latest front-end logs to the message subscription module. The message subscription module is configured to provide a high-speed queue caching mechanism for batch retrieval of the front-end logs from the cloud log service platform. For example, the message subscription module uses the Kafka open-source component. Kafka is a high-throughput distributed publish-subscribe messaging system that can handle all action stream data from consumers on a website. Kafka is included in the enterprise log platform and provides a high-speed queue caching mechanism. Users can push produced data to Kafka, and consumers can consume the produced data through subscriptions.

[0035] The enterprise log platform provides storage space and retrieves log streams from the message subscription module for local storage. Specifically, it subscribes to log stream data from a specified topic in the Kafka open-source component and persists the data to the local disk. This invention utilizes multi-coroutine monitoring (similar to the heartbeat detection and coroutine status detection mentioned above) on the Alibaba Cloud SLS platform to achieve log transfer to the local log system, achieving latency in the second range even for high data volumes.

[0036] As can be seen from the above embodiments, the self-developed enterprise log platform of this invention refreshes the encryption token at a certain refresh frequency, providing a temporary encryption token for the self-developed front-end application. Verification is only required by calling the interface once at set intervals to obtain the temporary encryption token, and authentication of log transmission can be completed without the front-end storing any data. The enterprise log platform subscribes to the log consumption service of the cloud log service platform to obtain the front-end logs and stores them, providing a localized storage solution for the front-end logs and achieving persistent storage. The enterprise log platform utilizes a high-speed queue caching mechanism to obtain the front-end logs from the cloud log service platform, effectively synchronizing the log stream of the cloud log service platform.

[0037] Furthermore, the encryption service module also includes a whitelist module. The whitelist module is configured to receive applications to add whitelisted applications and add the whitelisted applications to the encryption service module, enabling the whitelisted applications to obtain the encryption token. The encryption token interface in this invention requires the corresponding service to be whitelisted before access, improving security. For example, the APP needs to be managed to obtain preset APP front-end logs to prevent the encryption token from being invoked. A whitelist interface can be provided directly during development to add whitelisted applications to the system. Technical personnel can apply to add applications to the whitelist via email or a web interface, preventing anonymous users from arbitrarily calling the interface to obtain authenticated encryption tokens. Adding a whitelisted application takes effect after restarting the service. The service adopts a high-availability deployment method, and nodes can be scaled in parallel according to business volume. The service automatically polls and detects all added whitelisted applications.

[0038] Furthermore, to provide business development personnel with the most effective data for timely and efficient troubleshooting of user issues, and to allow them to fully utilize locally stored front-end logs, the enterprise log platform also includes a format conversion module. This module is configured to convert the front-end logs into a user-readable format. Specifically, it initiates multiple coroutines to perform secondary processing on the reported log stream, serializes the log stream data, matches the log content according to a predefined blacklist field ID, filters out some illegal log data, and pushes the processed data to the log platform for final display to the user.

[0039] like Figure 2 As shown, this invention also provides a front-end log collection method, which is applicable to enterprise log platforms within an enterprise. Enterprises can build corresponding log management platforms according to their needs. This invention designs an enterprise log management platform that meets the requirements of H5 log transmission using the Vue architecture. The front-end log collection method includes: Step 210: Provide the front-end application with an encryption token for obtaining encrypted front-end logs, and an interface for obtaining the encryption token, which is used for front-end log transmission authentication.

[0040] The enterprise log platform can provide a unified interface to the front-end application APP or website. The front-end application can use the unified interface to obtain the encryption token. After obtaining the encryption token, the front-end application uses it for authentication to transmit front-end logs to the cloud log service platform.

[0041] For example, the enterprise log platform provides a temporary encryption token to the front-end application, determines whether the encryption token is valid, and updates it if it expires. Alternatively, the front-end application can also determine the validity of the obtained temporary encryption token by setting a 30-minute validity period and verifying its expiration.

[0042] Step 220: Subscribe to the log consumption service on the cloud log service platform, compare the log storage offset with the latest front-end logs in the cloud log service platform, and synchronously obtain the front-end logs.

[0043] After the front-end application transmits encrypted front-end logs to the cloud log service platform, the cloud log service platform authenticates the encryption token, and the front-end logs are sent to the queue of the cloud log service platform for consumption. The enterprise log platform can consume the front-end logs in the queue using its built-in functional modules. As described above regarding the synchronization plugin module of this invention, it will not be repeated here.

[0044] Step 230: Push the front-end logs to the Kafka open-source component, consume production data from the Kafka open-source component, and store the front-end logs.

[0045] After consuming the front-end log reports from the cloud log service platform, the synchronization plugin module pushes the latest front-end logs to the Kafka open-source component. The Kafka open-source component is configured to provide a high-speed queue caching mechanism for batch retrieval of the front-end logs from the cloud log service platform. The Kafka open-source component, included in the enterprise log platform, provides a high-speed queue caching mechanism. Users can push production data to the Kafka open-source component, and consumers can consume the production data through subscriptions. The enterprise log platform then subscribes to the front-end logs in the Kafka open-source component and stores them within the enterprise log platform for long-term storage. This facilitates rapid utilization of front-end logs by the enterprise, reduces dependence on the cloud platform, and thus lowers costs. It also allows for subsequent iterative updates to capabilities such as log filtering and log verification, ultimately forming a fully self-developed log transmission solution.

[0046] This invention consumes front-end logs from a cloud-based log service platform through a self-developed enterprise log platform, and encrypts the front-end logs by using encryption tokens obtained by the front-end application, thereby improving the security of information transmission. For any aspects of the enterprise log platform not covered herein, please refer to the section on the functions and roles of the enterprise log platform in the front-end log collection system of this invention, which will not be elaborated upon further.

[0047] For details, please refer to Figure 3 The business process diagram of the front-end log collection method is shown in the figure: The browser app sends a request to the front-end application's front-end service SDK. The front-end application checks the validity of the encryption token; if valid, it pushes front-end logs to the cloud-based log service platform, SLS. The enterprise log platform subscribes to the front-end logs in the SLS queue. If the subscription fails, it retryes three times. If it still fails, it discards the subscription request and then pushes the front-end logs to the Kafka queue. The enterprise log platform subscribes to and stores the front-end logs in the Kafka queue.

[0048] Front-end developers can add whitelisted applications to the front-end log collection system by adding applications and updating their configurations. The encryption token service is also updated, allowing for asynchronous updates, interval queries, and scheduled updates every 30 minutes. The enterprise log platform provides a unified encryption token acquisition interface to front-end applications, which can be accessed via WAF (Web Application Firewall), SLB (Server Load Balancer), or Kong database / Kubernetes (K8S) to update encryption tokens. Front-end applications determine whether they have successfully acquired the encryption token and either terminate log uploads or save the logs locally on the front-end.

[0049] In existing technologies, accessing the app involves writing the Alibaba Cloud Access Key into the front-end service, posing a risk of exposure. Furthermore, there's no authentication mechanism for PC-side log reporting; once open, any anonymous user can report logs, and the cost of remediation after exposure is extremely high. This invention addresses this by requesting a temporary, encrypted token from the application server. This token can be used by the client application to access the log service, and a new token is required after its expiration. The client authenticates with Alibaba Cloud using this temporary token. Alibaba Cloud verifies the validity of the issued token using pre-defined user policies and encrypts it, thus improving the security of information transmission.

[0050] like Figure 4 As shown, the present invention provides a front-end log collection device corresponding to the aforementioned front-end log collection method, comprising: The encryption module 41 is configured to provide the front-end application with an encryption token for obtaining encrypted front-end logs, and an interface for obtaining the encryption token, wherein the encryption token is used for authentication of the transmission of front-end logs. Synchronization module 42 is configured to subscribe to log consumption services from the cloud log service platform, listen to the latest front-end logs in the cloud log service platform by comparing the offset of the log storage, and synchronously obtain the front-end logs. The collection module 43 is configured to push the front-end logs to the Kafka open-source component, consume production data from the Kafka open-source component, and store the front-end logs.

[0051] The present invention also provides a computer-readable storage medium on which a computer program is stored, and which, when executed by a processor, implements the aforementioned front-end log collection system.

[0052] It is understood that computer-readable storage media can include: any entity or device capable of carrying computer programs, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), and software distribution media, etc. Computer programs include computer program code. Computer program code can be in the form of source code, object code, executable files, or certain intermediate forms, etc. Computer-readable storage media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), and software distribution media, etc.

[0053] In some embodiments of the present invention, the device may include a controller, which is a microcontroller chip integrating a processor, memory, communication module, etc. The processor may refer to the processor included in the controller. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0054] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0055] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0056] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A front-end log collection system, characterized in that, include: Front-end applications, cloud-based log service platforms, and enterprise log platforms; The front-end application is configured to obtain an encryption token from the enterprise log platform, encrypt the front-end logs generated by the front-end application using the encryption token, and then encapsulate the encrypted front-end logs and push them to the cloud log service platform, wherein the encryption token is used for encrypted transmission of the front-end logs in the cloud log service platform. The cloud log service platform is configured to receive the front-end logs pushed by the front-end application and temporarily store the front-end logs for consumption by the enterprise log platform. The enterprise log platform is configured to refresh the encryption token at a certain refresh frequency, subscribe to log consumption services from the cloud log service platform to obtain the front-end logs, and store the front-end logs; After the front-end application fails to obtain the encryption token from the enterprise log platform, it stops pushing the front-end logs. The enterprise log platform then uses a high-speed queue caching mechanism to obtain the front-end logs from the cloud log service platform.

2. The front-end log collection system according to claim 1, characterized in that, The enterprise log platform includes an encryption service module, which is configured to provide the front-end application with an interface for obtaining the encryption token and to refresh the encryption token when the encryption token is not valid.

3. The front-end log collection system according to claim 2, characterized in that, The encryption service module also includes a whitelist module, which is configured to receive applications to add whitelisted applications and add the whitelisted applications to the encryption service module so that the whitelisted applications can obtain the encryption token.

4. The front-end log collection system according to claim 1, characterized in that, The enterprise log platform includes a message subscription module, which is configured to provide a high-speed queue caching mechanism to batch retrieve the front-end logs from the cloud log service platform.

5. The front-end log collection system according to claim 4, characterized in that, The enterprise log platform includes a synchronization plugin module, which is configured to listen to the latest front-end logs in the cloud log service platform by comparing the offset of the log storage, and push the latest front-end logs to the message subscription module.

6. The front-end log collection system according to claim 1, characterized in that, The enterprise log platform includes a format conversion module, which is configured to convert the front-end logs into a user-readable format.

7. The front-end log collection system according to claim 1, characterized in that, The cloud-based log service platform is also configured to receive the front-end logs transmitted by the front-end application through a log transmission method that encapsulates H5 using the Node.js programming language.

8. A front-end log collection method based on the front-end log collection system of claim 1, characterized in that, include: Provide the front-end application with an encryption token for obtaining encrypted front-end logs, and an interface for obtaining the encryption token, which is used for authentication of front-end log transmission; Subscribe to log consumption service from cloud log service platform, listen to the latest front-end logs in cloud log service platform by comparing the log storage offset, and synchronously obtain the front-end logs; The front-end logs are pushed to the Kafka open-source component, production data is consumed from the Kafka open-source component, and the front-end logs are stored.

9. The front-end log collection method according to claim 8, characterized in that, The provision of an encryption token to the front-end application for obtaining encrypted front-end logs includes: The front-end application is provided with a temporary encryption token. The validity of the encryption token is determined. If the encryption token is invalid, it is updated.

10. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a computer, performs the method as described in any one of claims 8 to 9.

Images