Method, device and storage medium for classified downloading of front-end logs

Abstract

The application relates to the field of log burying and downloading, and discloses a front-end log classification downloading method, equipment and a storage medium. The method comprises the following steps: a log collection system collects a DOM event triggered by user behavior to generate behavior JSON data; a data processing coordinator performs data filtering processing on the behavior JSON data according to a preset filtering rule to generate filtered JSON data; based on a preset classification algorithm, the filtered JSON data is subjected to label classification processing to obtain classification JSON constraint data; a storage system performs hierarchical encryption on metadata and content of the classification JSON constraint data to obtain encrypted data; a transmission system receives a log downloading request, and based on the log downloading request, corresponding log data is queried from a persistent queue to obtain requested log data. In the embodiment of the application, log collection, data processing and storage are integrated in the front end, the network delay and data loss are reduced, and the log data classification efficiency is improved by using classification label marking.

Description

Technical Field

[0001] This invention relates to the field of log tracking and downloading, and more particularly to a method, device, and storage medium for classifying and downloading front-end logs. Background Technology

[0002] In modern web applications, log tracking is an important data collection and analysis method. Several common front-end tracking methods are as follows:

[0003] 1. Image Beacon Request: Trigger the reporting by creating an Image object, appending the data to be reported as a URL parameter to the URL of a 1x1 pixel transparent image, and sending a GET request.

[0004] 2. XMLHttpRequest or Fetch API: Use the XMLHttpRequest or Fetch API to send asynchronous requests to report data. You can choose to use the GET or POST method and send the data as the request body or URL parameters.

[0005] 3. `Navigator.sendBeacon()`: The `Navigator.sendBeacon()` method allows data to be sent asynchronously when the page is unloaded. It is typically used to perform a final data upload when the page is closed to ensure that the data is sent successfully.

[0006] 4. WebSocket: Uses the WebSocket protocol to establish a persistent connection with the server and reports real-time event tracking data by sending messages.

[0007] However, existing log tracking technologies are highly dependent on the backend, and traditional solutions rely on backend processing, leading to high network latency and a significant risk of data loss. Furthermore, existing log classification methods are inefficient and cannot adapt to the complex scenarios of modern web applications. Therefore, a new technology is needed to address the technical problems of high network latency in log transmission and low efficiency in log classification in existing log tracking technologies. Summary of the Invention

[0008] The main objective of this invention is to solve the technical problems of high network latency in log transmission, high risk of data loss, and low efficiency in log classification in existing log tracking technologies.

[0009] The first aspect of this invention provides a method for classifying and downloading front-end logs. This method is applied to a front-end log classification and downloading system, which includes: a log collection system, a data processing system, a storage system, and a transmission system. The method for classifying and downloading front-end logs includes:

[0010] The log collection system collects DOM events triggered by user behavior, generates behavior JSON data, and sends the behavior JSON data to the data processing system;

[0011] The data processing system receives the behavior JSON data, performs data filtering processing on the behavior JSON data according to preset filtering rules, and generates filtered JSON data.

[0012] Based on a preset classification algorithm, the filtered JSON data is labeled and classified to obtain classified JSON constraint data, and the classified JSON constraint data is sent to the storage system.

[0013] The storage system receives the categorized JSON constraint data based on a preset real-time queue, and performs layered encryption on the categorized JSON constraint data, including metadata and content, to obtain encrypted data.

[0014] Write the encrypted data into a preset persistent queue;

[0015] The transmission system receives a log download request, retrieves the corresponding log data from the persistent queue based on the log download request, obtains the requested log data, and sends the requested log data to the address corresponding to the log download request.

[0016] Optionally, in a first implementation of the first aspect of the present invention, the transmission system includes: a front-end browser and a download server. The steps of receiving a log download request, querying corresponding log data from the persistent queue based on the log download request to obtain requested log data, and sending the requested log data to the address corresponding to the log download request include:

[0017] The front-end browser receives the log download request and sends the log download request to the download server;

[0018] The download server receives the log download request, generates a query token based on the log download request, and sends the query token to the front-end browser;

[0019] The front-end browser receives the query token and, based on the query token, queries the corresponding log data in the persistent queue to obtain the request log data;

[0020] The request log data is compressed to obtain a download compressed package, and the download compressed package is sent to the download server;

[0021] The download server receives the downloaded compressed package, generates a download link based on the downloaded compressed package, and sends the download link to the address corresponding to the log download request.

[0022] Optionally, in a second implementation of the first aspect of the present invention, the step of sending the download link to the address corresponding to the log download request includes:

[0023] When the number of logs for the download link exceeds a preset batch threshold, the download link is sent to the address corresponding to the log download request based on the HTTP protocol.

[0024] When the number of logs in the download link is not greater than a preset batch threshold, the download link is sent to the address corresponding to the log download request based on the WebSocket protocol.

[0025] Optionally, in a third implementation of the first aspect of the present invention, the steps of compressing the request log data to obtain a download compressed package and sending the download compressed package to the download server include:

[0026] The byte stream of the request log data is compressed into .gz format to generate a gzip download archive;

[0027] The gzip download package is sent to the download server.

[0028] Optionally, in a fourth implementation of the first aspect of the present invention, the step of performing label classification processing on the filtered JSON data based on a preset classification algorithm to obtain classified JSON constraint data includes:

[0029] The filtered JSON data is preprocessed to obtain preprocessed JSON data;

[0030] Based on the KNN classification algorithm, decision tree, or SVM algorithm, the preprocessed JSON data is classified and labeled to obtain labeled JSON data.

[0031] The tagged JSON data is converted into a JSON Schema structure to generate categorized JSON constraint data.

[0032] Optionally, in a fifth implementation of the first aspect of the present invention, the step of performing layered encryption of the metadata and content of the categorized JSON constraint data to obtain encrypted data includes:

[0033] The metadata of the categorized JSON constraint data is encrypted using the Web Crypto API to obtain encrypted content data.

[0034] Based on a preset sensitive dictionary, the sensitive fields of the encrypted content data are replaced with Base64 ciphertext to generate encrypted data.

[0035] Optionally, in a sixth implementation of the first aspect of the present invention, the step of performing layered encryption of the metadata and content of the categorized JSON constraint data to obtain encrypted data further includes:

[0036] The JSON constraint data for classification is processed by removing and converting JSON structure characters to obtain a typed array;

[0037] The typed array is encrypted using the AES-GCM algorithm to generate encrypted data.

[0038] Optionally, in the seventh implementation of the first aspect of the present invention, the step of performing data filtering processing on the behavioral JSON data according to preset filtering rules to generate filtered JSON data includes:

[0039] According to preset filtering rules, empty fields and duplicate log data in the behavior JSON data are filtered to generate filtered JSON data.

[0040] A second aspect of the present invention provides a front-end log classification and downloading device, comprising: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a line; the at least one processor invokes the instructions in the memory to cause the front-end log classification and downloading device to execute the aforementioned front-end log classification and downloading method.

[0041] A third aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the aforementioned method for classifying and downloading front-end logs.

[0042] In this embodiment of the invention, by integrating log tracking at the front end, the entire process of log collection, classification, and download is completed within the front end. Layered encryption using real-time and persistent queues improves log storage speed and security, avoiding network latency and log data loss. When collecting log data at the front end, invalid data is filtered out, and an efficient classification algorithm is introduced to label and classify the log data, adapting to the complex scenarios of modern web applications and improving log data classification efficiency. This invention solves the technical problems of high network latency in log transmission, high risk of data loss, and low efficiency in log classification in existing log tracking technologies. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of an embodiment of the front-end log classification and download method in this invention.

[0044] Figure 2 This is a schematic diagram of the first specific embodiment of the 104 steps of the front-end log classification and download method in this invention;

[0045] Figure 3 This is a schematic diagram of the second specific embodiment of the 104 steps of the front-end log classification and download method in this invention.

[0046] Figure 4 This is a schematic diagram of a specific embodiment of the 106 steps of the front-end log classification and download method in this invention.

[0047] Figure 5 This is a schematic diagram of an embodiment of a front-end log classification and downloading device according to an embodiment of the present invention. Detailed Implementation

[0048] This invention provides a method, device, and storage medium for classifying and downloading front-end logs.

[0049] The embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the accompanying drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0050] In the description of the embodiments disclosed in this invention, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0051] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 This invention provides an embodiment of a method for classifying and downloading front-end logs. The method is applied to a front-end log classification and downloading system, which includes a log collection system, a data processing system, a storage system, and a transmission system. The method for classifying and downloading front-end logs includes:

[0052] 101. The log collection system collects DOM events triggered by user behavior, generates behavior JSON data, and sends the behavior JSON data to the data processing system;

[0053] In this embodiment, the front-end log classification and download system includes: a log collection system, a data processing system, a storage system, and a transmission system. All four systems are integrated into the front-end, adopting a pure front-end architecture design. The log collection system collects user behavior (clicks, scrolling, etc.), performance metrics (loading time, FPS, etc.), and exception information in real time. When a user interacts (click, scroll, keydown, etc.), an Event object is generated. Calling `addEventListener('event name', handler, true)` captures the event. The listener reads the fields, assembles them into JSON format to generate behavioral JSON data, and sends the behavioral JSON data to the data processing system. The log collection interface design can be referenced as follows: `interface BaseLog{`

[0054] timestamp: number; / / Log timestamp

[0055] seasoned: string; / / Session ID

[0056] pageId: string; / / Unique identifier for the page

[0057] userId?: string; / / User ID

[0058] / / Other extended properties}.

[0059] 102. The data processing system receives the behavioral JSON data, performs data filtering processing on the behavioral JSON data according to preset filtering rules, and generates filtered JSON data.

[0060] In this embodiment, the data processing system receives behavioral JSON data, filters out empty fields and redundant characters in non-log content from the behavioral JSON data, and generates filtered JSON data.

[0061] Specifically, step 102 includes the following specific implementation methods:

[0062] 1021. According to the preset filtering rules, the empty fields and duplicate log data of the behavior JSON data are filtered to generate filtered JSON data.

[0063] In step 1021, the preset filtering rules can be set to filter empty fields and duplicate log data in the behavioral JSON data. Based on the set filtering rules, the behavioral JSON data is filtered and processed to generate filtered JSON data.

[0064] 103. Based on a preset classification algorithm, the filtered JSON data is processed by label classification to obtain classified JSON constraint data, and the classified JSON constraint data is sent to the storage system;

[0065] In this embodiment, a classification algorithm with clear mathematical principles from machine learning is first used to classify the filtered JSON data. The classification results of the filtered JSON data are then labeled, and type constraints are added to each data point in the labeled JSON data to generate categorized JSON constraint data. The categorized JSON constraint data is then sent to the storage system for storage.

[0066] Specifically, step 103 includes the following specific implementation methods:

[0067] 1031. The filtered JSON data is preprocessed to obtain preprocessed JSON data;

[0068] 1032. Based on the KNN classification algorithm, decision tree, or SVM algorithm, the preprocessed JSON data is classified and labeled to obtain labeled JSON data;

[0069] 1033. Convert the marked JSON data into a JSON Schema structure to generate categorized JSON constraint data.

[0070] In steps 1031-1033, the filtered JSON data undergoes processes such as filling in missing values ​​and data normalization to obtain preprocessed JSON data. Depending on user settings or the default selection, one of three algorithms—KNN classification, decision tree, or SVM—is chosen to classify and label the preprocessed JSON data, resulting in labeled JSON data. The JSON Schema structure specifies which fields must appear, field types, value ranges, regular expression validation, and field constraints, converting the labeled JSON data into a JSON Schema structure to generate categorical JSON constraint data.

[0071] For example, the original JSON data is as follows:

[0072] {

[0073] "userId": 10086,

[0074] "userName": "alice",

[0075] "vip": true,

[0076] "age": 18,

[0077] "tags": ["js", "node"]

[0078] }

[0079] The original JSON data is converted to the following JSON Schema structure:

[0080] {

[0081] "$schema": "http: / / XXXX.org / XXXX / XXXX#",

[0082] "title": "User Entity",

[0083] "type": "object",

[0084] "required": ["userId", "userName"],

[0085] "additionalProperties": false,

[0086] "properties": {

[0087] "userId":{

[0088] "type": "integer",

[0089] "minimum": 1

[0090] },

[0091] "userName": {

[0092] "type": "string",

[0093] "pattern": "^[a-zA-Z0-9]{3,20}$"

[0094] },

[0095] "vip": {

[0096] "type": "boolean",

[0097] "default": false

[0098] },

[0099] "age": {

[0100] "type": "integer",

[0101] "minimum": 0,

[0102] "maximum": 150

[0103] },

[0104] "tags": {

[0105] "type": "array",

[0106] "items": { "type": "string", "maxLength": 10},

[0107] "maxItems": 5

[0108] }

[0109] }

[0110] }

[0111] Constrain the entire JSON data in terms of content and data type format, and add type constraints and comments to the JSON data.

[0112] 104. The storage system receives the categorized JSON constraint data based on a preset real-time queue, and performs layered encryption on the categorized JSON constraint data, including metadata and content, to obtain encrypted data;

[0113] In this embodiment, the storage system employs a dual-queue design. A real-time queue is used to receive data stored in memory, while the IndexedDB database is connected to a persistent queue. Categorized JSON constraint data is received via the real-time queue. Data input into the persistent queue and stored in the IndexedDB database undergoes layered encryption; both the metadata and content of the categorized JSON constraint data are encrypted separately, resulting in encrypted data.

[0114] For details, please refer to Figure 2 , Figure 2 This is the first specific embodiment of the 104 steps of the front-end log classification and download method in this invention. The 104 steps include the following specific implementation methods:

[0115] 1041. Using the Web Crypto API, the metadata of the categorized JSON constraint data is encrypted to obtain encrypted content data;

[0116] 1042. Based on the preset sensitive dictionary, replace the sensitive fields of the encrypted content data with Base64 ciphertext to generate encrypted data.

[0117] In steps 1041-1042, the Web Crypto API directly implements the underlying encryption through the browser kernel, providing cryptographically secure primitives (hashing, signing, encryption / decryption, key derivation, etc.). JavaScript can only obtain the "handle" and the "result," while front-end scripts cannot access the raw key. The Web Crypto API encrypts the metadata (schema, field names, types, etc.) of the entire JSON string, resulting in encrypted content data. Then, using a "sensitive dictionary," sensitive keywords (such as phone numbers and ID card numbers) appearing in this ciphertext are replaced with Base64-encoded secondary ciphertext, ultimately forming "encrypted data" that can be transmitted externally. This results in a double-encrypted, transmittable string that hides both structural information and sensitive field names.

[0118] For further details, please refer to Figure 3 , Figure 3 This is the second specific embodiment of step 104 of the front-end log classification and download method in this invention. Step 104 includes the following specific implementation methods:

[0119] 1043. Perform JSON structure character removal and conversion processing on the categorized JSON constraint data to obtain a typed array;

[0120] 1044. Using the AES-GCM algorithm, the typed array is encrypted to generate encrypted data.

[0121] In steps 1043-1045, a typed array is a general term for a group of browser-provided "binary arrays" with fixed element types and sizes. It stores data as a continuous block of bytes (within the same ArrayBuffer), so it occupies a fixed amount of memory, reads and writes by byte offset, and its speed is close to that of a C array. It can be directly used by low-level APIs such as WebGL, encryption, files, and networks.

[0122] The JSON constraint data is categorized, and JSON structure characters are removed. It is then binary-compressed and encoded into a typed array (TypedArray). The AES-GCM algorithm is a symmetric encryption algorithm combining encryption and message authentication. It is a variant of the CCM mode and uses a finite-field hash function (GHASH) to provide data integrity verification. The AES-GCM algorithm is used to encrypt the typed array, generating encrypted data. Replacing traditional JSON serialization with a typed array (TypedArray) reduces memory usage by 30% and speeds up front-end log processing.

[0123] 105. Write the encrypted data into a preset persistent queue;

[0124] In this embodiment, encrypted data is written into a preset persistent queue. Since the persistent queue depends on the IndexedDB database, the encrypted data is stored in the IndexedDB database.

[0125] 106. The transmission system receives a log download request, retrieves the corresponding log data from the persistent queue based on the log download request, obtains the requested log data, and sends the requested log data to the address corresponding to the log download request.

[0126] In this embodiment, the transmission system receives a log download request sent by the user from the client login frontend. The system locates the corresponding data in the IndexedDB database using a persistent queue, obtaining the requested log data. Finally, the requested log data is sent back to the corresponding frontend address on the client, enabling the user to access and read the logs. The read logs clearly indicate various events and log types. All of the above processes are integrated at the frontend, resulting in fast transmission speeds, a low probability of log data loss, reduced collection latency, improved classification accuracy, and increased log download success rate.

[0127] For details, please refer to Figure 4 , Figure 4 This is a specific embodiment of step 106 of the front-end log classification and download method in this invention. The transmission system includes a front-end browser and a download server. Step 106 includes the following specific implementation methods:

[0128] 1061. The front-end browser receives the log download request and sends the log download request to the download server;

[0129] 1062. The download server receives the log download request, generates a query token based on the log download request, and sends the query token to the front-end browser;

[0130] 1063. The front-end browser receives the query token, and based on the query token, queries the corresponding log data in the persistent queue to obtain the request log data;

[0131] 1064. Compress the request log data to obtain a download compressed package, and send the download compressed package to the download server;

[0132] 1065. The download server receives the downloaded compressed package, generates a download link based on the downloaded compressed package, and sends the download link to the address corresponding to the log download request.

[0133] In steps 1061-1065, the front-end browser in the transmission system is the user interface. The front-end browser connects to the IndexedDB database on the local NVMe cache disk and the mechanical cold drive, while the download server manages the entire download and transmission process. The front-end browser receives a log download request and sends it to the download server. The download server generates a query token and a query task ID based on the log download request. The front-end browser receives the query token and uses it to query the corresponding log data in the persistent queue to obtain the requested log data. After retrieving the requested log data from the database, the front-end browser compresses the data to obtain a downloadable compressed package, and uploads this package along with the query task ID to the download server. The download server verifies the request object of the downloadable compressed package based on the query task ID and generates a download link. The download link is then sent back to the address corresponding to the query task ID, which is also the address corresponding to the log download request.

[0134] Furthermore, step 1065, "and sending the download link to the address corresponding to the log download request," includes the following specific implementation methods:

[0135] 10651. When the number of logs in the download link exceeds a preset batch threshold, the download link is sent to the address corresponding to the log download request based on the HTTP protocol.

[0136] 10652. When the number of logs in the download link is not greater than the preset batch threshold, the download link is sent to the address corresponding to the log download request based on the WebSocket protocol.

[0137] In steps 10651-10652, it is analyzed whether the download link corresponding to the log download request is for batch download. If the number of logs corresponding to the download link is greater than a preset batch threshold, the download link is sent to the address corresponding to the log download request using the HTTP protocol. When the user triggers the download on the client, the log data of the download link is transmitted to the client using the HTTP protocol.

[0138] If the number of logs corresponding to the download link is not greater than the preset batch threshold, the WebSocket protocol is used to send the download link to the address corresponding to the log download request. When the user triggers the download on the client, the log data of the download link is transmitted to the client using the WebSocket protocol.

[0139] Specifically, step 1064 includes the following specific implementation methods:

[0140] 10641. Compress the byte stream of the request log data into .gz format to generate a gzip download archive;

[0141] 10642. Send the gzip download compressed package to the download server.

[0142] In steps 10641-10642, the byte stream of the requested log data is compressed into .gz format to generate a gzip download archive, which is then uploaded to the download server. The download server only needs to use the standard Content-Encoding: gzip to decompress it. During the interaction between the two, memory usage is reduced by 60% while maintaining a 98% compression rate, improving data transmission efficiency and reducing resource consumption during the interaction process.

[0143] In this embodiment of the invention, by integrating log tracking at the front end, the entire process of log collection, classification, and download is completed within the front end. Layered encryption using real-time and persistent queues improves log storage speed and security, avoiding network latency and log data loss. When collecting log data at the front end, invalid data is filtered out, and an efficient classification algorithm is introduced to label and classify the log data, adapting to the complex scenarios of modern web applications and improving log data classification efficiency. This invention solves the technical problems of high network latency in log transmission, high risk of data loss, and low efficiency in log classification in existing log tracking technologies.

[0144] Figure 5This is a schematic diagram of a front-end log classification and download device 500 provided in an embodiment of the present invention. The front-end log classification and download device 500 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 510 (e.g., one or more processors) and a memory 520, and one or more storage media 530 (e.g., one or more mass storage devices) for storing application programs 533 or data 532. The memory 520 and storage media 530 can be temporary or persistent storage. The program stored in the storage media 530 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the front-end log classification and download device 500. Furthermore, the processor 510 may be configured to communicate with the storage media 530 and execute the series of instruction operations in the storage media 530 on the front-end log classification and download device 500.

[0145] The front-end log-based classification download device 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input / output interfaces 560, and / or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 5 The illustrated front-end log classification download device structure does not constitute a limitation on front-end log-based classification download devices, which may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0146] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when the instructions are executed on a computer, cause the computer to perform the steps of the front-end log classification and download method.

[0147] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0148] Furthermore, although the operations are described in a specific order, this should be understood as requiring that such operations be performed in the specific order shown or in sequential order, or requiring that all illustrated operations be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.

[0149] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for classified download of front-end logs, characterized in that, The front-end log classified downloading method is applied to a front-end log classified downloading system, and the front-end log classified downloading system comprises a log collection system, a data processing system, a storage system and a transmission system. The log collection system collects a DOM event triggered by user behavior, generates behavior JSON data, and sends the behavior JSON data to the data processing system; The data processing system receives the behavior JSON data, performs data filtering processing on the behavior JSON data according to a preset filtering rule, and generates filtered JSON data; Based on a preset classification algorithm, the filtered JSON data is subjected to label classification processing to obtain classified JSON constraint data, and the classified JSON constraint data is sent to the storage system; The storage system receives the classified JSON constraint data based on a preset real-time queue, performs hierarchical encryption of metadata and content on the classified JSON constraint data to obtain encrypted data; The encrypted data is written into a preset persistent queue; The transmission system receives a log download request, queries corresponding log data from the persistent queue based on the log download request, obtains request log data, and sends the request log data to an address corresponding to the log download request; The transmission system comprises a front-end browser and a download server, and the steps of receiving a log download request, querying corresponding log data from the persistent queue based on the log download request, obtaining request log data, and sending the request log data to an address corresponding to the log download request comprise: The front-end browser receives a log download request and sends the log download request to the download server; The download server receives the log download request, generates a query token based on the log download request, and sends the query token to the front-end browser; The front-end browser receives the query token, queries corresponding log data in the persistent queue based on the query token, and obtains request log data; The request log data is subjected to compression processing to obtain a download compressed package, and the download compressed package is sent to the download server; The download server receives the download compressed package, generates a download link based on the download compressed package, and sends the download link to an address corresponding to the log download request.

2. The method of claim 1, wherein, The steps of sending the download link to an address corresponding to the log download request comprise: When the number of logs in the download link is greater than a preset batch threshold, the download link is sent to the address corresponding to the log download request based on an HTTP protocol; When the number of logs in the download link is not greater than the preset batch threshold, the download link is sent to the address corresponding to the log download request based on a WebSocket protocol.

3. The method of claim 2, wherein, The step of compressing the request log data to obtain a download compressed package and sending the download compressed package to the download server comprises: Compressing the byte stream of the request log data into a.gz format to generate a gzip download compressed package; Sending the gzip download compressed package to the download server.

4. The method of claim 1, wherein, The step of performing label classification processing on the filtered JSON data based on a preset classification algorithm to obtain classification JSON constraint data comprises: Preprocessing the filtered JSON data to obtain preprocessed JSON data; Performing classification label processing on the preprocessed JSON data based on a KNN classification algorithm or a decision tree or an SVM algorithm to obtain labeled JSON data; Converting the labeled JSON data into a JSON Schema structure to generate classification JSON constraint data.

5. The method of claim 1, wherein, The step of performing hierarchical encryption of metadata and content on the classification JSON constraint data to obtain encrypted data comprises: Encrypting the metadata of the classification JSON constraint data using a Web Crypto API to obtain content encrypted data; According to a preset sensitive dictionary, replacing sensitive fields of the content encrypted data with Base64 ciphertext to generate encrypted data. 6.The method of claim 1, wherein, The step of performing hierarchical encryption of metadata and content on the classification JSON constraint data to obtain encrypted data further comprises: Performing JSON structure character removal conversion processing on the classification JSON constraint data to obtain a typed array; Using an AES-GCM algorithm to encrypt the typed array to generate encrypted data.

7. The method of claim 1, wherein, The step of performing data filtering processing on the behavior JSON data according to a preset filtering rule to generate filtered JSON data comprises: According to a preset filtering rule, filtering empty fields and duplicate log data of the behavior JSON data to generate filtered JSON data.

8. A device for categorized downloading of front-end logs, characterized by The front-end log classification download device comprises a memory and at least one processor, the memory stores instructions, and the memory and the at least one processor are interconnected through a circuit; The at least one processor calls the instructions in the memory to enable the front-end log classification download device to perform the front-end log classification download method of any one of claims 1-7.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by a processor to implement the front-end log classification download method of any one of claims 1-7.

Images