Log data processing method, device and equipment, and storage medium

Abstract

The application discloses a log data processing method and device, equipment and a storage medium. The method comprises the following steps: acquiring log data generated by a micro server in the process of processing at least one request task; wherein the log data is provided with an execution state identifier; storing the log data under different behavior categories through a preset message queue, and acquiring the log data from the message queue as to-be-processed data; processing the to-be-processed data by using a scheduling thread corresponding to the behavior category of the to-be-processed data, and obtaining to-be-stored log data; and storing and / or reloading processing the to-be-stored log data according to the execution state identifier corresponding to the to-be-stored log data. The embodiment of the application realizes real-time processing of log data, and improves the log data processing efficiency.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a log data processing method, apparatus, device, and storage medium. Background Technology

[0002] The middle platform system is complex and has a large number of services. The system for recording and analyzing fine-grained log data has not yet been fully established, and the speed of log data analysis is slow. If the data cannot be processed quickly, it will be impossible to perform real-time calculations on the requests and responses of each business. Summary of the Invention

[0003] This invention provides a log data processing method, apparatus, device, and storage medium to achieve real-time processing of log data and improve log data processing efficiency.

[0004] According to one aspect of the present invention, a log data processing method is provided, the method comprising:

[0005] Obtain log data generated by the microserver during the processing of at least one request task; wherein the log data carries an execution status identifier;

[0006] Log data for different behavior categories is stored in a preset message queue, and the log data is retrieved from the message queue as data to be processed.

[0007] The data to be processed is processed using a scheduling thread corresponding to the behavior category of the data to be processed, and log data to be stored is obtained.

[0008] Based on the execution status identifier corresponding to the log data to be stored, the log data to be stored is stored and / or reloaded.

[0009] According to another aspect of the present invention, a log data processing apparatus is provided, the apparatus comprising:

[0010] The log data acquisition module is used to acquire log data generated by the microserver during the processing of at least one request task; wherein the log data carries an execution status identifier.

[0011] The pending data acquisition module is used to store log data under different behavior categories through a preset message queue, and to acquire the log data from the message queue as pending data.

[0012] The pending data processing module is used to process the pending data using a scheduling thread corresponding to the behavior category of the pending data, and obtain log data to be stored.

[0013] The storage data processing module is used to store and / or reload the log data to be stored according to the execution status identifier corresponding to the log data to be stored.

[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0015] At least one processor; and

[0016] A memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the log data processing method according to any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the log data processing method according to any embodiment of the present invention.

[0019] This invention employs a scheduling thread corresponding to the behavior category of the data to be processed to process the data and obtain log data to be stored; based on the execution status identifier corresponding to the log data to be stored, the log data to be stored is stored and / or reloaded, thereby realizing real-time processing of log data and improving the processing efficiency of log data.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1A This is a flowchart of a log data processing method provided in Embodiment 1 of the present invention;

[0023] Figure 1B This is a schematic diagram of the structure of a log data processing system according to Embodiment 1 of the present invention;

[0024] Figure 2This is a schematic diagram of the structure of a log data processing device according to Embodiment 2 of the present invention;

[0025] Figure 3 This is a schematic diagram of the structure of an electronic device that implements the log data processing method of this invention. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] Example 1

[0029] Figure 1A This is a flowchart of a log data processing method provided in Embodiment 1 of the present invention. This embodiment is applicable to processing real-time log data under a large amount of microservice architecture. The method can be executed by a log data processing device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1A As shown, the method includes:

[0030] S110. Obtain log data generated by the microserver during the processing of at least one request task; wherein the log data carries an execution status identifier.

[0031] Microservers can be new types of server systems used to process business requests and responses. Typically, during the deployment of microservers, different microservers can perform different business tasks; therefore, a microservice cluster consisting of at least one microserver can be used to handle a large number of business requests.

[0032] During the execution of a request task, a large amount of request and response data can be generated. For example, a request task could be an order task. Log data can include both request and response data.

[0033] The log data includes execution status identifiers, which can represent the status throughout the entire execution cycle of the log data. For example, execution status identifiers include "acquired," "consumed," and "unconsumed." These execution status identifiers may change during different execution processes. For instance, when the log data completes execution in a subsequent step, its corresponding execution status identifier can be updated to "consumed."

[0034] For example, this involves retrieving log data generated by each microserver in a microserver cluster during the execution of request tasks. Different microservers can execute different request tasks, generating different log data.

[0035] It should be noted that in certain scenarios, the log data center acquired by the microserver may contain sensitive data involving personal information. To ensure the security of such log data, it can be encrypted.

[0036] In an optional embodiment, after obtaining the log data generated by the microserver during the processing of at least one request task, the method further includes: identifying sensitive information in the log data; encrypting the sensitive information to obtain encrypted log data.

[0037] Sensitive information in the log data is encrypted to obtain encrypted log data. Optionally, the log data can also be formatted and de-identified to filter the log data.

[0038] S120. Log data for different behavior categories is stored in a preset message queue, and log data is retrieved from the message queue as data to be processed.

[0039] The message queue can be pre-configured by relevant technical personnel according to actual needs. For example, the message queue can be Kafka (a message queue based on a distributed publish-subscribe pattern) or RabbitMQ (Rabbit MessageQueue).

[0040] The different behavior categories include storage behavior and computation behavior. It should be noted that some of the acquired log data is used for subsequent computations, while some is only used for data storage. Accordingly, the log data includes computation log data used for performing computational behaviors and storage log data used for performing storage behaviors.

[0041] For example, the obtained log data can be stored in a pre-defined message queue, and computation log data for performing computational actions and storage log data for performing storage actions can be retrieved from the message queue based on the corresponding scheduling thread. The computation log data and storage log data are then treated as data to be processed.

[0042] S130. Use a scheduling thread corresponding to the behavior category of the data to be processed to process the data and obtain the log data to be stored.

[0043] For example, a scheduling thread can be pre-defined for performing computational operations, and another scheduling thread can be pre-defined for performing storage operations. The acquired data to be processed is then processed based on these two different scheduling threads, and the processed data is used as log data to be stored.

[0044] In one optional embodiment, the behavior categories include storage behavior and computation behavior; correspondingly, a scheduling thread corresponding to the behavior category of the data to be processed is used to process the data to be processed to obtain log data to be stored, including: using a preset first scheduling thread to process the data to be processed that performs storage behavior to obtain first log data to be stored; and using a preset second scheduling thread to process the data to be processed that performs computation behavior to obtain second log data to be stored; generating log data to be stored that includes the first log data to be stored and the second log data to be stored.

[0045] The first and second scheduling threads can be pre-configured by relevant technical personnel according to actual needs. The first scheduling thread is used to process the pending data for performing storage operations; the second scheduling thread is specifically used to process the pending data for performing computational operations.

[0046] For example, a first scheduling thread can be used to process the pending data for storage operations, resulting in first log data to be stored. A second scheduling thread can be used to process the pending data for computation operations, resulting in second log data to be stored.

[0047] It should be noted that during the process of the first scheduling thread processing the data to be processed for storage, a write operation can be performed based on a preset thread pool.

[0048] In one optional embodiment, a preset first scheduling thread is used to process the pending data for performing storage actions to obtain first log data to be stored, including: constructing a thread pool for processing the pending data for performing storage actions; using the first scheduling thread to transfer the pending data for performing storage actions to the thread pool, and processing the pending data for performing storage actions in the thread pool to obtain the first log data to be stored.

[0049] Thread pools are a form of multi-threaded processing. During processing, tasks can be added to a queue, and these tasks are automatically started after threads are created. The purpose of using thread pools is to flexibly control the number of threads according to system requirements and hardware environment, and to provide unified management and control over all threads, thereby improving system efficiency and reducing system load.

[0050] For example, a thread pool for processing pending data to perform storage actions can be pre-built by relevant technical personnel. The pending data for performing storage actions is then transferred to the threads, where it is processed to obtain the first log data to be stored.

[0051] Understandably, in order to accurately construct the thread pool for processing the data to be processed and to avoid wasting space, the following methods can also be used to construct the thread pool.

[0052] In one optional embodiment, a thread pool for processing pending data for performing storage actions is constructed, including: constructing a thread pool for processing pending data for performing storage actions based on the task volume and task time of each request task.

[0053] A thread pool can be constructed by determining the core number of threads, the size of the blocking queue, and the maximum number of threads.

[0054] For example, the core thread count can be determined in the following way:

[0055]

[0056] Here, `tasks` represents the number of requests per second. For example, 100 business tasks are executed per second. `taskcost` represents the time consumed by each request task. For example, executing a query task takes 0.1 seconds.

[0057] For example, the blocking queue size, queueCapacity, can be determined in the following way:

[0058]

[0059] Here, `coreSizePool` represents the core thread pool size. For example, it can be understood as how many threads are used to process request tasks; `responsetime` represents the maximum acceptable response time of the system, for example, the maximum time for executing a query task is 1 second.

[0060] For example, the maximum number of threads, maxPoolSize, can be determined in the following way:

[0061]

[0062] Where max(tasks) represents the maximum number of tasks.

[0063] In one optional embodiment, a preset second scheduling thread is used to process the data to be processed for the execution of the operation to obtain second log data to be stored, including: using the second scheduling thread to perform stream processing on the data to be processed for the execution of the operation based on a preset model to obtain second log data to be stored; and updating the execution status identifier of the second log data to be stored according to the processing result obtained from processing the data to be processed for the execution of the operation.

[0064] The preset model can be a model used to perform computations on the data to be processed. For example, the preset model can be the Flink distributed processing engine and framework. Flink has high throughput and low latency, enabling it to process large amounts of data quickly. Furthermore, Flink supports different restart strategies, so the restart process can be controlled in the event of a failure.

[0065] For example, the Flink framework can perform stream or batch processing on the pending data for executing computational actions to obtain second log data to be stored. Based on the processing results of the pending data, the execution status flag of the second log data to be stored is updated. For instance, if the pending data has been successfully executed, its execution status flag can be updated to "consumed"; if the pending data has not been successfully executed, its execution status flag can be updated to "unconsumed".

[0066] S140. Based on the execution status identifier corresponding to the log data to be stored, perform storage and / or reload processing on the log data to be stored.

[0067] For example, if the execution status identifier corresponding to the log data to be stored is "consumed", then the log data to be stored is stored; if the execution status identifier corresponding to the log data to be stored is "not consumed", then the log data to be stored is reloaded.

[0068] It is understandable that once the pending data is processed in the thread pool, it disappears from the thread pool. At this point, the disappeared pending data can be identified as log data that has been processed.

[0069] In an optional embodiment, after obtaining the first log data to be stored, the method further includes: updating the execution status identifier of the first log data to be stored according to the processing result of the thread pool on the data to be processed for storage; the execution status identifier includes an unconsumed identifier and a consumed identifier; correspondingly, based on the execution status identifier corresponding to the log data to be stored, the log data to be stored is stored and / or reloaded, including: if the execution status identifier corresponding to the first log data to be stored is a consumed identifier, then the first log data to be stored is stored; if the execution status identifier corresponding to the first log data to be stored is an unconsumed identifier, then the first log data to be stored is reloaded.

[0070] For example, if the thread pool's processing result for the pending data to be stored is that the data has been processed successfully or the data has been processed completely, then the execution status flag of the first log data to be stored is updated to "consumed". If the thread pool's processing result for the pending data to be stored is that the data has not been processed successfully or the data processing has failed, then the execution status flag of the first log data to be stored is updated to "unconsumed". If the execution status flag corresponding to the first log data to be stored is "consumed", then the first log data to be stored is stored; if the execution status flag corresponding to the first log data to be stored is "unconsumed", then the first log data to be stored is reloaded.

[0071] This invention employs a scheduling thread corresponding to the behavior category of the data to be processed to process the data and obtain log data to be stored; based on the execution status identifier corresponding to the log data to be stored, the log data to be stored is stored and / or reloaded, thereby realizing real-time processing of log data and improving the processing efficiency of log data.

[0072] In an optional embodiment, the present invention also provides a log data processing system for executing any of the log data processing methods described in this embodiment. For example... Figure 1B The diagram shows the structure of a log data processing system.

[0073] A log data processing system may include a control layer for data filtering, an access layer for data storage and transmission, a computation layer for data calculation, a storage layer for data storage, a service layer for data integration services, and an application layer for data applications.

[0074] For example, microservers A, B, ..., N in a microservice cluster uniformly process the input and output data of request tasks. Request and response data are treated as log data, and the control layer performs filtering operations such as formatting and anonymization on sensitive information in the log data using regular expressions. A real-time data bus is used for reading and writing log data to prevent blocking. Kafka or RabbitMQ can be used as the real-time data bus. For log data that only needs to be stored, a single scheduling thread is used for reading to avoid dirty reads and duplicate reads. A thread pool is used for writing to the database. The core number of threads, the size of the blocking queue, and the maximum number of threads in the thread pool are determined, and the thread pool is constructed based on these parameters. A task status record table is set up to mark the execution status of each piece of data read from the real-time data bus, preventing data loss due to unexecuted tasks in the thread pool queue during a crash. For log information requiring complex calculations, Flink is used for computation. Flink supports different restart strategies, so the restart method can be controlled in the event of a failure. For both persistent and real-time data, various query services can be provided through the service layer to quickly locate and proactively identify problems. Optionally, log data can be applied to at least one application service in the application layer.

[0075] Example 2

[0076] Figure 2 This is a schematic diagram of a log data processing device provided in Embodiment 2 of the present invention. The log data processing device provided in this embodiment of the present invention is suitable for processing large amounts of real-time log data under a microservice architecture. This log data processing device can be implemented in hardware and / or software, such as... Figure 2 As shown, the device specifically includes: a log data acquisition module 201, a data to be processed acquisition module 202, a data to be processed processing module 403, and a data storage processing module 204. Among them,

[0077] The log data acquisition module 201 is used to acquire log data generated by the microserver during the processing of at least one request task; wherein the log data carries an execution status identifier.

[0078] The pending data acquisition module 202 is used to store log data under different behavior categories through a preset message queue, and to acquire the log data from the message queue as pending data.

[0079] The pending data processing module 203 is used to process the pending data using a scheduling thread corresponding to the behavior category of the pending data, and obtain log data to be stored.

[0080] The storage data processing module 204 is used to store and / or reload the log data to be stored according to the execution status identifier corresponding to the log data to be stored.

[0081] This invention employs a scheduling thread corresponding to the behavior category of the data to be processed to process the data and obtain log data to be stored; based on the execution status identifier corresponding to the log data to be stored, the log data to be stored is stored and / or reloaded, thereby realizing real-time processing of log data and improving the processing efficiency of log data.

[0082] Optionally, the behavior categories include storage behavior and computation behavior;

[0083] Accordingly, the data processing module 203 includes:

[0084] The first storage data determination unit is configured to process the pending data for performing the storage action using a preset first scheduling thread to obtain first log data to be stored; and

[0085] The second storage data determination unit is used to process the data to be processed for performing the operation using a preset second scheduling thread, so as to obtain the second log data to be stored.

[0086] The data processing unit is used to generate log data to be stored, which includes the first log data to be stored and the second log data to be stored.

[0087] Optionally, the first stored data determining unit includes:

[0088] The thread pool construction subunit is used to construct a thread pool for processing the pending data of the execution storage behavior;

[0089] The first storage data determination subunit is used to use the first scheduling thread to transfer the pending data of the execution storage behavior to the thread pool, and process the pending data of the execution storage behavior in the thread pool to obtain the first log data to be stored.

[0090] Optionally, the thread pool construction subunit is specifically used for:

[0091] Based on the task volume and task time of each request task, a thread pool is constructed to process the pending data for the execution storage behavior.

[0092] Optionally, the first stored data determining unit further includes:

[0093] The identifier update subunit is used to update the execution status identifier of the first log data to be stored based on the processing result of the thread pool on the data to be processed for the execution storage behavior; the execution status identifier includes an unconsumed identifier and a consumed identifier;

[0094] Accordingly, the storage data processing module includes:

[0095] The storage processing unit is configured to perform storage processing on the first log data to be stored if the execution status identifier corresponding to the first log data to be stored is a consumed identifier.

[0096] The reloading unit is used to reload the first log data to be stored if the execution status identifier corresponding to the first log data to be stored is an unconsumed identifier.

[0097] Optionally, the second stored data determining unit includes:

[0098] The second storage data determination subunit is used to perform stream processing on the data to be processed that performs the operation based on the second scheduling thread and a preset model to obtain the second log data to be stored.

[0099] The second storage data determination subunit is used to update the execution status identifier of the second log data to be stored based on the processing result obtained by processing the data to be processed to perform the operation.

[0100] Optionally, the device further includes:

[0101] The sensitive information identification module is used to identify sensitive information in the log data after the microserver has obtained the log data generated during the processing of at least one request task;

[0102] The information encryption module is used to encrypt the sensitive information to obtain encrypted log data.

[0103] The log data processing apparatus provided in the embodiments of the present invention can execute the log data processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0104] Example 3

[0105] Figure 3A schematic diagram of an electronic device 30 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0106] like Figure 3 As shown, the electronic device 30 includes at least one processor 31 and a memory, such as a read-only memory (ROM) 32 or a random access memory (RAM) 33, communicatively connected to the at least one processor 31. The memory stores computer programs executable by the at least one processor. The processor 31 can perform various appropriate actions and processes based on the computer program stored in the ROM 32 or loaded from storage unit 38 into the RAM 33. The RAM 33 can also store various programs and data required for the operation of the electronic device 30. The processor 31, ROM 32, and RAM 33 are interconnected via a bus 34. An input / output (I / O) interface 35 is also connected to the bus 34.

[0107] Multiple components in electronic device 30 are connected to I / O interface 35, including: input unit 36, such as keyboard, mouse, etc.; output unit 37, such as various types of monitors, speakers, etc.; storage unit 38, such as disk, optical disk, etc.; and communication unit 39, such as network card, modem, wireless transceiver, etc. Communication unit 39 allows electronic device 30 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0108] Processor 31 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 31 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 31 performs the various methods and processes described above, such as log data processing methods.

[0109] In some embodiments, the log data processing method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 38. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 30 via ROM 32 and / or communication unit 39. When the computer program is loaded into RAM 33 and executed by processor 31, one or more steps of the log data processing method described above may be performed. Alternatively, in other embodiments, processor 31 may be configured to perform the log data processing method by any other suitable means (e.g., by means of firmware).

[0110] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0111] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0112] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0113] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0114] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0115] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0116] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0117] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A log data processing method, characterized in that, include: Obtain log data generated by the microserver during the processing of at least one request task; wherein the log data carries an execution status identifier; Log data for different behavior categories is stored in a preset message queue, and the log data is retrieved from the message queue as data to be processed; the behavior categories include storage behavior and computation behavior. Build a thread pool to process the pending data that performs storage actions; The first scheduling thread is used to transfer the pending data of the storage behavior to the thread pool, and the pending data of the storage behavior is processed in the thread pool to obtain the first log data to be stored; Based on the processing result of the thread pool on the pending data of the execution storage behavior, update the execution status identifier of the first pending log data; the execution status identifier includes an unconsumed identifier and a consumed identifier; and, A second scheduling thread is used to perform stream processing on the data to be processed, based on a preset model, to obtain second log data to be stored; the preset model is a model used for processing the data to be processed. Update the execution status identifier of the second log data to be stored based on the processing result obtained from processing the data to be processed to perform the above operation. Generate log data to be stored, including the first log data to be stored and the second log data to be stored; Based on the execution status identifier corresponding to the log data to be stored, the log data to be stored is stored and / or reloaded.

2. The method according to claim 1, characterized in that, The thread pool for constructing the pending data for performing the storage behavior includes: Based on the task volume and task time of each request task, a thread pool is constructed to process the pending data for the execution storage behavior.

3. The method according to claim 1, characterized in that, The step of storing and / or reloading the log data to be stored according to the execution status identifier corresponding to the log data to be stored includes: If the execution status identifier corresponding to the first log data to be stored is a consumed identifier, then the first log data to be stored is stored. If the execution status identifier corresponding to the first log data to be stored is an unconsumed identifier, then the first log data to be stored will be reloaded.

4. The method according to any one of claims 1-3, characterized in that, After obtaining the log data generated by the microserver during the processing of at least one request task, the method further includes: Identify sensitive information in the log data; The sensitive information is encrypted to obtain encrypted log data.

5. A log data processing device, characterized in that, include: The log data acquisition module is used to acquire log data generated by the microserver during the processing of at least one request task; wherein the log data carries an execution status identifier. The pending data acquisition module is used to store log data under different behavior categories through a preset message queue, and to acquire the log data from the message queue as pending data. The pending data processing module is used to process the pending data using a scheduling thread corresponding to the behavior category of the pending data, and obtain log data to be stored. The data storage processing module is used to store and / or reload the log data to be stored according to the execution status identifier corresponding to the log data to be stored. The behavior categories include storage behavior and computation behavior; Accordingly, the data processing module includes: The first storage data determination unit is configured to process the pending data for performing the storage action using a preset first scheduling thread to obtain first log data to be stored; and The second storage data determination unit is used to process the data to be processed for performing the operation using a preset second scheduling thread, so as to obtain the second log data to be stored. A data processing unit is used to generate log data to be stored, including the first log data to be stored and the second log data to be stored. The first stored data determining unit includes: The thread pool construction subunit is used to construct a thread pool for processing the pending data of the execution storage behavior; The first storage data determination subunit is used to use the first scheduling thread to transfer the pending data of the storage behavior to the thread pool, and process the pending data of the storage behavior in the thread pool to obtain the first log data to be stored. The identifier update subunit is used to update the execution status identifier of the first log data to be stored based on the processing result of the thread pool on the data to be processed for the execution storage behavior; the execution status identifier includes an unconsumed identifier and a consumed identifier; The second stored data determination unit includes: The second storage data determination subunit is used to perform stream processing on the data to be processed for performing the operation, based on the second scheduling thread and a preset model, to obtain the second log data to be stored; the preset model is a model used for the operation processing of the data to be processed. The second identifier update subunit is used to update the execution status identifier of the second log data to be stored based on the processing result obtained by processing the data to be processed to perform the operation.

6. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the log data processing method according to any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the log data processing method according to any one of claims 1-4.

Images