Log data processing method and log storage engine
By storing log data in time-sequential blocks on a cloud computing platform and using an LSM database, the problem of increased log system costs was solved, efficient log data querying and sorting were achieved, and storage and indexing time was reduced.
Patent Information
- Application Number
- CN202311113231.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-05-10
- Filing Date
- 2023-08-31
- Publication Date
- 2024-11-12
AI Technical Summary
In cloud computing scenarios, existing technologies require additional copying of log data to Kafka, which increases the cost of the log system.
Log data is stored in time-order blocks using a log storage engine, and LSM databases such as LevelDB, ElasticSearch, and HBase are used to reduce the storage cost of log data. Timestamps and time offsets are used for fast location and sorting.
It reduces the storage and indexing time of log services, reduces the cost required to provide log services, and improves the efficiency of log data querying.
Smart Images

Figure CN118939624A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of cloud computing, and in particular to a log data processing method and a log storage engine. Background Art
[0002] In cloud computing scenarios, the log data of hosts and cloud services can be managed by the log system. That is, the log system is responsible for collecting and storing the log data of hosts and cloud services, and providing the collected log data to users, so that users can perform decision analysis, equipment operation and maintenance management, business trend analysis, security audits, etc. based on these log data.
[0003] In the log system, Kafka can be deployed to provide log data to users. However, this method requires the log system to not only store log data, but also to copy a copy of the log data to Kafka, which increases the cost of the log system. Summary of the invention
[0004] The present application provides a log data processing method and a log storage engine, which can reduce the cost of providing log services.
[0005] In the first aspect, the present application provides a method for processing log data. A cloud computing platform provides a log service, which is used to manage the log data of an application. The cloud computing platform includes a server on which a log collection device is deployed, and the log collection device is used to collect the log data of the application running on the server. The user obtains or queries the log data of the application through the log service provided by the cloud computing platform. The method can be applied to a log storage engine. The log storage engine obtains multiple log data of the application from the log collection device, and then stores the multiple log data in blocks in chronological order according to the timestamps of the multiple log data. Wherein, block storage refers to storing the timestamp of the first log included in the log block and the time offset of other logs included in the log block, and the log block includes at least two log data of the multiple log data. Afterwards, the log storage engine receives a log provision request from the user, obtains multiple log data from at least one log block according to the time or time interval specified by the log provision request, and provides the obtained multiple log data to the user in chronological order.
[0006] In the technical solution provided by the present application, the log storage engine can store multiple log data of the application in chronological order, so when providing the user with log data sorted in chronological order, the time consumed by indexing can be reduced, thereby reducing the cost required for providing log services. In addition, since the length of the time offset of the log data is less than the length of the timestamp of the log data, the technical solution provided by the present application can also reduce the cost of providing log services.
[0007] In a possible implementation of the first aspect, the log block further stores any one or more of the following: the timestamp of the last log data included in the log block, and the number of log data included in the log block. Through the above implementation, the log storage engine can determine the timestamp and number of log data in each log block, so that the log storage engine can quickly locate the log data required by the user.
[0008] In a possible implementation of the first aspect, the log storage engine stores multiple log data in blocks according to the time sequence, including: the log storage engine stores multiple log data in blocks according to the time sequence into a log-structured merge tree (LSM) database. LSM databases include, for example, LevelDB, ElasticSearch, Hbase, etc. In other words, the technical solution provided by the present application has good applicability for backend databases.
[0009] In a possible implementation of the first aspect, the timestamp of any log data is the time when the log collection device collects the log data (hereinafter referred to as the collection time) or the coded value of the collection time. The length of the coded value of the collection time is less than the length of the collection time, thereby further saving the storage cost of the log data.
[0010] In the second aspect, the present application provides a log storage engine. The cloud computing platform provides a log service, which is used to manage the log data of the application. The cloud computing platform includes a server on which a log collection device is deployed, and the log collection device is used to collect the log data of the application running on the server. The user obtains or queries the log data of the application through the log service provided by the cloud computing platform. The log storage engine includes an acquisition module and a streaming storage module. The acquisition module is used to obtain multiple log data of the application from the log collection device, and receive the user's log provision request. The streaming storage module is used to block store the multiple log data in chronological order according to the timestamps of the multiple log data, and obtain multiple log data from at least one log block according to the time or time interval specified by the log provision request, and provide the obtained multiple log data to the user in chronological order. Among them, block storage refers to storing the timestamp of the first log included in the log block and the time offset of other logs included in the log block, and the log block includes at least two log data of the multiple log data.
[0011] In a possible implementation manner of the second aspect, the log block further stores any one or more of the following items: a timestamp of the last log data included in the log block, and the amount of log data included in the log block.
[0012] In a possible implementation manner of the second aspect, the streaming storage module is used to store multiple log data in blocks in chronological order into the LSM database.
[0013] In a possible implementation of the second aspect, the timestamp of any log data is the time when the log collection device collects the log data (ie, the collection time) or a coded value of the collection time, wherein the length of the coded value of the collection time is less than the length of the collection time.
[0014] In the third aspect, the present application provides a log system. The log system is used to manage the log data of the application, and the user obtains or queries the log data of the application through the log system. The log system includes a log collection device, a log storage engine and an LSM database. The log collection device is used to collect multiple log data of the application running on the server. The log storage engine is used to obtain multiple log data of the application, and according to the timestamps of the multiple log data of the application, the multiple log data of the application are stored in blocks in chronological order. Block storage refers to storing the timestamp of the first log data included in the log block and the time offset of other log data included in the log block. The log block includes at least two log data of the multiple log data. In addition, the log storage engine is also used to receive the user's log provision request, obtain multiple log data from at least one log block according to the time or time interval specified by the log provision request, and provide the obtained multiple log data to the user in chronological order. The LSM database is used to store the above-mentioned at least one log block.
[0015] In a fourth aspect, the present application provides a computing device. The computing device includes a processor and a memory. The processor is used to execute instructions stored in the memory to implement part or all of the method described in the first aspect and any implementation thereof.
[0016] In a fifth aspect, the present application provides a computing device cluster. The computing device cluster includes at least one computing device, and each computing device includes a processor and a memory. The processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster performs part or all of the method described in the first aspect and any implementation thereof.
[0017] In a sixth aspect, the present application provides a computer program product. The computer program product may be software or a program product that includes instructions and can be run on a computing device or stored in any available medium. When the computer program product is run on a computing device or a computing device cluster, the computing device or the computing device cluster executes part or all of the method described in the first aspect and any implementation thereof.
[0018] In a seventh aspect, the present application provides a computer-readable storage medium. The computer storage medium includes computer program instructions, and when the computer program instructions are executed by a computing device or a computing device cluster, the computing device or the computing device cluster executes part or all of the method described in the first aspect and any implementation thereof. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 It is a structural diagram of a log system provided in an embodiment of the present application;
[0020] Figure 2 It is a schematic diagram of an application scenario provided by an embodiment of the present application;
[0021] Figure 3 It is a flowchart of a method for processing log data provided by an embodiment of the present application;
[0022] Figure 4 is a schematic diagram of a log data query method provided in an embodiment of the present application;
[0023] Figure 5 It is a structural diagram of a log storage engine provided by an embodiment of the present application;
[0024] Figure 6 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;
[0025] Figure 7 It is a structural diagram of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION
[0026] The technical solution provided by the present application is described below in conjunction with the accompanying drawings.
[0027] Network equipment, operating systems, service programs and other software and hardware will generate log data when they are running. Log data records the time, user, specific operation and other related descriptions of the above events, so it plays an important role in program debugging, fault location and other aspects. Especially in cloud computing scenarios, by analyzing massive log data, it can help users (for example, enterprises) to efficiently perform decision analysis, equipment operation and maintenance management, business trend analysis, security audit and other tasks. It should be understood that in actual applications, the log data of the host and cloud services are not convenient for users to view and will be cleared regularly, so cloud service providers usually provide log services to users. Log services can be implemented by a log system, which can specifically include: the log system collects log data, stores the collected log data, and provides log data to users, so that log data can flow from the cloud to the enterprise or customer business system under the cloud, which facilitates the subsequent secondary processing of log data such as archiving, deep mining, and correlation analysis, and helps users perform related decision analysis, equipment operation and maintenance management, business trend analysis, security audit and other related tasks.
[0028] At present, under the general trend of enterprise digital transformation, higher requirements are put forward for log services, including: supporting users to consume log data of any time period, supporting the provision of log data sorted in chronological order to users, and minimizing the computing resources and storage resources occupied by the service. In response to the above requirements, the embodiment of the present application provides a method for processing log data. This method can be applied to the log storage engine. Figure 1 Describe it.
[0029] like Figure 1 As shown, the log system 100 includes a log collection device 110, a log storage engine 120 and an LSM database 130. Optionally, the log system 100 may further include a log analysis device 140.
[0030] The log collection device 110 is used to collect multiple log data of applications running on the server, and send the collected multiple log data to the log storage engine 120. As an example, the log collection device 110 can be deployed on a server running an application, for example, an ICAgent, Logstash, Fluentd, etc., which has a log data collection function.
[0031] The log storage engine 120 is used to receive multiple log data sent by the log collection device 110, and store the multiple log data in blocks in chronological order according to the timestamps of the multiple log data. The log storage engine 120 also provides an access interface, such as an application programming interface (application programming interface), a graphical user interface (graphical user interface, GUI), etc., through which a user's log provision request can be received, and then multiple log data are obtained from at least one log block according to the time or time interval specified in the log provision request, and then the multiple log data obtained above are provided to the user in chronological order through the access interface.
[0032] The LSM database 130 refers to a database that uses LSM as the underlying storage model, such as LevelDB, ElasticSearch, and HBase. The LSM database 130 supports access by the log storage engine 120. For example, the log storage engine 120 can store multiple log data of an application in blocks in chronological order to the LSM database 130, and can also read the log data required by the user from the LSM database 130.
[0033] The log analysis device 140 is used to obtain log data from the LSM database 130, and perform processing such as archiving, deep mining, and correlation analysis on the obtained log data, and return the corresponding processing results to the user.
[0034] In the embodiment of the present application, the deployment of the log system 100 is flexible. For example, the log system 100 can be deployed in a data center of a cloud environment. The data center of the cloud environment includes a large number of basic resources (including computing resources, storage resources, and network resources). The computing resources in the data center of the cloud environment can be a large number of computing devices (for example, servers). In a specific implementation, the log system 100 can be independently deployed on a server or a virtual machine in a data center of a cloud environment, or distributedly deployed on multiple servers or multiple virtual machines in a data center of a cloud environment, or distributedly deployed on servers and virtual machines in a data center of a cloud environment. Figure 2As shown, the log system 100 is deployed on a cloud environment, and the cloud service provider provides the functions of the log system 100 as a cloud service (i.e., log service) to the user. Accordingly, the user registers an account on the cloud management platform, logs in to the cloud management platform through the account, and purchases the above-mentioned cloud service on the cloud management platform (for example, it can be pre-charged and then settled according to the final resource usage, or purchased on a yearly or monthly basis). After the purchase is successful, the log collection device 110 will collect multiple log data of the application, and send the collected multiple log data to the log storage engine 120. Accordingly, the log storage engine 120 stores these log data in blocks to the LSM database 130 according to the timestamps of the multiple log data. Further, the log storage engine 120 can also receive a log provision request input by the user, for example, push the log data generated in a certain period of time in chronological order, or index the log data generated in a certain period of time according to the time. Afterwards, the log storage engine 120 provides the user with multiple log data sorted in chronological order according to the log provision request. In addition, the log analysis device 140 can also obtain multiple log data of the application from the LSM database 130, analyze the obtained multiple log data, and then return the analysis results to the user, so that the user can perform relevant analysis for business decisions based on the analysis results.
[0035] For another example, the log system 100 can also be deployed on a terminal computing device, and the terminal computing device includes a terminal server, a smart phone, a notebook computer, a tablet computer, a personal desktop computer, etc. For another example, the log system 100 can also be distributedly deployed in different environments (i.e., a data center and a terminal computing device of a cloud environment), for example, the log collection device 110 is deployed on a terminal computing device, and the log storage engine 120 and the LSM database 130 are deployed on the same computing device in the cloud environment. The various parts of the log system 100 on different environments collaborate to provide the functions that the log system 100 has. It should be understood that the embodiment of the present application does not limit the division of the various parts of the log system 100, nor does it limit the deployment mode of the various parts of the log system 100. When actually applied, it can be adaptively deployed according to the computing power of specific application requirements or hardware equipment.
[0036] Combine the following Figure 3 , further describing how the log storage engine 120 implements log data processing.
[0037] S101: the log storage engine 120 obtains multiple log data of the application from the log collection device 110.
[0038] Specifically, the log collection device 110 collects multiple log data of the application, and then sends the multiple log data to the log storage engine 120. Accordingly, the log storage engine 120 receives the multiple log data sent by the log collection device 110.
[0039] S102: The log storage engine 120 stores the multiple log data in blocks in chronological order according to the timestamps of the multiple log data.
[0040] The timestamp of the log data may be the time when the log collection device 110 collects the log data (i.e., the collection time), or a code of the collection time. Alternatively, if the log data is generated by the user's program code during operation, the timestamp of the log data may also be the time set by the user (hereinafter referred to as the preset time) or a code of the preset time. In a specific implementation, the collection time or the preset time of the log data may be encoded using an existing encoding method in the industry that can reduce the length of the character string (e.g., a varint encoding method). It should be understood that since the length of the encoding of the collection time is less than the length of the collection time, and the length of the encoding of the preset time is less than the length of the preset time, compared to directly using the collection time or the preset time as the timestamp of the log data, using the encoding of the collection time or the preset time as the timestamp of the log data can reduce the occupancy of storage resources.
[0041] In some embodiments, the timestamp of the log data may be added by the log storage engine 120. For example, the log storage engine 120 obtains the collection time of the log data from the log collection device 110, and then adds a corresponding timestamp to the log data according to the collection time. The timestamp of the log data may also be added by the log collection device 110.
[0042] Block storage refers to storing one or more of the following: the timestamp of the first log data included in the log block, the timestamp of the last log data included in the log block, the time offset of other log data included in the log block, and the number of log data included in the log block. Among them, the time offset of other log data refers to the difference between the timestamp of other log data and the timestamp of the first log data. Moreover, the length of the time offset of the log data is less than the length of the timestamp of the log data, thereby saving the storage cost of the log data. For example, suppose that a log block includes M log data, and these M log data include log data A and log data B, and log data A is the first log data (that is, the timestamp of log data A is the smallest). The collection time of log data A is 2023-04-01 22:03:36, and the collection time of log data B is 2023-04-01 22:03:47. The log storage engine 120 uses varint encoding to encode the collection time of log data A and the collection time of log data B respectively, and the obtained character strings are 1680357816000000000 and 1680357827000000000 respectively. Then, the timestamp of log data A is 1680357816000000000 (occupies 8 bytes), the timestamp of log data B is 1680357827000000000 (occupies 8 bytes), and the time offset value corresponding to log data B is 11000000000 (occupies 4 bytes). It can be seen that the length of the time offset value corresponding to log data B is less than the length of the timestamp of log data B. Therefore, storing the time offset of log data B can reduce the occupation of storage resources.
[0043] In some embodiments, the log storage engine 120 stores multiple log data in blocks in chronological order according to the timestamps of the multiple log data, including: the log storage engine 120 generates at least one log block according to the timestamps of the multiple log data. Each log block includes metadata and at least one log data of the multiple log data, and the metadata of the log block includes one or more of the following: the timestamp of the first log data included in the log block, the timestamp of the last log data included in the log block, the time offset of other log data included in the log block, and the number of log data included in the log block. Then, the log storage engine 120 stores the at least one log block in chronological order according to the metadata of the at least one log block.
[0044] Depend on Figure 1It can be seen that the log system 100 includes an LSM database 130, and the LSM database 130 is used to store log data. Therefore, the log storage engine 120 can store the above-mentioned multiple log data in blocks in chronological order to the LSM database 130, that is, the log storage engine 120 sends the above-mentioned at least one log block to the LSM database 130 in chronological order according to the metadata in the at least one log block. Correspondingly, after the LSM database 130 receives the at least one log block sent by the log storage engine 120, it establishes an index according to the metadata in the at least one log block, and stores the above-mentioned at least one log block in chronological order. It is worth noting that the LSM database 130 in the embodiment of the present application itself includes a database storage engine, so the log storage engine 120 can be understood as an extension of the database storage engine in the LSM database 130.
[0045] S103: The log storage engine 120 receives a log providing request from a user, and provides corresponding log data to the user according to the log providing request.
[0046] Specifically, the log storage engine 120 receives a user's log provision request, then obtains multiple log data from the at least one log block according to the time or time interval specified in the log provision request, and provides the obtained multiple log data to the user in chronological order.
[0047] In some embodiments, the log storage engine 120 obtains multiple log data from the at least one log block, including: the log storage engine 120 obtains multiple log data indicated by the log provision request from the LSM database 130.
[0048] In some other embodiments, as can be seen from the above S102, the log storage engine 120 caches the at least one log block before storing the at least one log block in the LSM database 130. Therefore, the log storage engine 120 obtains multiple log data from the at least one log block, including: the log storage engine 120 obtains multiple log data indicated by the log provision request from the cache.
[0049] In some embodiments, the log provision request can be used to instruct the log storage engine 120 to actively push log data of a specified time or time interval to the user, or can be used to instruct the log storage engine 120 to query log data of a specified time or time interval and return the queried log data to the user.
[0050] For example, Figure 4As shown, the LSM database 130 stores multiple log blocks, for example, log block 1, log block 2, log block 3, ..., log block 6. The timestamp of the first log data of log block 3 is 1679980224000000000, and the timestamp of the last log data is 1680671424000000000. The user's log provision request is to query 100 log data starting from 2023-04-01 10:11:24. After receiving the log provision request, the log storage engine 120 encodes "2023-04-0110:11:24" using varint encoding, and obtains the corresponding timestamp 1680315084000000000. Then, the log storage engine 120 indexes the LSM database 130 according to the timestamp, determines the log block 3, and then determines the log data 2 in the log block 3 by binary search, and sequentially reads 100 log data (including log data 2) starting from log data 2, and then returns these 100 log data to the user.
[0051] It should be understood that in actual applications, the number of files, the number of records in a single file, and the maximum number of single pages are less than or equal to a certain constant, so the time complexity of a user querying log data through the log storage engine 120 is O(1). In other words, the technical solution provided by the present application can reduce the time consumed by indexing, so that users can quickly query the required log data.
[0052] Combined with the previous article Figure 3-Figure 4 The functions of the log storage engine 120 are described in detail below. Figure 5 , the log storage engine 120 is described from the structure of the log storage engine 120.
[0053] Figure 5 The structural diagram of the log storage engine 120 is shown as an example. Figure 5 As shown, the log storage engine 120 includes an acquisition module 121 and a streaming storage module 122. The acquisition module 121 and the streaming storage module 122 work together to implement the steps performed by the log storage engine 120 in the above method embodiment. Specifically, the acquisition module 121 is used to execute the above S101, the steps of adding timestamps to log data in the above S102, and the steps of receiving the user's log provision request in the above S103; the streaming storage module 122 is used to execute the above S102, and the steps of providing corresponding log data to the user according to the user's log provision request in the above S103.
[0054] In the embodiment of the present application, both the acquisition module 121 and the streaming storage module 122 can be implemented by software or by hardware. Exemplarily, the implementation of the acquisition module 121 is described below by taking the acquisition module 121 as an example. Similarly, the implementation of the streaming storage module 122 can refer to the implementation of the acquisition module 121.
[0055] As an example of a software functional unit, the acquisition module 121 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the acquisition module 121 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with similar geographical locations. Among them, usually a region may include multiple AZs.
[0056] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.
[0057] As an example of a hardware functional unit, the acquisition module 121 may include at least one computing device, such as a server, etc. Alternatively, the acquisition module 121 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.
[0058] The multiple computing devices included in the acquisition module 121 can be distributed in the same region or in different regions. The multiple computing devices included in the acquisition module 121 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the acquisition module 121 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.
[0059] It should be noted that, in other embodiments, the acquisition module 121 can be used to execute any one or more steps in the above S101-S103; the streaming storage module 122 can be used to execute any one or more steps in the above S101-S103. The steps that the acquisition module 121 and the streaming storage module 122 are responsible for implementing can be specified as needed, and the acquisition module 121 and the streaming storage module 122 respectively implement different steps in the log data processing method described above, so as to realize all the functions of the log storage engine 120.
[0060] The present application also provides a computing device. Figure 6 As shown, the computing device 200 includes a bus 210, a processor 220, a memory 230, and a communication interface 240. The processor 220, the memory 230, and the communication interface 240 communicate through the bus 210. The computing device 200 can be a server or a terminal device. It should be understood that the embodiment of the present application does not limit the number of processors and memories in the computing device.
[0061] The bus 210 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Figure 6 Although only one line is used in the figure, this does not mean that the computing device 200 has only one bus or one type of bus. The bus 210 may include a path for transmitting information between various components of the computing device 200 (eg, memory 230, processor 220, communication interface 240).
[0062] The processor 220 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).
[0063] The memory 230 may include a volatile memory, such as a random access memory (RAM). The memory 230 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).
[0064] The memory 230 stores executable program codes. The processor 220 executes the executable program codes stored in the memory 230 to respectively implement the functions of the aforementioned acquisition module 121 and the streaming storage module 122, thereby implementing the log data processing method described above. In other words, the memory 230 stores instructions for executing the log data processing method described above.
[0065] The communication interface 240 uses a transceiver module such as but not limited to a network interface card or a transceiver to implement communication between the computing device 200 and other devices or a communication network. For example, the computing device 200 obtains multiple log data of an application from the log collection device 110 through the communication interface 240, or provides multiple log data sorted in chronological order to the user through the communication interface 240.
[0066] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes one or more computing devices. The computing device can be a server, for example, a central server, an edge server, or a local server in a local data center. The computing device can also be a terminal device such as a desktop, a laptop or a smart phone. When the computing device cluster includes multiple computing devices, the multiple computing devices can be connected through a network (for example, a wide area network or a local area network).
[0067] like Figure 7As shown, the computing device cluster 300 includes one or more computing devices 200. The memory 230 in one or more computing devices 200 in the computing device cluster 300 may store the same instructions for executing the log data processing method described above. Alternatively, the memory 230 of one or more computing devices 200 in the computing device cluster 300 may also store partial instructions for executing the log data processing method described above. In other words, the combination of one or more computing devices 200 can jointly execute the instructions for executing the log data processing method described above.
[0068] It should be noted that the memory 230 in different computing devices 200 in the computing device cluster 300 can store different instructions to respectively implement part of the functions of the log storage engine 120. In other words, the instructions stored in the memory 230 in different computing devices 200 can implement the functions of the aforementioned acquisition module 121 and / or streaming storage module 122. It should be understood that the embodiment of the present application does not specifically limit which module in the log storage engine 120 is deployed on which computing device 200. In actual applications, it can be adaptively deployed according to the computing power or specific needs of the computing device 200.
[0069] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the log data processing method described above.
[0070] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media, wherein the available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital versatile disc (DVD)), or a semiconductor medium (e.g., a solid state drive (SSD)). The computer-readable storage medium includes instructions, wherein the instructions instruct the computing device to execute the log data processing method described above.
[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present application.
Claims
1. A method for processing log data, It is characterized in that The cloud computing platform provides a log service, the log service is used to manage the log data of the application, the cloud computing platform includes a server on which a log collection device is deployed, the log collection device is used to collect the log data of the application running on the server; The user obtains or queries the log data of the application through the log service provided by the cloud computing platform, and the method includes: The log storage engine obtains multiple log data of the application from the log collection device; The log storage engine stores the multiple log data in blocks in chronological order according to the timestamps of the multiple log data, wherein the block storage refers to storing the timestamp of the first log data included in the log block and the time offset of other log data included in the log block, and the log block includes at least two log data of the multiple log data; The log storage engine receives a log provision request from the user; The log storage engine obtains multiple log data from at least one of the log blocks according to the time or time interval specified by the log provision request, and provides the obtained multiple log data to the user in chronological order.
2. The method according to claim 1, It is characterized in that The log block further stores any one or more of the following items: the timestamp of the last log data included in the log block, and the amount of log data included in the log block.
3. The method according to claim 2, It is characterized in that The log storage engine stores the plurality of log data in blocks according to time sequence, including: The log storage engine stores the multiple log data in blocks in chronological order into a log merge tree LSM database.
4. The method according to any one of claims 1 to 3, It is characterized in that The timestamp of any log data is the time when the log collection device collects the any log data, or the coded value of the time, and the length of the coded value of the time is smaller than the length of the time.
5. A log storage engine, It is characterized in that The cloud computing platform provides a log service, the log service is used to manage the log data of the application, the cloud computing platform includes a server on which a log collection device is deployed, the log collection device is used to collect the log data of the application running on the server; The user obtains or queries the log data of the application through the log service provided by the cloud computing platform, and the log storage engine includes: An acquisition module, configured to acquire a plurality of log data of the application from the log collection device; and receive a log provision request from the user; A streaming storage module is used to store the multiple log data in blocks in chronological order according to the timestamps of the multiple log data, wherein the block storage refers to storing the timestamp of the first log data included in the log block and the time offset of other log data included in the log block, and the log block includes at least two log data of the multiple log data; and, according to the time or time interval specified by the log provision request, obtain the multiple log data from at least one of the log blocks, and provide the obtained multiple log data to the user in chronological order.
6. The log storage engine according to claim 5, It is characterized in that The log block further stores any one or more of the following items: the timestamp of the last log data included in the log block, and the amount of log data included in the log block.
7. The log storage engine according to claim 6, It is characterized in that The streaming storage module is used to store the multiple log data in blocks in chronological order into a log merge tree LSM database.
8. The log storage engine according to any one of claims 5 to 7, It is characterized in that The timestamp of any log data is the time when the log collection device collects the any log data, or the coded value of the time, and the length of the coded value of the time is smaller than the length of the time.
9. A logging system, It is characterized in that The log system is used to manage the log data of the application. The user obtains or queries the log data of the application through the log system. The log system includes: A log collection device, used to collect multiple log data of the application running on the server; A log storage engine, configured to obtain the plurality of log data, and store the plurality of log data in blocks in chronological order according to the timestamps of the plurality of log data, wherein the block storage refers to storing the timestamp of the first log data included in the log block and the time offset of other log data included in the log block, wherein the log block includes at least two log data of the plurality of log data; and receiving a log provision request of the user, obtaining the plurality of log data from at least one of the log blocks according to the time or time interval specified in the log provision request, and providing the obtained plurality of log data to the user in chronological order; The LSM database is used to store at least one of the log blocks.
10. A computing device, It is characterized in that The method comprises a processor and a memory, wherein the processor is used to execute instructions stored in the memory to implement the method according to any one of claims 1 to 4.
11. A computer program product comprising instructions, It is characterized in that When the instructions are executed by a computing device, the computing device cluster executes the method according to any one of claims 1 to 4.
12. A computer-readable storage medium, It is characterized in that The method comprises computer program instructions, which, when executed by a computing device, cause the computing device to perform the method according to any one of claims 1 to 4.
