Log storage and transmission processing method and system, electronic equipment and medium

Abstract

The invention discloses a log storage and transmission processing method and system, electronic equipment and a medium, and the method comprises the steps: obtaining an access parameter and a link log requested by a user at a client, and obtaining to-be-stored log information; performing serialization and first compression processing on the to-be-stored log information to obtain first log information; sending the first log information to a service cluster through a user datagram protocol or a hypertext transfer protocol according to the message size of the first log information and the number of servers in the service cluster; in the service cluster, writing the first log information into a double-buffer queue, and transmitting the first log information to a storage cluster in a multi-thread aggregation mode; and in the storage cluster, performing second compression processing on the first log information, and storing the first log information in a database of the storage cluster. The log storage and transmission efficiency and the application performance are improved, read-write operation does not need to be conducted on a disk in the log transmission process, resources of a server are saved, the technical cost is reduced, and the method can be widely applied to the technical field of computers.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a log storage transmission processing method, system, electronic device and medium. Background Art

[0002] In the daily operation of application services, it is often necessary to store various logs, such as recording the input and output parameters of user requests, information printed during system operation, and error logs, so that they can be checked when problems occur in the system. In the related art, you can log in to the server to view frameworks such as log4j, print logs to local files, and when the number of logs increases, you can use the ELK architecture to process the logs. If the amount of logs increases further, you can use a message queue (such as kafka) to receive the logs, and then store them in a database or write them to a local file.

[0003] However, as the business expands, the log system needs to withstand greater data throughput and higher hardware consumption, which requires huge costs to support. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a log storage and transmission processing method, system, electronic device and medium, aiming to achieve efficient log storage and transmission and reduce the cost of log storage and transmission.

[0005] To achieve the above purpose, an embodiment of the present application provides a log storage transmission processing method, the method comprising:

[0006] Obtain the input and output parameters and link logs of the user request on the client, and obtain the log information to be stored;

[0007] Performing serialization and first compression processing on the log information to be stored to obtain first log information;

[0008] Sending the first log information to the service cluster through the User Datagram Protocol or the Hypertext Transfer Protocol according to the message size of the first log information and the number of servers in the service cluster;

[0009] In the service cluster, writing the first log information into a double buffer queue, and transmitting the first log information to the storage cluster by multi-thread aggregation;

[0010] In the storage cluster, a second compression process is performed on the first log information, and the log information is stored in a database of the storage cluster.

[0011] In some embodiments, obtaining the input and output parameters and link logs requested by the user on the client to obtain the log information to be stored includes the following steps:

[0012] In a user request, the input and output parameters of the user request are obtained through a custom interceptor;

[0013] In a user request, the output source is customized through the log framework, and the logs generated during system operation are temporarily stored in the memory to obtain the link log;

[0014] The input and output parameters and the link log are used as log information to be stored.

[0015] In some embodiments, the serializing and first compressing the log information to be stored to obtain the first log information includes the following steps:

[0016] Perform protobuf serialization processing on the log information to be stored to obtain a serialized log;

[0017] A ZSTD compression method is adopted to perform a first compression process on the serialized log to obtain first log information.

[0018] In some embodiments, sending the first log information to the service cluster through the user datagram protocol or the hypertext transfer protocol according to the message size of the first log information and the number of servers in the service cluster includes the following steps:

[0019] Obtain the number of servers in the service cluster and the maximum message value of the User Datagram Protocol;

[0020] Using a hash algorithm, a server index is calculated according to the number of servers;

[0021] When the message size of the first log information does not exceed the maximum message value, sending the first log information to the server corresponding to the server index through the user datagram protocol;

[0022] When the message size of the first log information exceeds the maximum message value, the first log information is sent to a server corresponding to the server-side index through the Hypertext Transfer Protocol.

[0023] In some embodiments, in the service cluster, writing the first log information into a double buffer queue and transmitting it to the storage cluster by multi-thread aggregation includes the following steps:

[0024] Writing the first log information into a double buffer queue; wherein the double buffer queue includes two buffers, each of which is used for producing and consuming queue messages;

[0025] The first log information is obtained from the double buffer queue through multithreading, decompressed for consumption, and transmitted to the storage cluster.

[0026] In some embodiments, performing a second compression process on the first log information in the storage cluster and storing the first log information in a database of the storage cluster includes the following steps:

[0027] Using the clickhouse database, under the compression standard of the first log information, performing a second compression process on the first log information to obtain target log information;

[0028] The target log information is written into the database of the storage cluster, and an index field of the target log information is returned; wherein the index field is used for business query.

[0029] In some embodiments, the method further comprises the following steps:

[0030] Using a configuration center to store the IP addresses of the servers in the service cluster;

[0031] The server is allocated to the client through the configuration center, and the IP address of the allocated server is synchronized to the client.

[0032] To achieve the above purpose, another aspect of the embodiment of the present application provides a log storage transmission processing system, the system comprising:

[0033] The first module is used to obtain the input and output parameters and link logs of the user request on the client side, and obtain the log information to be stored;

[0034] The second module is used to perform serialization and first compression processing on the log information to be stored to obtain first log information;

[0035] A third module is used to send the first log information to the service cluster through the user datagram protocol or the hypertext transfer protocol according to the message size of the first log information and the number of servers in the service cluster;

[0036] A fourth module is used to write the first log information into a double buffer queue in the service cluster, and transmit it to the storage cluster by multi-thread aggregation;

[0037] The fifth module is used to perform a second compression process on the first log information in the storage cluster, and store the second compression process in a database of the storage cluster.

[0038] In some embodiments, the system may further include the following modules:

[0039] The sixth module is used to use the configuration center to store the IP addresses of the servers in the service cluster;

[0040] The seventh module allocates the server to the client through the configuration center, and synchronizes the IP address of the allocated server to the client.

[0041] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0042] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0043] The embodiments of the present application include at least the following beneficial effects: the present application provides a log storage transmission processing method, system, electronic device and medium, the scheme obtains the input and output parameters and link logs of the user request at the client to obtain the log information to be stored; serializes and performs a first compression process on the log information to be stored to obtain the first log information; according to the message size of the first log information and the number of servers in the service cluster, the first log information is sent to the service cluster through the user datagram protocol or the hypertext transfer protocol; in the service cluster, the first log information is written into a double buffer queue and transmitted to the storage cluster through multi-threaded aggregation; in the storage cluster, the first log information is subjected to a second compression process and stored in the database of the storage cluster. The overall steps improve the efficiency of log storage transmission and the application performance, and do not require read and write operations on the disk during the log transmission process, saving server resources and reducing technical costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0045] Figure 1 This is a schematic diagram of the steps of a log storage transmission processing method provided by an embodiment of the present application;

[0046] Figure 2 The embodiment of this application provides Figure 1 Schematic diagram of step S100 in FIG.

[0047] Figure 3 The embodiment of this application provides Figure 1 Schematic diagram of step S200 in FIG.

[0048] Figure 4 The embodiment of this application provides Figure 1Schematic diagram of step S300 in FIG.

[0049] Figure 5 The embodiment of this application provides Figure 1 Schematic diagram of step S400 in FIG.

[0050] Figure 6 The embodiment of this application provides Figure 1 A schematic diagram of step S500 in FIG.

[0051] Figure 7 It is a schematic diagram of the business process architecture provided by the embodiment of the present application;

[0052] Figure 8 This is a schematic diagram of the interaction between the client, service cluster and storage cluster provided in an embodiment of the present application;

[0053] Fig. 9 It is a schematic diagram of the data processing flow provided by the embodiment of the present application;

[0054] Fig.10 It is a module schematic diagram of a log storage transmission processing system provided by an embodiment of the present application;

[0055] Fig.11 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0057] Although the functional modules are divided in the system schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first / S100", "second / S200", etc. in the specification, claims and the above drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0058] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0059] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0060] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0062] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0063] (1) User Datagram Protocol: also known as UDP protocol.

[0064] (2) Hypertext Transfer Protocol: HTTP protocol.

[0065] In the related technologies, in the daily operation of applications, it is often necessary to store various logs, such as recording the input and output parameters of user requests, information printed during system operation, and error logs, so that problems can be investigated when they occur. With the rapid development of big data technology, in order to meet the needs of log storage and retrieval, there are many related frameworks to choose from, which can process and analyze massive data more efficiently.

[0066] When the number of machines is relatively small, you can print logs to local files by logging into the server and viewing frameworks such as log4j. However, when the number of logs increases, you can use the ELK architecture to process the logs. The ELK architecture is a combination of three open source software: Elasticsearch, Logstash, and Kibana. Elasticsearch is a distributed, RESTful search and analysis engine for full-text search, structured search, and analysis. It is based on the Lucene library and can extract useful information from large amounts of data in seconds. Logstash is a server-side data processing pipeline that can collect data from multiple sources at the same time, then transform the data and finally send the data to the selected storage "library". Kibana is an open source data visualization plug-in for Elasticsearch that provides ways to search, view, and interact with data stored in the Elasticsearch index. Kibana can be used to perform various data visualization operations, such as making charts and tables.

[0067] If the amount of logs increases further, you can use a message queue (such as Kafka) to receive the logs, then store them in a database or write them to a local file, and use tools such as filebeat to report and store them. The above are relatively common log transmission and storage solutions. In general, the main functions of the log framework include temporary storage of logs, transmission of logs, saving to a database, and providing fast retrieval.

[0068] The above-mentioned Filebeat is a component of the Elastic Stack (ELK Stack) and is used to collect, forward and centralize log data. It is a lightweight, open source log file data porter that focuses on sending log data to Elasticsearch or Logstash; Kafka is an open source stream processing platform that provides a high-throughput, scalable, reliable and real-time data stream processing framework, which is mainly used to process and analyze real-time data, allowing applications to publish and subscribe to data streams, and also store and process these data streams. A key feature of Kafka is the ability to process massive real-time data and ensure the order and reliability of this data in a distributed system.

[0069] As business systems become more and more complex, the log system has evolved from using the ELK architecture to using Hbase for extreme compression for storage, transmission and query. However, as the business expands, the number of machines is also increasing. By calculating the amount of data processed per second and per hour, as well as the required server resources and database occupancy, it can be found that the control or resources that need to be reserved for the log system require huge costs to support. Especially under high load conditions, the system needs to withstand greater throughput and higher hardware resource consumption.

[0070] In view of this, the embodiment of the present application provides a log storage transmission processing method, system, electronic device and medium. By constructing a configuration center, a client, a service cluster and a storage cluster, the logs collected by the client are serialized and compressed, and then sent to the server server in the server cluster with the same ID as the configuration center through the user datagram protocol (UDP protocol). After the server receives the log sent by the client, it parses the corresponding fields and writes them in batches into the database of the storage cluster. The server cluster uses large memory machines and double buffer queues to improve buffering capacity. The single machine has large memory and the overall number of machines is small, so the original logs can be processed efficiently.

[0071] A log storage transmission processing method provided in an embodiment of the present application relates to the field of computer technology. A log storage transmission processing method provided in an embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The cloud server, the server can also be a node server in a blockchain network; the software can be an application that implements a log storage transmission processing method, etc., but is not limited to the above forms.

[0072] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0073] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0074] Figure 1 This is an optional step diagram of a log storage transmission processing method provided by an embodiment of the present application. Figure 1 The method may include but is not limited to the following steps S100 to S500.

[0075] Step S100, obtaining the input and output parameters and link logs requested by the user at the client, and obtaining the log information to be stored.

[0076] Step S200: performing serialization and first compression processing on the log information to be stored to obtain first log information.

[0077] Step S300: sending the first log information to a service cluster through a user datagram protocol or a hypertext transfer protocol according to the message size of the first log information and the number of servers in the service cluster.

[0078] Step S400: In the service cluster, the first log information is written into a double buffer queue, and transmitted to the storage cluster by multi-thread aggregation.

[0079] Step S500: In the storage cluster, a second compression process is performed on the first log information, and the second compression process is performed and the first log information is stored in a database of the storage cluster.

[0080] In steps S100 to S500 shown in the embodiment of the present application, the log data to be stored is serialized and compressed, and sent to the service cluster through the User Datagram Protocol or the Hypertext Transfer Protocol. The service cluster processes the log data and then compresses and stores them in the storage, thereby improving the efficiency of log storage and transmission and the application performance. During the log transmission process, there is no need to read and write the disk, which saves server resources and reduces technical costs.

[0081] See also Figure 2 In some embodiments, step S100 may include but is not limited to the following steps S110 to S130:

[0082] Step S110, in a user request, obtaining the input and output parameters of the user request through a custom interceptor.

[0083] Step S120, in a user request, the output source is customized through the log framework, and the logs generated during system operation are temporarily stored in the memory to obtain the link log.

[0084] Step S130: Use the input and output parameters and the link log as log information to be stored.

[0085] Through the above steps S110 to S130, the user request input parameters and link logs are obtained as information to be stored for subsequent optimization processing of storage transmission.

[0086] See also Figure 3 In some embodiments, step S200 includes but is not limited to the following steps S210 to S220:

[0087] Step S210, performing protobuf serialization processing on the log information to be stored to obtain a serialized log.

[0088] Step S220: Using the ZSTD compression method, perform a first compression process on the serialized log to obtain first log information.

[0089] Through the above steps S210 to S220, the log to be stored is serialized by protobuf, which is efficient and has a small data volume after processing; the ZSTD compression method can quickly, efficiently and losslessly compress the serialized log data, which is conducive to subsequent data transmission through the UDP protocol or HTTP protocol.

[0090] See also Figure 4 In some embodiments, step S300 includes but is not limited to the following steps S310 to S340:

[0091] Step S310, obtaining the number of servers in the service cluster and the maximum message value of the User Datagram Protocol.

[0092] Step S320: Using a hash algorithm, calculate a server index according to the number of servers.

[0093] Step S330: When the message size of the first log information does not exceed the maximum message value, the first log information is sent to the server corresponding to the server index through the user datagram protocol.

[0094] Step S340: When the message size of the first log information exceeds the maximum message value, the first log information is sent to the server corresponding to the server index through the Hypertext Transfer Protocol.

[0095] Through the above steps S310 to S340, the server of the service cluster can be quickly indexed, and the first log information can be quickly transmitted to the server through the user datagram protocol as the main method and the hypertext transfer protocol as the candidate.

[0096] See also Figure 5 In some embodiments, step S400 includes but is not limited to the following steps S410 to S420:

[0097] Step S410: write the first log information into a double-buffered queue; wherein the double-buffered queue includes two buffers, each of which is used for producing and consuming queue messages.

[0098] Step S420: Obtain the first log information from the double buffer queue through multithreading, decompress it for consumption, and transmit it to the storage cluster.

[0099] Through the above steps S410 to S420, a double buffer queue is adopted in the service cluster, and log processing is performed by multi-thread aggregation, which can improve the buffering capacity of the service cluster.

[0100] See also Figure 6 In some embodiments, step S500 includes but is not limited to the following steps S510 to S520:

[0101] Step S510: Using the clickhouse database, under the compression standard of the first log information, perform a second compression process on the first log information to obtain target log information.

[0102] Step S520, writing the target log information into the database of the storage cluster, and returning the index field of the target log information; wherein the index field is used for business query.

[0103] Through the above steps S510 to S520, the log information is compressed again and stored in the database of the storage cluster, which is beneficial to saving the storage space of the database in the storage cluster.

[0104] In some embodiments, the method further includes but is not limited to the following steps S610-S620:

[0105] Step S610: Using a configuration center to store IP addresses of servers in the service cluster.

[0106] Step S620: Allocate the server to the client through the configuration center, and synchronize the IP address of the allocated server to the client.

[0107] Through the above steps S610 to S620, the configuration center configures the IP address of the server for the client, so that the client and the server can quickly and accurately connect and interact.

[0108] Below, in conjunction with a specific log storage and transmission scenario application example, the solution of the embodiment of the present application is described in detail and explained:

[0109] In an embodiment of the present application, a log storage transmission processing method is provided, which can be applied to efficient log storage transmission and can reduce the cost of log storage transmission. Specifically, in some embodiments, the following can be achieved:

[0110] See also Figure 7 First, we build an architecture including the configuration center, client, server cluster, and storage cluster. We can build several server clusters and storage clusters to meet business needs, and we can flexibly adjust the number of clusters according to business needs. Figure 8 and Fig. 9 , introduces the above-mentioned architectural elements and data interaction processes:

[0111] 1. The configuration center is used to store the IP addresses of the servers in the service cluster and allocate servers to various modules of the client.

[0112] 2. The client is used to obtain the server IP address assigned by the configuration center, and in one request, the input and output parameters are obtained through a custom interceptor or a corresponding system interceptor, and the input and output parameters are serialized through protobuf, compressed by ZSTD, and transmitted to the server configured by the configuration center through UDP. In some embodiments, serialization can also be performed using JSN, CML, etc., which are more readable but less efficient. In some embodiments, Bzip2 and Gzip compression methods can also be used, but this solution has the problem of slow compression speed and relatively high memory usage.

[0113] The sending rule of UDP is: the client will calculate the server index through the hash algorithm (i.e., hash algorithm) according to the number of server clusters, and then distribute the data to the indexed server. For example: there are two clients and ten servers, then the hash algorithm is modulo the number of servers (ten) to get an index value, which is the index value assigned to the server, and then the client sends the message to the indexed server through the UDP connection.

[0114] When the compressed message still exceeds the maximum UDP message value, it is sent via HTTP.

[0115] The client SDK can use TransmittableThreadLocal to prevent the unique ID of the link tracer from being lost in multiple threads or thread pools; TransmittableThreadLocal is a tool that solves variable transfer and inheritance problems in a multi-threaded environment.

[0116] The client implementation omits the steps of writing to the local disk and consuming files and sending them to MQ (Message Queue), which has little impact on the CPU and access performance. UDP is used for sending, so there is no need to consider the problem of blocking caused by slow server consumption, which is simple and efficient.

[0117] 3. The server side can use several servers with memory greater than or equal to 64GB to build a service cluster. For example, several single servers with memory of 64GB or 128GB can be used to build a service cluster, so that the single memory is large and the total number of servers is small, which can effectively save hardware resources.

[0118] For example, in the service cluster constructed by the embodiment of the present application, a single Docker container can process 10,000 to 50,000 lines of raw logs per second, can process 20,000 client QPS per second, and can write more than 200M per second to the storage cluster. Since the data is compressed and only decompressed when the end user queries, only 100 servers are needed to efficiently process the raw logs in the service cluster as shown in the example.

[0119] The server uses a large memory machine and a double-buffered queue to improve the buffering capacity, writes the received bytes into the double-buffered queue, decompresses the bytes through multi-threading, achieves the ultimate compression processing performance, and extracts the index fields for warehousing for subsequent query services. The objects to be stored are written into the double-buffered queue, and are stored in batches through multi-threaded aggregation. At the same time, the large message data of the input and output parameters is compressed from the beginning to the end, saving bandwidth disk and improving throughput. Compared with the MQ writing solution, the pure memory exchange link system of the embodiment of the present application is suitable for high-speed access requirements and low latency and high concurrency business scenarios because it exchanges and processes data completely in memory without involving disks or other persistent storage media. Through pure memory exchange, the consumption of reading and writing disks is avoided, the speed of data exchange is improved, and the data throughput during log storage and transmission is improved.

[0120] 4. The storage cluster can use the clickhouse database, which is a powerful database with high compression ratio and writing performance. It is sharded by day and is suitable for log systems that write large amounts of data and query less. In the embodiment of the present application, under the original compression benchmark, the storage cluster compresses the logs again when entering the warehouse, so that the disk space occupied by the data is only about 1 / 20 of the original data of the client. At the same time, a low-concurrency, large-volume warehousing method is adopted to improve the throughput capacity of the storage cluster.

[0121] In summary, the specific operation steps may include:

[0122] (1) Install the clickhouse database, test the connection, and initialize and create a table through table design.

[0123] (2) Start the server, configure the database address information and the configuration center address, introduce the configurator from the config module, the default file configurator, and it is recommended to inherit FileConfigrator to maintain the worker configuration and adapt it according to the CustomConfigurator class. Start the server and start receiving data.

[0124] (3) Configure the center deployment and register the server information.

[0125] (4) Start the client. Register the client address to the server and obtain the server information. According to its own log framework, introduce the corresponding clientjar package. The clientjar package is a java archive file for the client application. Select the corresponding log.xml configuration and override the appender (the appender is used to define the output method and destination of the log). Refer to the corresponding clientlog module or example-TracerLogbackAppender for adaptation. Configure the extraction and compression configurations in the configuration file or other configuration center. If workers are not configured, send data to the local computer, start the client service, and use the database management software Dbeaver to view the data.

[0126] (5) Description of recorded data: Log data can be divided into two types: input and output parameters and link logs. The basic information of input and output parameters is relatively fixed, while the intermediate log is the log in the request and can be added by yourself. You can customize the log recording format, log storage format and display logic, etc.

[0127] In the experimental results of the embodiments of the present application, due to the extreme compression of the data and the use of pure memory exchange during the transmission process, the disk occupancy is only 0.8 of Clickhouse, and the actual occupied space is less than 80% of the storage capacity. In this way, the disk alone can save more than 75% of the storage cost. Since the second-level throughput requires the consumption of server CPU, a single server requires 5 or more servers to process 1G of disk writes per second. Therefore, with the substantial savings in disk, a large number of intermediate process CPU servers can be saved linearly. The actual statistical results show that more than 70% of the servers in the process can be saved, achieving cost reduction.

[0128] In terms of software performance, the consumption of the client of the related technology mainly includes the consumption of steps such as serialization, writing disk, reading disk and deserialization, while the UDP of this application only needs one serialization. Assuming that the MQ cluster has unlimited writing capabilities and can process all the logs sent in the past, it is compared with the consumption performance of the server. When pulling and consuming logs from MQ, if MQ has no backlog, it can support zero-copy operations for high-speed pulling; but if there is a backlog, a large number of MQ disk IOs may be generated, resulting in a significant decrease in the pulling speed. The efficiency of this process is significantly lower than the efficiency of using UDP to send to the server for processing, and it occupies twice the network bandwidth. In fact, the strong performance shown by the server is more than 10 times higher than the consumption performance of a single pull MQ cluster in the related technology. More importantly, it reduces the load on the data layer and reduces the need for continuous expansion of the cluster, improves the performance of the application layer, and saves server resources. In addition, the various cluster architectures of the embodiment of the present application interact directly with each other, and interact directly with the client through the UDP protocol or HTTP protocol, without over-reliance on middleware, and does not invade the application system.

[0129] Based on this, the embodiments of the present application have at least the following beneficial effects:

[0130] ① Improved log processing efficiency and application layer performance;

[0131] ② Reduce the data layer load and cluster complexity, saving server resources;

[0132] ③Do not over-rely on middleware, do not invade application systems, and ensure access is friendly and efficient.

[0133] See also Fig.10 The embodiment of the present application also provides a log storage transmission processing system, which can implement the above-mentioned log storage transmission processing method, and the system includes:

[0134] The first module 101 is used to obtain the input and output parameters and link logs requested by the user on the client, and obtain the log information to be stored;

[0135] The second module 102 is used to perform serialization and first compression processing on the log information to be stored to obtain first log information;

[0136] The third module 103 is used to send the first log information to the service cluster through the user datagram protocol or the hypertext transfer protocol according to the message size of the first log information and the number of servers in the service cluster;

[0137] The fourth module 104 is used to write the first log information into a double buffer queue in the service cluster, and transmit it to the storage cluster by multi-thread aggregation;

[0138] The fifth module 105 is used to perform a second compression process on the first log information in the storage cluster, and store the second compression process in a database of the storage cluster.

[0139] In some embodiments, the system may further include the following modules:

[0140] The sixth module is used to use the configuration center to store the IP addresses of the servers in the service cluster;

[0141] The seventh module allocates the server to the client through the configuration center, and synchronizes the IP address of the allocated server to the client.

[0142] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0143] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned log storage transmission processing method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0144] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0145] See also Fig.11 , Fig.11 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0146] The processor 201 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0147] The memory 202 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 202 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 202, and the processor 201 calls and executes a log storage transmission processing method of the embodiment of this application;

[0148] Input / output interface 203, used to implement information input and output;

[0149] The communication interface 204 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0150] Bus 205 , which transmits information between various components of the device (e.g., processor 201 , memory 202 , input / output interface 203 , and communication interface 204 );

[0151] The processor 201 , the memory 202 , the input / output interface 203 and the communication interface 204 are connected to each other in communication within the device via the bus 205 .

[0152] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned log storage and transmission processing method is implemented.

[0153] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0154] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0155] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0156] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0157] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0158] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0159] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0160] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0161] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0162] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0163] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0164] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0165] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A log storage transmission processing method, characterized in that: The following steps are involved: Obtain the input and output parameters and link logs of the user request on the client, and obtain the log information to be stored; Performing serialization and first compression processing on the log information to be stored to obtain first log information; Sending the first log information to the service cluster through the User Datagram Protocol or the Hypertext Transfer Protocol according to the message size of the first log information and the number of servers in the service cluster; In the service cluster, writing the first log information into a double buffer queue, and transmitting the first log information to the storage cluster by multi-thread aggregation; In the storage cluster, a second compression process is performed on the first log information, and the log information is stored in a database of the storage cluster.

2. The method according to claim 1, characterized in that The step of obtaining the input and output parameters and link logs requested by the user on the client to obtain the log information to be stored includes the following steps: In a user request, the input and output parameters of the user request are obtained through a custom interceptor; In a user request, the output source is customized through the log framework, and the logs generated during system operation are temporarily stored in the memory to obtain the link log; The input and output parameters and the link log are used as log information to be stored.

3. The method according to claim 1, characterized in that The serializing and first compressing the log information to be stored to obtain the first log information includes the following steps: Perform protobuf serialization processing on the log information to be stored to obtain a serialized log; A ZSTD compression method is adopted to perform a first compression process on the serialized log to obtain first log information.

4. The method according to claim 1, characterized in that: The sending the first log information to the service cluster through the user datagram protocol or the hypertext transfer protocol according to the message size of the first log information and the number of servers in the service cluster comprises the following steps: Obtain the number of servers in the service cluster and the maximum message value of the User Datagram Protocol; Using a hash algorithm, a server index is calculated according to the number of servers; When the message size of the first log information does not exceed the maximum message value, sending the first log information to the server corresponding to the server index through the user datagram protocol; When the message size of the first log information exceeds the maximum message value, the first log information is sent to a server corresponding to the server-side index through the Hypertext Transfer Protocol.

5. The method according to claim 1, characterized in that In the service cluster, writing the first log information into a double buffer queue and transmitting it to the storage cluster by multi-thread aggregation includes the following steps: Writing the first log information into a double buffer queue; wherein the double buffer queue includes two buffers, each of which is used for producing and consuming queue messages; The first log information is obtained from the double buffer queue through multithreading, decompressed for consumption, and transmitted to the storage cluster.

6. The method according to claim 1, characterized in that The step of performing a second compression process on the first log information in the storage cluster and storing the first log information in a database of the storage cluster includes the following steps: Using the clickhouse database, under the compression standard of the first log information, performing a second compression process on the first log information to obtain target log information; The target log information is written into the database of the storage cluster, and an index field of the target log information is returned; wherein the index field is used for business query.

7. The method according to claim 1, characterized in that The method further comprises the following steps: Using a configuration center to store the IP addresses of the servers in the service cluster; The server is allocated to the client through the configuration center, and the IP address of the allocated server is synchronized to the client.

8. A log storage, transmission and processing system, characterized in that: include: The first module is used to obtain the input and output parameters and link logs of the user request on the client side, and obtain the log information to be stored; The second module is used to perform serialization and first compression processing on the log information to be stored to obtain first log information; A third module is used to send the first log information to the service cluster through the user datagram protocol or the hypertext transfer protocol according to the message size of the first log information and the number of servers in the service cluster; A fourth module is used to write the first log information into a double buffer queue in the service cluster, and transmit it to the storage cluster by multi-thread aggregation; The fifth module is used to perform a second compression process on the first log information in the storage cluster, and store the second compression process in a database of the storage cluster.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.

10. A computer storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to implement the method according to any one of claims 1 to 7 when executed by the processor.

Images