Log management method and device for distributed system, equipment and medium

Abstract

The invention discloses a log management method and device for a distributed system, equipment and a medium, and belongs to the technical field of Internet. The method comprises the following steps: in response to a unified interface calling request of a toolkit of an application end node to a log center, receiving log data sent by the application end node through a unified interface; sending the log data to message middleware; determining a target storage node of the log data in the distributed storage system according to a user of the log data defined by the message-oriented middleware; and using the log data through the target storage node. According to the technical scheme, the log data in the standard writing format sent by the toolkit of each application end node is received through the uniform interface of the log center, the format of the log data can be unified, the management efficiency of the log data is improved, decoupling of the application end nodes and the log center is realized, and the log data is transmitted through the message middleware; the log center and the distributed storage system can be decoupled, and the log service performance is improved.

Description

Technical Field

[0001] The present application belongs to the field of Internet technology, and specifically relates to a log management method, device, equipment and medium for a distributed system. Background Art

[0002] With the rapid development of technology and the increasing richness of application scenarios, the characteristics of highly integrated and diversified software have become more and more prominent, and the distributed platform architecture has become the mainstream trend. The services of the distributed platform architecture can not only cross technical boundaries and integrate diverse programming languages ​​such as Java, C++, C#, HTML5, Android and iOS, but also be closely connected through intricate interaction modes, jointly supporting a large and flexible application ecosystem.

[0003] Logs are recorded information generated by computer systems, applications or services during operation. For distributed platform architecture, each service may use different log recording methods, resulting in different log formats and confusing level divisions. If the logs are not processed in a unified manner, it will cause great trouble for subsequent log analysis and problem tracking. Therefore, how to achieve unified processing of logs of various services in a distributed platform architecture is an urgent problem that people in this field need to solve. Summary of the invention

[0004] The embodiments of the present application provide a log management method, apparatus, device and medium for a distributed system, the purpose of which is to unify the format of log data, decouple application end nodes from the log center, decouple the log center from the distributed storage system, and improve the management efficiency of log data and the performance of log services.

[0005] In a first aspect, an embodiment of the present application provides a log management method for a distributed system, the method comprising:

[0006] In response to a call request from a toolkit of an application end node to a unified interface of a log center, log data sent by the application end node is received through the unified interface; wherein the distributed system includes at least two application end nodes, the application end nodes are pre-packaged with a toolkit, and the toolkit is used to provide a standard writing format for log data;

[0007] Sending the log data to the message middleware;

[0008] Determine a target storage node for the log data in a distributed storage system according to a user of the log data defined by the message middleware;

[0009] The log data is used by the target storage node.

[0010] In a second aspect, an embodiment of the present application provides a log management device for a distributed system, the device comprising:

[0011] A log data acquisition module, configured to respond to a call request from a toolkit of an application end node to a unified interface of a log center, and receive log data sent by the application end node through the unified interface; wherein the distributed system includes at least two application end nodes, the application end nodes are pre-packaged with a toolkit, and the toolkit is used to provide a standard writing format for log data;

[0012] A log data sending module, used for sending the log data to the message middleware;

[0013] A storage node determination module, used to determine a target storage node of the log data in a distributed storage system according to a user of the log data defined by the message middleware;

[0014] The log data using module is used to use the log data through the target storage node.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0016] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0017] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0018] In an embodiment of the present application, in response to a call request from the toolkit of the application end node to the unified interface of the log center, log data issued by the application end node is received through the unified interface; wherein the distributed system includes at least two application end nodes, the application end node is pre-packaged with a toolkit, and the toolkit is used to provide a standard writing format for log data; the log data is sent to the message middleware; according to the user of the log data defined by the message middleware, the target storage node of the log data in the distributed storage system is determined; and the log data is used through the target storage node. The log management method of the above-mentioned distributed system receives the log data in the standard writing format issued by the toolkit of each application end node through the unified interface of the log center, which can unify the format of the log data, improve the management efficiency of the log data, and realize the decoupling of the application end node and the log center. By transmitting the log data through the message middleware, the decoupling of the log center and the distributed storage system can be realized, and the performance of the log service can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flow chart of the log management method of the distributed system provided in the first embodiment of the present application;

[0020] Figure 2 This is a schematic diagram of the process of sending log data by an application end node provided in Example 1 of the present application;

[0021] Figure 3 This is a scenario diagram of a log management method for a distributed system provided in Example 1 of the present application;

[0022] Figure 4 It is a flow chart of the log management method of the distributed system provided in the second embodiment of the present application;

[0023] Figure 5 It is a flow chart of the log management method of the distributed system provided in the third embodiment of the present application;

[0024] Figure 6 This is a scenario diagram of a log management method for a distributed system provided in Example 3 of the present application;

[0025] Figure 7 It is a structural diagram of a log management device for a distributed system provided in Embodiment 4 of the present application;

[0026] Figure 8 It is a schematic diagram of the structure of an electronic device provided in Example 5 of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only the part related to the present application but not all the contents are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0029] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0030] In the following, in conjunction with the accompanying drawings, the log management method, device, equipment and medium of the distributed system provided by the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0031] Embodiment 1

[0032] Figure 1 It is a flow chart of the log management method for a distributed system provided in Example 1 of the present application.

[0033] like Figure 1 As shown, the specific steps include:

[0034] S101, in response to a call request from a toolkit of an application end node to a unified interface of a log center, receiving log data sent by the application end node through the unified interface; wherein the distributed system includes at least two application end nodes, the application end nodes are pre-packaged with a toolkit, and the toolkit is used to provide a standard writing format for log data;

[0035] First, the present application is applicable to scenarios where the log contents of each application end node included in the distributed system need to be stored or otherwise processed. Based on the above usage scenarios, it can be understood that the execution subject of the present application can be a log management system connected to the distributed system. Specifically, the reception, transmission and storage of log data can be performed by the log management system, so as to achieve unified processing of the log contents of each application end node, and facilitate the management of the log contents of each application end node.

[0036] A distributed system refers to a computing architecture, specifically: multiple application nodes work together to complete tasks or provide services, and these application nodes communicate and coordinate with each other through the network. Among them, the application node can refer to the software application used to complete part of the task or provide part of the service in the distributed system. The application node can be implemented in different programming languages, such as java, c++, c#, h5, android, and ios.

[0037] The toolkit may refer to the form of an SDK (Software Development Kit) of an application node. Among them, SDK is a collection of tools and resources for developing software applications. The toolkit of the application node can be pre-packaged by building the code of the application node into smaller modules, adding interfaces and configuration options to each module, and finally packaging each module into a format suitable for distribution and storing it in the application node.

[0038] The log management system can be a software system for unified management of the log content of each application end node of the distributed system, and can include a log center, a message middleware, and a distributed storage system. Among them, the log center can be a platform for unified reception and transmission of log data sent by each application end node.

[0039] The log center provides a unified interface, which can be an interface for receiving log data sent by each application node. The interface refers to a set of rules and protocols that define the interaction between software. The interface allows different software systems or components to communicate with each other and share functions and data.

[0040] Log data may be data including log content and in a unified format for each application end node. The standard writing format is the format in a unified format for each application end node, the standard writing format is defined by a unified interface, and the toolkit includes data for describing the standard writing format, which may include a global identifier, an application end node identifier, log content, request time, request IP address, and log level.

[0041] In the present technical solution, optionally, the standard writing format includes one or more of a global identifier, an application node identifier, log content, a request time, a request IP address, and a log level.

[0042] The global identifier may refer to an identifier used to uniquely identify each unified interface call request in the log management system. The global identifier can facilitate tracking of the call link log of each unified interface call request; wherein, the call link log is also called a distributed tracking system, which is a tool for recording and analyzing the request call path in a distributed system. The application end node identifier may be an identifier used to represent the application end node that issued the unified interface call request. The log content may refer to the specific content of the log file generated by the application end node. The request time may refer to the time point when the application end node issued the unified interface call request. The request IP address may refer to the IP address of the device that issued the unified interface call request; wherein, the IP address (Internet Protocol Address) is a unique identifier assigned to each device in the network. The log level is the result of classification according to the importance and urgency of the log content, and may include DEBUG, INFO, WARN, ERROR, and FATAL / CRITICAL, etc.

[0043] The advantage of this setting of the present scheme is that by stipulating a standard writing format including a global identifier, an application node identifier, log content, request time, request IP address and log level, the format of the log content of each application node can be unified, thereby improving the management efficiency of the log content of each application node.

[0044] Figure 2 This is an example diagram of the process of sending log data by the application end node provided in the first embodiment of the present application. Figure 2As shown, the process of the application end node sending log data includes: S201, the application end node calls the log writing interface of the toolkit; S202, verifies whether the input parameter is legal; if so, executes S203; if not, ends the process; S203, the toolkit assembles log data; S204, the toolkit calls the unified interface of the log center. Specifically, after generating the log file, the application end node calls the log writing interface of the toolkit, wherein the log content is used as the input parameter of the log writing interface; the toolkit assembles the log data according to the standard writing format, wherein the global identifier can be defaulted; the toolkit calls the unified interface of the log center, that is, the toolkit sends a unified interface call request to the log center, wherein the unified interface call request can be a message used to inform the log center that the current log data needs to be processed, and the unified interface call request includes the log data. Such a design can decouple each application end node from the log center, and the adjustment of the unified interface of the log center will not affect the log content of each application end node.

[0045] The toolkit uses the log data as the input parameter of the unified interface and calls the unified interface of the log center. The log center receives the log data by reading the input parameter of the unified interface. If the log data sent by the toolkit does not include a global identifier, it can be generated uniformly by the log center. In this technical solution, optionally, after reading the input parameter of the unified interface, the log center can verify the input parameter.

[0046] S102, sending the log data to the message middleware;

[0047] Message middleware is a software architecture that provides message delivery and communication functions in distributed systems. Message middleware can act as an intermediary between log centers and distributed storage systems, allowing different systems or components to exchange information asynchronously. In this technical solution, the message middleware can use Kafka middleware.

[0048] The log center can send log data to the message middleware by calling the message production interface of the message middleware.

[0049] Optionally, in the present technical solution, after the log data is sent to the message middleware, it is possible to verify whether the log data is sent successfully, so as to achieve unified processing of abnormal situations in which the log data is lost in sending.

[0050] S103, determining a target storage node for the log data in a distributed storage system according to a user of the log data defined by the message middleware;

[0051] A distributed storage system is a system that stores data in multiple storage nodes. A user can be a component or application that is responsible for receiving and processing messages sent by a producer. Each storage node in a distributed storage system can be a user of log data, and the target storage node is the user corresponding to the current log data.

[0052] The user of each log data can be pre-defined by a developer or a backend administrator, and can be pre-defined based on one or more of the global identifier, application end node identifier, log content, request time, request IP address, and log level in the log data. For example, based on the application end node identifier, the distributed storage system can include java storage nodes, c++ storage nodes, c# storage nodes, h5 storage nodes, android storage nodes, and ios storage nodes, and the user of the log data whose application end node identifier is java is pre-defined as a java storage node. Using the current log data as the query condition, the data describing each pre-defined user is queried to determine the target storage node of the current log data in the distributed storage system.

[0053] S104: Using the log data through the target storage node.

[0054] The target storage node uses log data means storing the log data in the target storage node. Depending on the different message middleware, the target storage node obtains log data in different ways, for example, the message middleware pushes log data to the target storage node, or the target storage node pulls log data from the message middleware.

[0055] Figure 3 Schematic diagram of the scenario of the log management method of the distributed system provided in the first embodiment of the present application. Figure 3 As shown, the log center uniformly receives the log data sent by each application node through the toolkit, and sends the log data to the message middleware, and the distributed storage system stores the log data received by the message middleware.

[0056] In an embodiment of the present application, in response to a call request from the toolkit of the application end node to the unified interface of the log center, log data issued by the application end node is received through the unified interface; wherein the distributed system includes at least two application end nodes, the application end node is pre-packaged with a toolkit, and the toolkit is used to provide a standard writing format for log data; the log data is sent to the message middleware; according to the user of the log data defined by the message middleware, the target storage node of the log data in the distributed storage system is determined; and the log data is used through the target storage node. The log management method of the above-mentioned distributed system receives the log data in the standard writing format issued by the toolkit of each application end node through the unified interface of the log center, which can unify the format of the log data, improve the management efficiency of the log data, and realize the decoupling of the application end node and the log center. By transmitting the log data through the message middleware, the decoupling of the log center and the distributed storage system can be realized, and the performance of the log service can be improved.

[0057] Embodiment 2

[0058] Figure 4 This is a flow chart of the log management method for a distributed system provided in the second embodiment of the present application. This solution makes a better improvement on the above embodiment, and the specific improvement is: the message middleware adopts kafka middleware.

[0059] like Figure 4 As shown, the specific steps include:

[0060] S401, in response to a call request from a toolkit of an application end node to a unified interface of a log center, receiving log data sent by the application end node through the unified interface; wherein the distributed system includes at least two application end nodes, the application end nodes are pre-packaged with a toolkit, and the toolkit is used to provide a standard writing format for log data;

[0061] S402, sending the log data to the message middleware;

[0062] S403, determining a target storage node for the log data in a distributed storage system according to a user of the log data defined by the message middleware;

[0063] S404: Using the log data through the target storage node.

[0064] Kafka middleware is an open source distributed stream processing platform, mainly used to build real-time data pipelines and stream processing applications. Kafka middleware uses a pull mode, where users actively request messages and control the frequency and quantity of pulls.

[0065] In the technical solution, optionally, using the log data through the target storage node includes:

[0066] Pulling the log data from the kafka middleware through the target storage node according to a preset period to store the log data in the target storage node;

[0067] The usage status of the log data is returned to the kafka middleware through the target storage node.

[0068] The preset period may refer to a preset time interval for repeated occurrence of the event that the target storage node pulls log data from the Kafka middleware. The preset period may be set according to the performance of the target storage node and the storage and processing requirements of the log data.

[0069] Each storage node in the distributed storage system subscribes to the topic of the corresponding log data in the Kafka middleware. After subscription, the target storage node can use the poll method to pull log data from the topic of the corresponding log data and store it.

[0070] The usage status of the log data can be represented by the storage result of the log data in the target storage node. Specifically, the usage status of the log data is used, which means that the log data is successfully stored in the target storage node.

[0071] The kafka middleware can be configured to automatically commit offsets. In this mode, the target storage node will automatically commit the offset of the used message, that is, the usage status of the log data, after each call to the poll method.

[0072] The benefit of this arrangement is that by returning the usage status of the log data to the Kafka middleware through the target storage node, it can be ensured that the log data will not be lost due to errors or exceptions.

[0073] The benefit of this setting is that by using Kafka middleware through the message middleware, the log center and the distributed storage system can be decoupled. In addition, Kafka middleware has the advantages of high throughput, low latency, and flexible publishing / subscription, which can improve the processing performance and efficiency of log data.

[0074] Embodiment 3

[0075] Figure 5It is a flow chart of the log management method of the distributed system provided in the third embodiment of the present application. This scheme makes a better improvement on the above embodiment, and the specific improvement is: the unified interface includes at least two interface nodes; accordingly, in response to the toolkit of the application end node calling the unified interface request of the log center, the log data sent by the application end node is received through the unified interface, including: in response to the toolkit of the application end node calling the unified interface request of the log center, the target interface node is determined according to the load status of each interface node; the log data sent by the application end node is received through the target interface node.

[0076] like Figure 5 As shown, the specific steps include:

[0077] S501, in response to a unified interface call request from a toolkit of an application end node to a log center, determining a target interface node according to a load status of each interface node;

[0078] The interface node may refer to an independent interface capable of receiving log data sent by each application end node, as a component of the unified interface of the log center to the outside world. Figure 6 Schematic diagram of the scenario of the log management method of the distributed system provided in the third embodiment of the present application. Figure 6 As shown, the unified interface of the log center includes multiple interface nodes, each of which can be used to receive log data from each application end node.

[0079] The load status of an interface node may refer to the ability and load of the interface node to currently process unified interface call requests, which may include the number of requests, response time, resource usage, error rate, and throughput, etc. A suitable load balancing algorithm may be used to determine the target interface node according to the load status of each interface node. Among them, common load balancing algorithms include polling, least connection, weighted polling, and state-based allocation, etc. The target interface node is the interface node determined to process the current unified interface call request.

[0080] In the technical solution, optionally, in response to a unified interface call request from a toolkit of an application end node to a log center, a target interface node is determined according to a load status of each interface node, including:

[0081] In response to the unified interface call request from the toolkit of the application node to the log center, a consistent hashing algorithm is used to determine the target interface node according to the load status of each interface node.

[0082] The consistent hashing algorithm is a hashing algorithm used in distributed systems, mainly used for data distribution and load balancing. Unlike traditional hashing algorithms, the consistent hashing algorithm aims to minimize the impact of data migration when nodes are changed.

[0083] Specifically, the consistent hashing algorithm maps each interface node to a logical ring. In response to the toolkit of the application node's unified interface call request to the log center, the unified interface call request will be mapped to this logical ring and processed by the first interface node in the clockwise direction.

[0084] The benefit of this setting is that the consistent hashing algorithm is flexible, efficient, and balanced, and can easily handle the dynamic addition and removal of interface nodes, reduce the cost of data migration, and achieve more uniform load balancing.

[0085] In this technical solution, optionally, the method further includes:

[0086] In response to an instruction to adjust the number of interface nodes, the at least two interface nodes are added or deleted; wherein the instruction to adjust the number is generated based on the resource space of the distributed system and the load data of the established interface nodes.

[0087] The instruction to adjust the number of interface nodes may be a computer instruction for adding or deleting interface nodes, which may be generated by a backend administrator or by the log center itself based on the resource space of the distributed system and the load data of the established interface nodes. The resource space of the distributed system may refer to the total resources used to store and process data in the distributed system, mainly referring to computing resources; the load data of the established interface nodes may be statistical data on the load status of the established interface nodes. It is understandable that when the load data of the established interface nodes cannot meet the resource space requirements of the distributed system, it is necessary to add interface nodes; when the load data of the established interface nodes exceeds the resource space requirements of the distributed system, it is necessary to delete the interface nodes.

[0088] To add an interface node, the interface node to be added can be mapped to the logical ring of the consistent hashing algorithm, and part of the data of the first interface node found clockwise can be migrated to the added interface node; to delete an interface node, the data of the deleted interface node can be migrated to the first interface node found clockwise.

[0089] The benefit of this setting of the present solution is that by adjusting the number of interface nodes based on the resource space of the distributed system and the load data of the established interface nodes, dynamic optimization of interface resources can be achieved, ensuring that the unified interface has sufficient processing power under high load, while effectively saving resources under low load, thereby improving the performance, availability and cost-effectiveness of the unified interface.

[0090] S502, receiving log data sent by the application end node through the target interface node;

[0091] The target interface node reads the input parameters of the application node calling the unified interface of the log center to receive the log data.

[0092] S503, sending the log data to the message middleware;

[0093] S504, determining a target storage node for the log data in a distributed storage system according to a user of the log data defined by the message middleware;

[0094] S505: Using the log data through the target storage node.

[0095] The benefit of this setting is that by load balancing multiple interface nodes to receive log data sent by each application node, the processing capacity and availability of the unified interface can be improved to ensure that no processing bottleneck occurs under high concurrency conditions.

[0096] Embodiment 4

[0097] Figure 7 It is a structural diagram of the log management device of the distributed system provided in Example 4 of the present application.

[0098] like Figure 7 As shown, the device comprises:

[0099] The log data acquisition module 710 is used to respond to the unified interface call request of the toolkit of the application end node to the log center, and receive the log data sent by the application end node through the unified interface; wherein the distributed system includes at least two application end nodes, and the application end nodes are pre-packaged with a toolkit, and the toolkit is used to provide a standard writing format for log data;

[0100] The log data sending module 720 is used to send the log data to the message middleware;

[0101] A storage node determination module 730 is used to determine a target storage node of the log data in a distributed storage system according to a user of the log data defined by the message middleware;

[0102] The log data using module 740 is used to use the log data through the target storage node.

[0103] In an embodiment of the present application, a log data acquisition module is used to respond to a call request from the toolkit of the application end node to the unified interface of the log center, and receive log data sent by the application end node through the unified interface; wherein the distributed system includes at least two application end nodes, and the application end node is pre-packaged with a toolkit, and the toolkit is used to provide a standard writing format for log data; a log data sending module is used to send the log data to the message middleware; a storage node determination module is used to determine the target storage node of the log data in the distributed storage system according to the user of the log data defined by the message middleware; a log data use module is used to use the log data through the target storage node. The log management device of the above-mentioned distributed system receives the log data in the standard writing format sent by the toolkit of each application end node through the unified interface of the log center, which can unify the format of the log data, improve the management efficiency of the log data, and realize the decoupling of the application end node and the log center. By transmitting the log data through the message middleware, the decoupling of the log center and the distributed storage system can be realized, and the log service performance can be improved.

[0104] The log management device of the distributed system in the embodiment of the present application can be a device, or a component, an integrated circuit, or a chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a palmtop computer, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and the non-mobile electronic device can be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0105] The log management device of the distributed system in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0106] The log management device for a distributed system provided in the embodiment of the present application can implement each process implemented in the above-mentioned embodiments one to three, and will not be described again here to avoid repetition.

[0107] Embodiment 5

[0108] like Figure 8 As shown, an embodiment of the present application also provides an electronic device 800, including a processor 801, a memory 802, and a program or instruction stored in the memory 802 and executable on the processor 801. When the program or instruction is executed by the processor 801, each process of the log management method embodiment of the above-mentioned distributed system is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0109] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0110] Embodiment 6

[0111] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the log management method embodiment of the above-mentioned distributed system is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0112] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0113] Embodiment 7

[0114] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned distributed system log management method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0115] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0116] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0117] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0118] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0119] The above are only preferred embodiments of the present application and the technical principles used. The present application is not limited to the specific embodiments described herein, and various obvious changes, readjustments and substitutions that can be made by those skilled in the art will not deviate from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the claims.

Claims

1. A log management method for a distributed system, characterized in that: The method comprises: In response to a call request from a toolkit of an application end node to a unified interface of a log center, log data sent by the application end node is received through the unified interface; wherein the distributed system includes at least two application end nodes, the application end nodes are pre-packaged with a toolkit, and the toolkit is used to provide a standard writing format for log data; Sending the log data to the message middleware; Determine a target storage node for the log data in a distributed storage system according to a user of the log data defined by the message middleware; The log data is used by the target storage node.

2. The log management method for a distributed system according to claim 1, characterized in that: The message middleware adopts kafka middleware.

3. The log management method for a distributed system according to claim 2, characterized in that: Using the log data through the target storage node includes: Pulling the log data from the kafka middleware through the target storage node according to a preset period to store the log data in the target storage node; The usage status of the log data is returned to the kafka middleware through the target storage node.

4. The log management method for a distributed system according to claim 1, characterized in that: The unified interface includes at least two interface nodes; Accordingly, in response to the toolkit of the application end node calling the unified interface request of the log center, receiving the log data sent by the application end node through the unified interface includes: In response to the toolkit of the application end node calling the unified interface request of the log center, the target interface node is determined according to the load status of each interface node; The log data sent by the application end node is received through the target interface node.

5. The log management method for a distributed system according to claim 4, characterized in that: In response to the toolkit of the application end node calling the unified interface request of the log center, the target interface node is determined according to the load status of each interface node, including: In response to the unified interface call request from the toolkit of the application node to the log center, a consistent hashing algorithm is used to determine the target interface node according to the load status of each interface node.

6. The log management method for a distributed system according to claim 4, characterized in that: The method further comprises: In response to an instruction to adjust the number of interface nodes, the at least two interface nodes are added or deleted; wherein the instruction to adjust the number is generated based on the resource space of the distributed system and the load data of the established interface nodes.

7. The log management method for a distributed system according to claim 1, characterized in that: The standard writing format includes one or more of a global identifier, an application end node identifier, log content, a request time, a request IP address, and a log level.

8. A log management device for a distributed system, characterized in that: The device comprises: A log data acquisition module, configured to respond to a call request from a toolkit of an application end node to a unified interface of a log center, and receive log data sent by the application end node through the unified interface; wherein the distributed system includes at least two application end nodes, the application end nodes are pre-packaged with a toolkit, and the toolkit is used to provide a standard writing format for log data; A log data sending module, used for sending the log data to the message middleware; A storage node determination module, used to determine a target storage node of the log data in a distributed storage system according to a user of the log data defined by the message middleware; The log data using module is used to use the log data through the target storage node.

9. An electronic device, characterized in that: The invention comprises a processor, a memory and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the log management method for a distributed system as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the log management method for a distributed system according to any one of claims 1 to 7 are implemented.

Images