Configuration method and device for log management platform, equipment and medium

Abstract

The invention provides a configuration method and device for a log management platform, equipment and a medium, and can be applied to the technical field of computers or the field of financial science and technology. The configuration method for the log management platform comprises the steps of obtaining historical configuration information of a first target acquisition node in response to received target configuration information from a configuration management platform; based on the target configuration information, generating first identification information used for identifying the content of the target configuration information; based on the historical configuration information, second identification information used for identifying the content of the historical configuration information is generated; and comparing the first identification information with the second identification information, responding to the difference between the first identification information and the second identification information, and updating the historical configuration information of the data acquisition node by using the target configuration information, so as to acquire the log data of the target server by using the updated data acquisition node.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology or financial technology, and more specifically to a configuration method, device, equipment, and medium for a log management platform. Background Art

[0002] With the development of information technology, various applications are becoming more and more complex, and the amount of log data generated by the system is also growing explosively. The effective management and analysis of log data is crucial for system monitoring, troubleshooting and security upgrades.

[0003] In related technologies, the configuration and delivery of data collection nodes on the log management platform depends on third-party agent services. It is necessary to connect to the third party corresponding to the agent in advance to ensure the normal operation of the agent. In addition, the configuration and delivery of large-scale data collection nodes need to be delivered sequentially, which is time-consuming and inefficient. Summary of the invention

[0004] In view of the above problems, the present disclosure provides a configuration method, apparatus, device, and medium for a log management platform.

[0005] According to a first aspect of the present disclosure, a configuration method for a log management platform is provided, wherein the log management platform includes a plurality of data collection nodes, and the data collection nodes are used to collect log data of each server end, including: in response to receiving target configuration information from the configuration management platform, obtaining historical configuration information of a first target collection node; wherein the configuration information is obtained by a configuration pull request periodically sent by the data collection node to the configuration management platform, the configuration information corresponds to the IP address of the first target collection node, and the configuration information includes collection configuration information and transmission configuration information; based on the target configuration information, generating first identification information for identifying the content of the target configuration information; based on the historical configuration information, generating second identification information for identifying the content of the historical configuration information; by comparing the first identification information with the second identification information, in response to the first identification information being different from the second identification information, updating the historical configuration information of the data collection node using the target configuration information, so as to collect the log data of the target server end using the updated data collection node.

[0006] According to an embodiment of the present disclosure, the target configuration information is the configuration information of the first target acquisition node, and the method further includes: if the target configuration information is not received within a preset response time, waiting for a preset time to resend a configuration pull request to the configuration management platform.

[0007] According to an embodiment of the present disclosure, the target configuration information is the configuration information of all data acquisition nodes, and the method further includes: if the target configuration information is not received within a preset response time, determining a second target acquisition node based on the IP address of the first target acquisition node, the IP address of the second target acquisition node being in the same network segment as the IP address of the first target acquisition node; and sending a configuration pull request to the second target acquisition node to obtain first configuration information corresponding to the first target acquisition node.

[0008] According to an embodiment of the present disclosure, the above-mentioned determining the second target acquisition node based on the IP address of the above-mentioned first target acquisition node includes: obtaining network segment data corresponding to the above-mentioned first target acquisition node based on the IP address of the above-mentioned first target acquisition node; obtaining network topology data of the above-mentioned network segment data, the above-mentioned network topology data representing the network structure between the above-mentioned data acquisition nodes; and determining the second target acquisition node based on the above-mentioned network topology data.

[0009] According to an embodiment of the present disclosure, the above-mentioned first identification information for identifying the content of the above-mentioned target configuration information is generated based on the above-mentioned target configuration information, including: filling in the above-mentioned target configuration information to obtain the filled target configuration information; grouping the above-mentioned target configuration information based on preset grouping configuration data to obtain multiple groups of to-be-processed grouped data; for each group of to-be-processed grouped data, using preset hash parameters to calculate the above-mentioned to-be-processed grouped data to obtain sub-identification information; and obtaining the above-mentioned first identification information based on all the above-mentioned sub-identification information.

[0010] According to an embodiment of the present disclosure, the above-mentioned log management platform also includes a log storage monitoring module, and the above-mentioned method also includes: in response to receiving a log query request sent by the terminal, obtaining the query log identifier in the above-mentioned query request; querying the above-mentioned log storage module based on the above-mentioned query log identifier to obtain target log data, the above-mentioned target log data has a corresponding relationship with the above-mentioned query identifier, and the above-mentioned target log data includes updated log data and historical log data; sending the above-mentioned target log data to the above-mentioned terminal, so that the target user determines the configuration update result of the above-mentioned data collection node based on the above-mentioned updated log data and the above-mentioned historical log data.

[0011] According to an embodiment of the present disclosure, first identification information for identifying the content of the above-mentioned target configuration information is generated based on any one of the MD5 message digest algorithm, SHA-256 secure hash algorithm, SHA-512 secure hash algorithm, Argon2 algorithm and bcrypt algorithm.

[0012] The second aspect of the present disclosure provides a configuration method device for a log management platform, wherein the log management platform includes multiple data collection nodes, and the data collection nodes are used to collect log data of each server end, including: an acquisition module, which is used to obtain historical configuration information of a first target collection node in response to receiving target configuration information from the configuration management platform, wherein the configuration information is obtained by a configuration pull request periodically sent by the data collection node to the configuration management platform, the configuration information corresponds to the IP address of the first target collection node, and the configuration information includes collection configuration information and transmission configuration information; a first generation module, which is used to generate first identification information for identifying the content of the target configuration information based on the target configuration information; a second generation module, which is used to generate second identification information for identifying the content of the historical configuration information based on the historical configuration information; an update module, which is used to update the historical configuration information of the data collection node using the target configuration information in response to the first identification information being different from the second identification information, so as to collect the log data of the target server end using the updated data collection node.

[0013] According to an embodiment of the present disclosure, the above-mentioned target configuration information is the configuration information of the first target acquisition node, and the above-mentioned device also includes: a resending request module, which is used to wait for a preset time to resend a configuration pull request to the above-mentioned configuration management platform when the above-mentioned target configuration information is not received within a preset response time.

[0014] According to an embodiment of the present disclosure, the target configuration information is the configuration information of all data acquisition nodes, and the device further includes: a second target acquisition node determination module, which is used to determine the second target acquisition node based on the IP address of the first target acquisition node when the target configuration information is not received within a preset response time, and the IP address of the second target acquisition node is in the same network segment as the IP address of the first target acquisition node; and a second configuration pulling module, which is used to send a configuration pulling request to the second target acquisition node to obtain the first configuration information corresponding to the first target acquisition node.

[0015] According to an embodiment of the present disclosure, the above-mentioned second target acquisition node determination module includes: a network segment data acquisition submodule, which is used to obtain the network segment data corresponding to the above-mentioned first target acquisition node based on the IP address of the above-mentioned first target acquisition node; a topology data acquisition submodule, which is used to obtain the network topology structure data of the above-mentioned network segment data, and the above-mentioned network topology structure data characterizes the network structure between the above-mentioned data acquisition nodes; a second determination submodule, which is used to determine the second target acquisition node based on the above-mentioned network topology structure data.

[0016] According to an embodiment of the present disclosure, the above-mentioned first generation module includes: a filling submodule, which is used to fill the above-mentioned target configuration information to obtain the filled target configuration information; a grouping submodule, which is used to group the above-mentioned target configuration information based on preset grouping configuration data to obtain multiple groups of to-be-processed grouped data; a calculation submodule, which is used to calculate the above-mentioned to-be-processed grouped data for each group of to-be-processed grouped data using preset hash parameters to obtain sub-identification information; a first identification information submodule, which is used to obtain the above-mentioned first identification information based on all the above-mentioned sub-identification information.

[0017] According to an embodiment of the present disclosure, the above-mentioned log management platform also includes a log storage monitoring module, and the above-mentioned device also includes: a query log identifier acquisition module, which is used to obtain the query log identifier in the above-mentioned query request in response to receiving a log query request sent by the terminal; a log query module, which is used to query the above-mentioned log storage module based on the above-mentioned query log identifier to obtain target log data, and the above-mentioned target log data has a corresponding relationship with the above-mentioned query identifier, and the above-mentioned target log data includes updated log data and historical log data; a sending module, which is used to send the above-mentioned target log data to the above-mentioned terminal, so that the target user determines the configuration update result of the above-mentioned data collection node based on the above-mentioned updated log data and the above-mentioned historical log data.

[0018] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0019] The fourth aspect of the present disclosure further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the above computer program or instructions are executed by a processor.

[0020] The fifth aspect of the present disclosure further provides a computer program product, including a computer program or instructions, which implement the steps of the above method when the above computer program or instructions are executed by a processor.

[0021] According to the embodiments of the present disclosure, multiple data acquisition nodes periodically send configuration pull requests to the configuration management platform, obtain target configuration information to generate first identification information, obtain historical configuration information to generate second identification information, compare the first identification information and the second identification information, and when the first identification information and the second identification information are different, use the target configuration information to update the historical configuration information of the data acquisition node, and request configuration data in parallel through multiple data acquisition nodes, thereby reducing dependence on third-party proxy services, saving time for configuration changes, and improving the efficiency of data changes when large-scale configurations are issued. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above contents and other purposes, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0023] Figure 1 The following schematically shows an application scenario diagram of a configuration method and device for a log management platform according to an embodiment of the present disclosure;

[0024] Figure 2 A flowchart of a configuration method for a log management platform according to an embodiment of the present disclosure is schematically shown;

[0025] Figure 3 A schematic diagram of a log management platform according to an embodiment of the present disclosure is schematically shown;

[0026] Figure 4 The structure block diagram of the configuration device for the log management platform according to the embodiment of the present disclosure is schematically shown;

[0027] Figure 5 A block diagram of an electronic device suitable for implementing a configuration method for a log management platform according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0028] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0029] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0030] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0031] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0032] It should be noted that in the embodiments of the present disclosure, certain software, components, models and other existing solutions in the industry may be mentioned, which should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present disclosure, but it does not mean that the applicant has or will necessarily use the solution.

[0033] In the technical solution of the present disclosure, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0034] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided by the embodiments of the present disclosure provide users with corresponding operation portals for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating a person's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0035] With the rapid development of information technology, various application systems are becoming more and more complex, and the amount of log data generated by the system is also growing explosively. Effective management and analysis of log data are crucial for system monitoring, troubleshooting, and security auditing. In related technologies, when faced with the management and configuration changes of large-scale data collection nodes, the distribution of configuration files depends on third-party agent services, which not only requires pre-connection with third-party services, but also requires the normal operation of third-party agents on each node, increasing the complexity of system management. Secondly, the sequential configuration distribution of large-scale nodes is inefficient and time-consuming.

[0036] In view of this, an embodiment of the present disclosure provides a configuration method for a log management platform, wherein the log management platform includes multiple data collection nodes, and the data collection nodes are used to collect log data from each server end. The method includes: in response to receiving target configuration information from the configuration management platform, obtaining historical configuration information of a first target collection node; wherein the configuration information is obtained by a configuration pull request periodically sent by the data collection node to the configuration management platform, the configuration information corresponds to the IP address of the first target collection node, and the configuration information includes collection configuration information and transmission configuration information; based on the target configuration information, generating first identification information for identifying the content of the target configuration information; based on the historical configuration information, generating second identification information for identifying the content of the historical configuration information; by comparing the first identification information and the second identification information, in response to the first identification information being different from the second identification information, using the target configuration information to update the historical configuration information of the data collection node, so as to use the updated data collection node to collect the log data of the target server end.

[0037] Figure 1 The application scenario diagram of the configuration method and device for the log management platform according to the embodiment of the present disclosure is schematically shown.

[0038] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0039] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0040] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0041] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0042] It should be noted that the configuration method for the log management platform provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the configuration device for the log management platform provided in the embodiment of the present disclosure can generally be set in the server 105. The configuration method for the log management platform provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the configuration device for the log management platform provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0043] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0044] The following will be based on Figure 1 The scene described by Figure 2~Figure 3 The configuration method for the log management platform of the disclosed embodiment is described in detail.

[0045] Figure 2 A flowchart of a configuration method for a log management platform according to an embodiment of the present disclosure is schematically shown.

[0046] like Figure 2 As shown, the configuration method for the log management platform of this embodiment includes operations S210 to S240.

[0047] In operation S210 , in response to receiving target configuration information from a configuration management platform, historical configuration information of a first target acquisition node is acquired.

[0048] The configuration information is obtained by periodically sending a configuration pull request from the data collection node to the configuration management platform. The configuration information corresponds to the IP address of the first target collection node. The configuration information includes collection configuration information and transmission configuration information.

[0049] Figure 3 A schematic diagram of a log management platform according to an embodiment of the present disclosure is schematically shown.

[0050] like Figure 3 As shown, the data collection node on the server is used to collect file logs, syslog (system log) and http interface logs on server A. The log management platform includes data collection nodes, log cache modules, log processing modules and log storage modules, wherein the data collection nodes are distributed on each server, the log cache module, log processing module and log storage module are located on some nodes of the above-mentioned log management platform, the data collection node sends the collected log content to the log cache module for caching, the log processing module extracts the data in the log cache module for filtering, parsing and conversion, and transmits the processed data to the log storage module. The log storage module saves the log data transmitted by the log processing module to the configured data node. The log query platform provides a front-end interface to realize the functions of querying, monitoring, and alarming the log data stored in the log storage node.

[0051] According to an embodiment of the present disclosure, the above target configuration information can be obtained by the user configuring the data acquisition node based on the configuration management platform. After the configuration is completed, the configuration content is stored in the database, waiting for the configuration pull request of the data acquisition node.

[0052] According to the embodiment of the present disclosure, after the target configuration information is acquired, it can be stored in the memory variable of the server.

[0053] According to an embodiment of the present disclosure, the above-mentioned collection configuration information may, for example, include data such as log input type, monitored log file path, file encoding, document type, etc. The transmission configuration information may, for example, be the log data sending target. The configuration information may also include data such as log level, log, configuration and storage path of data files.

[0054] According to the embodiment of the present disclosure, after the data collection node is started, a new thread can be started to periodically pull the latest configuration from the configuration management platform.

[0055] According to an embodiment of the present disclosure, the target configuration information may be, for example, a collection path and an IP address of a collection server.

[0056] In operation S220, first identification information for identifying content of the target configuration information is generated based on the target configuration information.

[0057] In operation S230, second identification information for identifying content of the historical configuration information is generated based on the historical configuration information.

[0058] According to an embodiment of the present disclosure, the above historical configuration information is stored on a server corresponding to the data collection node, and the above first identification information and the second identification information can both uniquely identify the corresponding configuration information.

[0059] In operation S240, by comparing the first identification information and the second identification information, in response to the first identification information being different from the second identification information, the historical configuration information of the data collection node is updated using the target configuration information, so as to use the updated data collection node to collect log data of the target server.

[0060] According to the embodiment of the present disclosure, the target configuration information may be written into a corresponding file of a data acquisition node of a corresponding path, and the data acquisition node may be restarted to make the target configuration information effective.

[0061] According to the embodiments of the present disclosure, multiple data acquisition nodes periodically send configuration pull requests to the configuration management platform, obtain target configuration information to generate first identification information, obtain historical configuration information to generate second identification information, compare the first identification information and the second identification information, and when the first identification information and the second identification information are different, use the target configuration information to update the historical configuration information of the data acquisition node, and request configuration data in parallel through multiple data acquisition nodes, thereby reducing dependence on third-party proxy services, saving time for configuration changes, and improving the efficiency of data changes when large-scale configurations are issued.

[0062] According to an embodiment of the present disclosure, the target configuration information is the configuration information of the first target acquisition node, and the method further includes: if the target configuration information is not received within a preset response time, waiting for a preset time to resend a configuration pull request to the configuration management platform.

[0063] According to an embodiment of the present disclosure, the target configuration information may be the configuration information corresponding to the first target acquisition node. If the target data acquisition node does not receive the target configuration information within the preset response time, it waits for the preset time and then sends a configuration pull request to the configuration management platform again. The preset response time may be, for example, 3s, 5s, 20s, etc., and the present application does not impose any restrictions on the preset response time.

[0064] According to the embodiments of the present disclosure, when the target configuration information is not received within the preset response time, a configuration pull request is resent to the configuration management platform after waiting for the preset time. In an unstable network environment, the data collection node can still obtain the latest configuration information, thereby improving the success rate of the configuration update and ensuring the normal operation of the log management platform.

[0065] According to an embodiment of the present disclosure, the target configuration information is the configuration information of all data acquisition nodes, and the method further includes: if the target configuration information is not received within a preset response time, determining a second target acquisition node based on the IP address of the first target acquisition node, the IP address of the second target acquisition node and the IP address of the first target acquisition node being in the same network segment; and sending a configuration pull request to the second target acquisition node to obtain first configuration information corresponding to the first target acquisition node.

[0066] According to an embodiment of the present disclosure, when the target configuration information is not received within a preset corresponding time period, a new thread can be started to determine the second target acquisition node. Before obtaining the first configuration information sent by the second target acquisition node, the original thread continues to periodically send configuration pull requests to the configuration management platform.

[0067] According to an embodiment of the present disclosure, after the first target acquisition node establishes a communication connection with the second target acquisition node, an interrupt instruction may be sent to the original thread to stop periodically sending a configuration pull request to the configuration management platform.

[0068] According to an embodiment of the present disclosure, determining a second target acquisition node based on an IP address of a first target acquisition node includes: obtaining network segment data corresponding to the first target acquisition node based on the IP address of the first target acquisition node; obtaining network topology data of the network segment data, the network topology data representing a network structure between data acquisition nodes; and determining the second target acquisition node based on the network topology data.

[0069] According to an embodiment of the present disclosure, the configuration information of all data acquisition nodes can be sent to the data acquisition node. When there is a data acquisition node that cannot pull the corresponding configuration information from the configuration management platform, the data acquisition node can obtain the first configuration information corresponding to the target acquisition node from the second target acquisition node in the same network segment.

[0070] According to the embodiment of the present disclosure, when determining the second target collection node, the network segment to which the first target collection node belongs can be first determined based on the IP address of the first target collection node, and the corresponding network topology data can be obtained based on the network segment data. In the network topology data, if there is a collection node that shares a router or switch with the first target collection node, it is determined as the second target collection node.

[0071] According to an embodiment of the present disclosure, after sending a configuration pull request to the second target acquisition node, an alarm message may also be sent to relevant staff to remind the first target acquisition node that it is unable to communicate with the configuration management platform.

[0072] According to the embodiments of the present disclosure, when the configuration information in a network segment is sent down, several collection nodes can be selected to send the configuration information of all data collection nodes, and other nodes send the corresponding configuration information. When some nodes cannot receive the corresponding configuration information, the configuration sending time can be reduced by pulling the configuration data from the collection node (which can be determined as the second target collection node) that has the configuration information of all the collection nodes in the network segment.

[0073] Exemplarily, for a first target collection node with an IP address of 192.169.1.10 and a subnet mask of 255.255.255.0, it is calculated that the network segment of the target collection node is between 192.168.1.0 and 192.168.254, then a request can be sent to all nodes in the network segment and the collection node with the fastest response can be determined as the second target collection node; or a backup configuration sending node in the network segment can be configured for each collection node in the program and the network topology can be stored. If the target configuration information is not received within a preset corresponding time period, the router and switch connection information in the network segment can be determined based on the network topology, and the backup configuration sending nodes can be sorted according to the number of interval routers or switches, and pull requests can be sent to the backup configuration sending nodes on the sorting table in turn until the first configuration information corresponding to the first target collection node is received.

[0074] According to the embodiment of the present disclosure, by sending the data of all data collection nodes to the collection node, when the first target collection node cannot obtain the corresponding first configuration data, the second target collection node can be determined according to the network segment data, and the corresponding first configuration information can be obtained from the second target collection node, thereby improving the reliability of the log management platform, enhancing the platform's fault tolerance, and improving the efficiency of configuration sending.

[0075] According to an embodiment of the present disclosure, the above-mentioned log management platform also includes a log storage monitoring module, and the method also includes: in response to receiving a log query request sent by a terminal, obtaining a query log identifier in the query request; querying the log storage module based on the query log identifier to obtain target log data, the target log data has a corresponding relationship with the query identifier, and the target log data includes updated log data and historical log data; sending the target log data to the terminal, so that the target user determines the configuration update result of the data acquisition node based on the updated log data and the historical log data.

[0076] According to the embodiments of the present disclosure, after the configuration information of the target data acquisition node is updated, the target log data of the target data acquisition node can be found by querying the log identifier such as the IP address or name of the target data acquisition node. By sending the updated log data and the historical log data to the target user, the target user can determine whether the configuration information has been successfully sent based on the updated log data and the historical log data.

[0077] Exemplarily, the collection frequency of the target data collection node is changed from every 4 seconds before the update to every second. The updated log data received by the target user may be, for example, "1640995200, A"; "1640995201, B", where "1640995200" and "1640995201" are Unix timestamps, and "A" and "B" are log data. It can be seen that the configuration has been successfully delivered and the collection frequency has been changed to every second.

[0078] According to an embodiment of the present disclosure, first identification information for identifying the content of target configuration information is generated based on any one of the MD5 message digest algorithm, SHA-256 secure hash algorithm, SHA-512 secure hash algorithm, Argon2 algorithm and bcrypt algorithm.

[0079] According to an embodiment of the present disclosure, based on the target configuration information, first identification information for identifying the content of the target configuration information is generated, including: filling in the target configuration information to obtain the filled target configuration information; grouping the target configuration information based on preset grouping configuration data to obtain multiple groups of to-be-processed grouped data; for each group of to-be-processed grouped data, using preset hash parameters to calculate the to-be-processed grouped data to obtain sub-identification information; and obtaining the first identification information based on all the sub-identification information.

[0080] According to the embodiment of the present disclosure, taking the MD5 message digest algorithm as an example, when generating the first identification information for identifying the content of the target configuration information based on the target configuration information, because the MD5 message digest algorithm processes 512 bits as a group, the target configuration information must first be padded to a multiple of 512 bits, and a "1" is first padded followed by k "0s" during the padding, where k satisfies (n+1+k) mod 512 = 448, and finally append 64 bits of data length, divide the filled target configuration information into multiple 512-bit groups, and initialize 4 preset hash parameters, then split the 512-bit group into 16 32-bit sub-groups, and start the main loop. The main loop has four rounds, and each round is very similar. Take the operation in the first round as an example. In the first round, 16 operations are performed. In each operation, a nonlinear function operation is performed on 3 of the above 4 preset hash parameters, and then the result is added to the 4th parameter that does not participate in the operation, the i-th sub-group and the constant value, and then an indefinite number is shifted to the left and a preset hash parameter is replaced by the result. Based on these 4 hash parameters, the processed group data is transformed in turn to obtain sub-identification information (that is, the 4 hash parameters after 4 rounds of main loops), and finally based on all the sub-identification information, the first identification information that can uniquely identify the target configuration information is obtained. The above first identification information can be, for example, 128-bit hash value data.

[0081] According to the embodiments of the present disclosure, by generating first identification information that uniquely identifies the target configuration information, it is possible to quickly determine whether the target configuration information has changed compared to the historical configuration information. Furthermore, by filling the data and dividing it into groups for efficient cyclic transformation processing, the performance of the hardware device can be fully utilized, and the time for generating the first identification information and the second identification information can be effectively reduced, thereby reducing the configuration change time and improving efficiency.

[0082] Based on the above configuration method for the log management platform, the present disclosure also provides a configuration device for the log management platform. Figure 4 The device is described in detail.

[0083] Figure 4 The structural block diagram of a configuration device for a log management platform according to an embodiment of the present disclosure is schematically shown.

[0084] like Figure 4 As shown, the configuration device 400 for the log management platform of this embodiment includes an acquisition module 410 , a first generation module 420 , a second generation module 430 and an update module 440 .

[0085] The acquisition module 410 is used to obtain the historical configuration information of the first target acquisition node in response to receiving the target configuration information from the configuration management platform, wherein the configuration information is obtained by the configuration pull request sent by the data acquisition node to the configuration management platform at regular intervals, the configuration information corresponds to the IP address of the first target acquisition node, and the configuration information includes acquisition configuration information and transmission configuration information. In one embodiment, the acquisition module 410 can be used to perform the operation S210 described above, which will not be repeated here.

[0086] The first generating module 420 is used to generate first identification information for identifying the content of the target configuration information based on the target configuration information. In one embodiment, the first generating module 420 can be used to perform the operation S220 described above, which will not be described in detail here.

[0087] The second generation module 430 is used to generate second identification information for identifying the content of the historical configuration information based on the historical configuration information. In one embodiment, the second generation module 430 can be used to perform the operation S230 described above, which will not be described in detail here.

[0088] The update module 440 is used to compare the first identification information with the second identification information, and in response to the first identification information being different from the second identification information, update the historical configuration information of the data collection node using the target configuration information, so as to collect the log data of the target server using the updated data collection node. In one embodiment, the update module 440 can be used to perform the operation S240 described above, which will not be described in detail here.

[0089] According to an embodiment of the present disclosure, the above-mentioned target configuration information is the configuration information of the first target acquisition node, and the above-mentioned device also includes: a resending request module, which is used to wait for a preset time to resend a configuration pull request to the above-mentioned configuration management platform when the above-mentioned target configuration information is not received within a preset response time.

[0090] According to an embodiment of the present disclosure, the target configuration information is the configuration information of all data acquisition nodes, and the device further includes: a second target acquisition node determination module, which is used to determine the second target acquisition node based on the IP address of the first target acquisition node when the target configuration information is not received within a preset response time, and the IP address of the second target acquisition node is in the same network segment as the IP address of the first target acquisition node; and a second configuration pulling module, which is used to send a configuration pulling request to the second target acquisition node to obtain the first configuration information corresponding to the first target acquisition node.

[0091] According to an embodiment of the present disclosure, the above-mentioned second target acquisition node determination module includes: a network segment data acquisition submodule, which is used to obtain the network segment data corresponding to the above-mentioned first target acquisition node based on the IP address of the above-mentioned first target acquisition node; a topology data acquisition submodule, which is used to obtain the network topology structure data of the above-mentioned network segment data, and the above-mentioned network topology structure data characterizes the network structure between the above-mentioned data acquisition nodes; a second determination submodule, which is used to determine the second target acquisition node based on the above-mentioned network topology structure data.

[0092] According to an embodiment of the present disclosure, the above-mentioned first generation module includes: a filling submodule, which is used to fill the above-mentioned target configuration information to obtain the filled target configuration information; a grouping submodule, which is used to group the above-mentioned target configuration information based on preset grouping configuration data to obtain multiple groups of to-be-processed grouped data; a calculation submodule, which is used to calculate the above-mentioned to-be-processed grouped data for each group of to-be-processed grouped data using preset hash parameters to obtain sub-identification information; a first identification information submodule, which is used to obtain the above-mentioned first identification information based on all the above-mentioned sub-identification information.

[0093] According to an embodiment of the present disclosure, the above-mentioned log management platform also includes a log storage monitoring module, and the above-mentioned device also includes: a query log identifier acquisition module, which is used to obtain the query log identifier in the above-mentioned query request in response to receiving a log query request sent by the terminal; a log query module, which is used to query the above-mentioned log storage module based on the above-mentioned query log identifier to obtain target log data, and the above-mentioned target log data has a corresponding relationship with the above-mentioned query identifier, and the above-mentioned target log data includes updated log data and historical log data; a sending module, which is used to send the above-mentioned target log data to the above-mentioned terminal, so that the target user determines the configuration update result of the above-mentioned data collection node based on the above-mentioned updated log data and the above-mentioned historical log data.

[0094] According to an embodiment of the present disclosure, any multiple modules of the acquisition module 410, the first generation module 420, the second generation module 430, and the update module 440 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the acquisition module 410, the first generation module 420, the second generation module 430, and the update module 440 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, at least one of the acquisition module 410 , the first generation module 420 , the second generation module 430 , and the update module 440 may be at least partially implemented as a computer program module, and when the computer program module is executed, a corresponding function may be performed.

[0095] Figure 5 A block diagram of an electronic device suitable for implementing a configuration method for a log management platform according to an embodiment of the present disclosure is schematically shown.

[0096] like Figure 5 As shown, the electronic device 500 according to an embodiment of the present disclosure includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 to a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include an onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0097] In RAM 503, various programs and data required for the operation of electronic device 500 are stored. Processor 501, ROM 502 and RAM 503 are connected to each other via bus 504. Processor 501 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 502 and / or RAM 503. It should be noted that the program can also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0098] According to an embodiment of the present disclosure, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device 500 may further include one or more of the following components connected to the input / output (I / O) interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 508 including a hard disk, etc.; and a communication portion 509 including a network interface card such as a LAN card, a modem, etc. The communication portion 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage portion 508 as needed.

[0099] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0100] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 502 and / or RAM 503 described above and / or one or more memories other than ROM 502 and RAM 503.

[0101] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program includes a program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the configuration method for the log management platform provided by the embodiment of the present disclosure.

[0102] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 501. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0103] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 509, and / or installed from the removable medium 511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0104] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the processor 501, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0105] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0106] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0107] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0108] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A configuration method for a log management platform, characterized in that: The log management platform includes a plurality of data collection nodes, and the data collection nodes are used to collect log data of each server. The method includes: In response to receiving target configuration information from the configuration management platform, acquiring historical configuration information of the first target acquisition node; wherein the configuration information is acquired by a configuration pull request periodically sent by the data acquisition node to the configuration management platform, the configuration information corresponds to the IP address of the first target acquisition node, and the configuration information includes acquisition configuration information and transmission configuration information; Based on the target configuration information, generating first identification information for identifying the content of the target configuration information; Based on the historical configuration information, generating second identification information for identifying the content of the historical configuration information; By comparing the first identification information and the second identification information, in response to the first identification information being different from the second identification information, the historical configuration information of the data collection node is updated using the target configuration information, so as to use the updated data collection node to collect log data of the target server.

2. The method according to claim 1, characterized in that The target configuration information is configuration information of a first target acquisition node, and the method further includes: If the target configuration information is not received within the preset response time, wait for a preset time and resend a configuration pull request to the configuration management platform.

3. The method according to claim 1, characterized in that The target configuration information is the configuration information of all data acquisition nodes, and the method further includes: If the target configuration information is not received within a preset response time, determining a second target collection node based on the IP address of the first target collection node, wherein the IP address of the second target collection node is in the same network segment as the IP address of the first target collection node; Send a configuration pull request to the second target acquisition node to obtain first configuration information corresponding to the first target acquisition node.

4. The method according to claim 3, characterized in that The determining the second target collection node based on the IP address of the first target collection node includes: Obtaining network segment data corresponding to the first target collection node based on the IP address of the first target collection node; Acquire network topology data of the network segment data, wherein the network topology data represents the network structure between the data collection nodes; A second target acquisition node is determined based on the network topology data.

5. The method according to claim 1, characterized in that The step of generating first identification information for identifying content of the target configuration information based on the target configuration information includes: Filling the target configuration information to obtain filled target configuration information; Grouping the target configuration information based on preset grouping configuration data to obtain multiple groups of grouping data to be processed; For each group of to-be-processed packet data, the to-be-processed packet data is calculated using a preset hash parameter to obtain sub-identification information; The first identification information is obtained based on all the sub-identification information.

6. The method according to claim 1, characterized in that The log management platform also includes a log storage monitoring module, and the method also includes: In response to receiving a log query request sent by a terminal, obtaining a query log identifier in the query request; Based on the query log identifier, the log storage module is queried to obtain target log data, the target log data has a corresponding relationship with the query identifier, and the target log data includes updated log data and historical log data; The target log data is sent to the terminal, so that a target user determines a configuration update result of the data collection node based on the updated log data and the historical log data.

7. The method according to claim 1, characterized in that The first identification information for identifying the content of the target configuration information is generated based on any one of the MD5 message digest algorithm, the SHA-256 secure hash algorithm, the SHA-512 secure hash algorithm, the Argon2 algorithm and the bcrypt algorithm.

8. A configuration method and device for a log management platform, characterized in that: The log management platform includes a plurality of data collection nodes, and the data collection nodes are used to collect log data of each server. The device includes: an acquisition module, configured to acquire historical configuration information of a first target acquisition node in response to receiving target configuration information from a configuration management platform, wherein the configuration information is acquired by a configuration pull request periodically sent by a data acquisition node to the configuration management platform, the configuration information corresponds to an IP address of the first target acquisition node, and the configuration information includes acquisition configuration information and transmission configuration information; A first generating module, configured to generate first identification information for identifying content of the target configuration information based on the target configuration information; A second generating module, configured to generate second identification information for identifying content of the historical configuration information based on the historical configuration information; An update module is used to compare the first identification information with the second identification information, and in response to the first identification information being different from the second identification information, use the target configuration information to update the historical configuration information of the data acquisition node, so as to use the updated data acquisition node to collect log data of the target server.

9. An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images