Client observation method and system based on log data

Abstract

The invention provides a client observation method and system based on log data, and belongs to the technical field of service observation.The client observation method comprises the steps that log data on a client device collecting end is compressed, and compressed log data is obtained; the server decompresses the compressed log data through the log service component, performs data format verification, and obtains decompressed data after verification is passed; the distributed queue component encapsulates the decompressed data and converts the decompressed data into log stream data; establishing an index directory based on log stream data stored in the distributed queue component; and based on a preset analysis index, performing analysis according to the log stream data to obtain log analysis data. According to the technical scheme, intelligent analysis of the log stream data is realized by establishing the preset analysis index, and a worker can set the preset analysis index and access the log stream data according to a business demand so as to support a more detailed client index and business index scene and meet a user-defined observation demand of an actual scene.

Description

Technical Field

[0001] The present invention relates to the technical field of service observation, and in particular to a client observation method and system based on log data. Background Art

[0002] Log data refers to various types of data generated by events during the operation of applications, middleware, and machines. By observing various types of log data on the device, it is possible to achieve real-time observation of the status of the device and the applications running on it.

[0003] In traditional technology, log data on client devices is observed by referencing general observation software tools, and the server performs data analysis on the various types of log data obtained to obtain log analysis results. In actual log data observation, the server cannot effectively identify different types of log data, resulting in the inability to conduct a comprehensive analysis of log data, and general observation software tools cannot achieve intelligent analysis of log data, and cannot provide effective support for more detailed client indicators and business indicators, making it difficult to meet the customized observation needs of actual scenarios.

[0004] Therefore, a client observation method and system based on log data are proposed. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a client observation method and system based on log data, which is used to solve the problem that traditional technology cannot perform intelligent analysis on log data and is difficult to meet customized observation needs.

[0006] An embodiment of the present invention provides a client observation method based on log data, including:

[0007] Collect log data on the client side through the client device, compress it, and transmit the compressed log data to the server side;

[0008] The server decompresses the compressed log data through the log service component and performs data format verification. After the verification passes, the decompressed data is transmitted to the distributed queue component;

[0009] The distributed queue component encapsulates the decompressed data and converts it into log stream data;

[0010] Establishing an index directory based on the log stream data stored in the distributed queue component;

[0011] Based on preset analysis indicators, the log stream data is analyzed to obtain log analysis data.

[0012] Preferably, the present invention provides a client observation method based on log data, the steps of: collecting log data on the client through a client device, compressing the data, and transmitting the compressed log data to the server; including:

[0013] Obtaining a network log by recording network request information of the client device;

[0014] Obtaining a log by recording operation information of the user based on the client device;

[0015] Obtaining an error log by recording operation error information of the client device;

[0016] Obtaining an execution log by recording event operation information and event troubleshooting information of the client device;

[0017] The log data includes: network log, dot log, error log and execution log;

[0018] Acquire corresponding meta information data according to the log data, compress the log data and the meta information data, generate compressed log data and save it locally on the client device;

[0019] The meta information data includes the application name, log type and log level corresponding to the log data.

[0020] The client device detects the current network transmission rate, and when the network transmission rate is higher than a preset rate threshold, transmits the compressed log data to the server; when the network transmission rate is lower than the preset rate threshold, delays the transmission operation of the compressed log data.

[0021] Preferably, the present invention provides a client observation method based on log data, the steps of: analyzing the log stream data based on preset analysis indicators to obtain log analysis data; and further comprising:

[0022] When effective analysis cannot be performed based on the log stream data, sending a detailed log acquisition request to the client device through the server;

[0023] The client device obtains the detailed log data to the server according to the detailed log acquisition request and the user's request operation based on the client device;

[0024] The server performs data format verification on the detailed log data, and after the verification is passed, the distributed queue component converts the detailed log data into detailed log stream data for indicator analysis to obtain log analysis data.

[0025] Preferably, the present invention provides a client observation method based on log data, the steps of: the server decompresses the compressed log data through a log service component, performs data format verification, and transmits the decompressed data to a distributed queue component after the verification passes; including:

[0026] The server performs decompression processing on the compressed log data transmitted by the client device through the log service component to obtain decompressed data;

[0027] A format check is performed based on the decompressed data. When the data format of the decompressed data is a preset data format, the decompressed data is transmitted to the distributed queue component. When the data format of the decompressed data is not a preset data format, the compressed log data is refused to be received, and a prompt message is returned to the client device.

[0028] Preferably, the present invention provides a client observation method based on log data, the steps of: the distributed queue component encapsulates the decompressed data and converts it into log stream data; comprising:

[0029] Construct a distributed queue component, obtain a set of configuration parameters of the distributed queue component, perform throughput detection and operation resolution time test under a preset workload, and obtain the optimal configuration parameters of the distributed queue component for configuration;

[0030] The distributed queue component stores the decompressed data in a partition unit under a corresponding topic subcomponent in the distributed queue component according to a log topic of the decompressed data;

[0031] The partition unit obtains the application name, log name, log level and client IP information in the decompressed data, and generates metadata; and constructs log stream data according to the metadata and the decompressed data and stores it in the partition unit.

[0032] Preferably, the present invention provides a client observation method based on log data, the steps of: constructing a distributed queue component, obtaining a configuration parameter set of the distributed queue component, performing a throughput detection and a running solution time test under a preset workload, and obtaining the optimal configuration parameters of the distributed queue component for configuration; comprising:

[0033] Obtaining a set of configuration parameters of the distributed queue component;

[0034] Dividing each configuration parameter in the configuration parameter set into a plurality of non-overlapping intervals, and constructing a cube parameter set according to the plurality of intervals of the configuration parameters;

[0035] Based on the cube parameter set, only one sampling point is randomly selected in the interval of each configuration parameter to construct the training configuration parameter;

[0036] The parameters of the distributed queue component are configured according to the training configuration parameters, and the throughput of the distributed queue component for a preset workload based on the training configuration parameters is calculated. After repeating the above steps for a preset number of times, the training configuration parameters with a throughput higher than a preset standard throughput are selected to construct a training configuration parameter set;

[0037] The attention mechanism layer configures the training parameter set P = [p1, ..., p n ] to adjust the attention mechanism, input the generator built based on the feedforward neural network for processing, and output n sets of generator parameter sets Q = [q1,…,q n ];

[0038] Constructing a discriminator based on the first stochastic gradient function;

[0039]

[0040] in, is the gradient operator of the discriminator. There are n sets of training configuration parameters in the training configuration parameter set. i ) is the discriminator D based on the i-th group of training configuration parameters p in the training configuration parameter set P i Detection of G(q i ) is the generator G based on the training configuration parameter p i The resulting generator parameter q i ;

[0041] The discriminator performs parameter detection according to the training configuration parameters and the generator parameters, and when the discriminator cannot effectively distinguish the training configuration parameters from the generator parameters, obtains the generator parameters as network configuration parameters;

[0042] The generator is updated based on the second stochastic gradient function and the generator parameters are re-output;

[0043]

[0044] Repeat the above steps, and the discriminator detects several groups of generator parameters generated by the generator, obtains network configuration parameters, and constructs a network configuration parameter set;

[0045] Constructing a component parameter optimization set based on the network configuration parameter set and the training configuration parameter set;

[0046] Under a preset workload, the running solution time of the distributed queue component based on the component parameters in the component parameter optimization set is calculated, and the component parameters corresponding to the shortest running solution time are obtained as the optimal configuration parameters.

[0047] Preferably, the present invention provides a client observation method based on log data, the steps of: establishing an index directory based on the log stream data stored in the distributed queue component; comprising:

[0048] The distributed queue component is constructed based on the service management node, and counts the task load of the service management node;

[0049] The partition unit processes the decompressed data to obtain log stream data, performs a backup operation to obtain log stream backup data, and stores the log stream data and the log stream backup data in the partition unit;

[0050] According to the task load of the service management node in the distributed queue component, the partition unit is intelligently deployed based on the service management node;

[0051] The partition unit generates a log summary according to the log stream data, obtains a node address of a server management node where the partition unit is located, and constructs log index data;

[0052] An index directory is constructed by acquiring log index data of all partition units in the distributed queue component.

[0053] Preferably, the present invention provides a client observation method based on log data, the steps of: analyzing the log stream data based on preset analysis indicators to obtain log analysis data; comprising:

[0054] According to the index directory, obtaining log stream data in the distributed queue component;

[0055] Performing anomaly detection on the log stream data, and when it is detected that the log stream data has no anomaly, using the log stream data for indicator analysis;

[0056] Based on preset analysis indicators, a data analysis template is constructed, the log stream data is imported into the data analysis template for calculation, and log analysis data is obtained; when the log stream data changes, the log analysis data is updated in real time.

[0057] Preferably, the present invention provides a client observation method based on log data, the steps of: performing anomaly detection on the log stream data, and when it is detected that the log stream data has no anomaly, using the log stream data for indicator analysis; comprising:

[0058] Obtain log stream data for vectorization processing to generate log vector data;

[0059] Constructing a time relationship vector network according to the time vector elements in the log vector data;

[0060] Analyze the influence of adjacent log vector data on the log vector data in the temporal relationship vector network, and perform attention optimization to obtain a preprocessed log vector;

[0061]

[0062] Among them, T′ j For the j-th log vector data T j The preprocessed log vector obtained by processing is assumed to be the log vector data T j Neighborhood T(T j -δ,T j +δ) has N adjacent log vector data, σ is the first activation function, ∏() is the vector concatenation operator, θ(T i ,T j ,t′ i,j ) is the log vector data T j With adjacent log vector data T i The connection weight, θ(T i ,T j ,t′ i,j ) i is the log vector data T j With all adjacent log vector data T i The connection weight iteration, t′ i,j is the log vector data T j With adjacent log vector data T i The normalized log time difference of is the linear transformation matrix;

[0063] θ(T i ,T j ,t′ i,j )=tanh[(T i F T ) T T j +F t t′ i,j ]

[0064] Among them, tanh is the second activation function, F T is the vector transformation matrix, F t is the time transformation matrix;

[0065] t i,j =t i -tj

[0066]

[0067] Among them, t i is the adjacent log vector data T i The time vector element, t j is the log vector data T j The time vector element, t i,j is the adjacent log vector data T i With log vector data T j The log time difference, t max is the maximum log time difference in the temporal relationship vector network, t min is the minimum log time difference in the time relationship vector network;

[0068] The preprocessed log vector is input into the anomaly detection model to perform anomaly detection, and the anomaly detection result of the log stream data is obtained.

[0069] The present invention provides a client observation system based on log data, comprising:

[0070] The data collection module is used to collect the log data on the client device, compress it, and transmit the compressed log data to the server;

[0071] A data verification module is used for the server to decompress the compressed log data through the log service component and perform data format verification, and transmit the decompressed data to the distributed queue component after the verification passes;

[0072] A data encapsulation module, used to encapsulate the decompressed data through the distributed queue component and convert it into log stream data;

[0073] A data indexing module, used to establish an index directory based on the log stream data stored in the distributed queue component;

[0074] The data analysis module is used to analyze the log stream data based on preset analysis indicators to obtain log analysis data.

[0075] Compared with the traditional technology, the beneficial effects of the present invention are: the log data on the client device is collected, the compressed log data is transmitted to the server, the server decompresses the data, and after checking the data format, transmits it to the distributed queue component to obtain the log stream data, thereby realizing the log data collection and storage of the client device; an index directory is established based on the log stream data, which is convenient for the staff and the query and acquisition of the log stream data during data analysis, and by establishing preset analysis indicators, intelligent analysis of the log stream data is realized, and log analysis data is obtained; based on the client observation method of log data and the server in the system checks the data format of the decompressed data and then transmits it to the distributed queue component to obtain the log stream data for indicator analysis, which solves the problem that different types of log data cannot be effectively identified in the traditional technology, and by establishing preset analysis indicators, intelligent analysis of the log stream data is realized, and the staff can set preset analysis indicators according to business needs and access the log stream data to support more detailed client indicators and business indicator scenarios, thereby meeting the customized observation needs of the actual scenario.

[0076] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0077] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 A schematic diagram of a process flow of a client observation method based on log data provided by the present invention;

[0079] Figure 2 A schematic diagram of the structure of a client observation system based on log data provided by the present invention. DETAILED DESCRIPTION

[0080] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0081] Embodiment 1:

[0082] The embodiment of the present invention provides a client observation method based on log data, referring to Figure 1 ,include:

[0083] Collect log data on the client side through the client device, compress it, and transmit the compressed log data to the server side;

[0084] The server uses the log service component to decompress the compressed log data and perform data format verification. After the verification passes, the decompressed data is transmitted to the distributed queue component.

[0085] The distributed queue component encapsulates the decompressed data and converts it into log stream data;

[0086] Create an index directory based on the log stream data stored in the distributed queue component;

[0087] Based on the preset analysis indicators, analyze the log stream data to obtain log analysis data.

[0088] In the above embodiments, the client device collects log data on the end and compresses it to obtain compressed log data; the server decompresses the compressed log data through the log service component and performs data format verification. After the verification, the decompressed data is transmitted to the distributed queue component, and the distributed queue component encapsulates the decompressed data and converts it into log stream data; an index directory is established based on the log stream data stored in the distributed queue component, and based on preset analysis indicators, analysis is performed according to the log stream data to obtain log analysis data.

[0089] In the above embodiments, based on the preset analysis index, the corresponding log stream data analysis is obtained according to the index directory to obtain the log analysis data.

[0090] In the above embodiments, the preset analysis indicators are set by the staff according to the log analysis requirements. The staff can write indicator analysis programs according to business needs, access log stream data for data analysis to meet monitoring and reporting requirements, and update in real time.

[0091] The beneficial effects of the above technology are: the client device collects log data on the other end, transmits the compressed log data to the server, the server decompresses it, and after checking the data format, transmits it to the distributed queue component to obtain log stream data, thereby realizing the collection and storage of log data for the client device; an index directory is established based on the log stream data, which is convenient for staff and for querying and obtaining log stream data during data analysis, and by establishing preset analysis indicators, the method realizes intelligent analysis of log stream data and obtains log analysis data; compared with traditional technology, in the above method, the server verifies the data format of the decompressed data and then transmits it to the distributed queue component to obtain log stream data for indicator analysis, which solves the problem that different types of log data cannot be effectively identified in traditional technology, and by establishing preset analysis indicators, the method realizes intelligent analysis of log stream data, and staff can set preset analysis indicators according to business needs and access log stream data to support more detailed client indicators and business indicator scenarios, thereby meeting the customized observation needs of actual scenarios.

[0092] Embodiment 2:

[0093] The embodiment of the present invention provides a client observation method based on log data, comprising the steps of collecting log data on the client through a client device, compressing the log data, and transmitting the compressed log data to the server; comprising:

[0094] Obtain network logs by recording network request information of client devices;

[0095] Obtain the management log by recording the user's operation information based on the client device;

[0096] Obtain error logs by recording the operation error information of the client device;

[0097] Obtain execution logs by recording event operation information and event troubleshooting information of client devices;

[0098] Log data, including: network logs, RBI logs, error logs, and execution logs;

[0099] Obtain corresponding meta information data according to the log data, compress the log data and the meta information data, generate compressed log data and save it locally on the client device;

[0100] Meta information data, including the application name, log type, and log level corresponding to the log data.

[0101] The client device detects the current network transmission rate, and when the network transmission rate is higher than the preset rate threshold, transmits the compressed log data to the server; when the network transmission rate is lower than the preset rate threshold, the transmission operation of the compressed log data is delayed.

[0102] In the above embodiments, log data and corresponding metadata data are obtained and compressed to generate compressed log data which is stored locally on the client device. When the network transmission rate is higher than a preset rate threshold, the compressed log data is transmitted to the server. When the network transmission rate is lower than the preset rate threshold, the transmission operation of the compressed log data is delayed.

[0103] In the above embodiments, the log data includes: network log, dot log, error log and execution log; the meta information data includes the application name, log type and log level corresponding to the log data.

[0104] The beneficial effects of the above technology are: by obtaining network logs, dot logs, error logs, and execution logs, the collection of log data on the client is realized, and the metadata data is obtained for compression processing to generate compressed log data. When the network transmission rate is higher than the preset rate threshold, the compressed log data is transmitted in batches at a regular interval to effectively reduce bandwidth and disk usage. When the network transmission rate is lower than the preset rate threshold, the transmission operation is delayed to avoid transmission interruption.

[0105] Embodiment 3:

[0106] The embodiment of the present invention provides a client observation method based on log data, comprising the steps of: analyzing log stream data based on preset analysis indicators to obtain log analysis data; and further comprising:

[0107] When effective analysis cannot be performed based on the log stream data, a detailed log acquisition request is sent to the client device through the server;

[0108] The client device obtains the request based on the detailed log, obtains the user's request operation based on the client device, and transmits the detailed log data to the server;

[0109] The server performs data format verification on the detailed log data, and after the verification passes, the distributed queue component converts it into detailed log stream data for indicator analysis to obtain log analysis data.

[0110] In the above embodiments, when effective analysis cannot be performed based on the preset analysis indicators according to the log stream data, a detailed log acquisition request is sent to the client device through the server; the client device obtains the detailed log acquisition request and obtains the user's request operation based on the client device, and transmits the detailed log data to the server; the server performs data format verification on the detailed log data, and after the verification is passed, the distributed queue component converts it into detailed log stream data for indicator analysis to obtain log analysis data.

[0111] In the above embodiments, the compressed log data transmitted by the client device is passive data collection. When effective analysis cannot be performed based on the log stream data, the server actively collects data from the client device.

[0112] The beneficial effect of the above technology is that when the log stream data cannot meet the log analysis needs, a detailed log acquisition request is initiated to the client device through the server, and the user's request operation based on the client device is obtained, and the detailed log data is transmitted to the distributed queue component through the server, thereby realizing the method of actively collecting detailed log data when the log stream data cannot meet the analysis needs, so as to ensure that the log analysis data is obtained based on the preset analysis indicators.

[0113] Embodiment 4:

[0114] The embodiment of the present invention provides a client observation method based on log data, comprising the steps of: a server decompresses compressed log data through a log service component, performs data format verification, and transmits the decompressed data to a distributed queue component after the verification passes; comprising:

[0115] The server decompresses the compressed log data transmitted by the client device through the log service component to obtain the decompressed data;

[0116] Perform format check on the decompressed data, and when the data format of the decompressed data is the preset data format, transmit the decompressed data to the distributed queue component; when the data format of the decompressed data is not the preset data format, refuse to receive the compressed log data, and return a prompt message to the client device.

[0117] In the above embodiments, the server decompresses the compressed log data transmitted by the client device through the log service component, obtains the decompressed data, and performs format verification on the decompressed data. When the data format of the decompressed data is a preset data format, the decompressed data is transmitted to the distributed queue component.

[0118] In the above embodiment, when the data format of the decompressed data is not the preset data format, the compressed log data is rejected and a prompt message is returned to the client device.

[0119] In the above embodiments, the preset data format may be implemented as a json format.

[0120] The beneficial effect of the above technology is that: through the log service component on the server side, the decompression processing of the compressed log data transmitted by the client device is realized, and the data format is verified. The verification is transmitted to the backward distributed queue component for storage, so that the received decompressed data is in a unified format, which facilitates the distributed queue component to process and store the decompressed data.

[0121] Embodiment 5:

[0122] The embodiment of the present invention provides a client observation method based on log data, comprising the steps of: a distributed queue component encapsulates and processes the decompressed data and converts it into log stream data; comprising:

[0123] Build a distributed queue component, obtain the configuration parameter set of the distributed queue component, perform throughput detection and operation resolution time test under the preset workload, and obtain the optimal configuration parameters of the distributed queue component for configuration;

[0124] The distributed queue component stores the decompressed data in a partition unit under a corresponding topic subcomponent in the distributed queue component according to the log topic of the decompressed data;

[0125] The partition unit obtains the application name, log name, log level and client IP information in the decompressed data and generates metadata; and constructs log stream data based on the metadata and the decompressed data and stores it in the partition unit.

[0126] In the above embodiments, the distributed queue component performs throughput detection and operation resolution time test under a preset workload according to a set of configuration parameters, and obtains optimal configuration parameters for configuration.

[0127] In the above embodiments, the distributed queue component stores the decompressed data in the partition unit under the corresponding topic subcomponent in the distributed queue component according to the log topic of the decompressed data. The partition unit obtains the metadata in the decompressed data, and constructs log stream data for storage based on the metadata and the decompressed data.

[0128] In the above embodiment, the metadata includes the application name, log name, log level and client IP information in the decompressed data.

[0129] In the above embodiments, the relationship between the topic subcomponent and the partition unit in the distributed queue component is a logical correspondence.

[0130] The beneficial effect of the above technology is that by testing the configuration parameter set of the distributed queue component based on throughput detection and running solution time under a preset workload, the optimal configuration parameters are obtained, and the distributed queue component is set according to the optimal configuration parameters, which effectively improves the working performance of the distributed queue component, and then the processing of decompressed data is realized through the distributed queue component, and the log stream data is obtained for encapsulation and storage.

[0131] Embodiment 6:

[0132] The embodiment of the present invention provides a client observation method based on log data, comprising the steps of: constructing a distributed queue component, obtaining a configuration parameter set of the distributed queue component, performing a throughput detection and an operation resolution time test under a preset workload, and obtaining the optimal configuration parameters of the distributed queue component for configuration; comprising:

[0133] Get the configuration parameter set of the distributed queue component;

[0134] Each configuration parameter in the configuration parameter set is divided into a number of non-overlapping intervals, and a cube parameter set is constructed according to the number of intervals of the configuration parameters;

[0135] Based on the cube parameter set, only one sampling point is randomly selected in the interval of each configuration parameter to construct the training configuration parameters;

[0136] Configure the parameters of the distributed queue component according to the training configuration parameters, calculate the throughput of the distributed queue component for the preset workload based on the training configuration parameters, repeat the above steps for a preset number of times, select the training configuration parameters with a throughput higher than the preset standard throughput, and construct a training configuration parameter set;

[0137] The attention mechanism layer has a training configuration parameter set P = [p1,…,p n ] to adjust the attention mechanism, input the generator built based on the feedforward neural network for processing, and output n sets of generator parameter sets Q = [q1,…,q n ];

[0138] Constructing a discriminator based on the first stochastic gradient function;

[0139]

[0140] in, is the gradient operator of the discriminator. There are n sets of training configuration parameters in the training configuration parameter set. i ) is the discriminator D based on the i-th group of training configuration parameters p in the training configuration parameter set P i Detection of G(q i ) is the generator G based on the training configuration parameter p i The resulting generator parameter q i ;

[0141] The discriminator performs parameter detection based on the training configuration parameters and the generator parameters. When the discriminator cannot effectively distinguish the training configuration parameters from the generator parameters, the generator parameters are obtained as network configuration parameters.

[0142] The generator is updated based on the second stochastic gradient function and the generator parameters are re-output;

[0143]

[0144] Repeat the above steps, and the discriminator detects several groups of generator parameters generated by the generator, obtains network configuration parameters, and constructs a network configuration parameter set;

[0145] Based on the network configuration parameter set and the training configuration parameter set, a component parameter optimization set is constructed;

[0146] Under the preset workload, the running solution time of the distributed queue component based on the component parameters in the component parameter optimization set is calculated, and the component parameters corresponding to the shortest running solution time are obtained as the optimal configuration parameters.

[0147] In the above embodiments, a configuration parameter set of a distributed queue component is obtained, each configuration parameter in the configuration parameter set is divided into a number of non-overlapping intervals, and a cube parameter set is constructed according to the several intervals of the configuration parameters; based on the cube parameter set, only one sampling point is randomly selected in the interval of each configuration parameter to construct a training configuration parameter; the parameters of the distributed queue component are configured according to the training configuration parameters, and the throughput of the distributed queue component for a preset workload based on the training configuration parameters is calculated; after repeating the above steps for a preset number of times, the training configuration parameters with a throughput higher than the preset standard throughput are selected to construct a training configuration parameter set.

[0148] In the above embodiments, the attention mechanism layer performs an attention mechanism adjustment on the training configuration parameters of the training configuration parameter set, inputs a generator constructed based on a feedforward neural network for processing, outputs a generator parameter set, and detects the training configuration parameters and the generator parameters through a discriminator. When the discriminator cannot effectively distinguish between the training configuration parameters and the generator parameters, the generator parameters are obtained as network configuration parameters.

[0149] In the above embodiments, a component parameter optimization set is constructed based on a network configuration parameter set and a training configuration parameter set. Under a preset workload, the running solution time of the distributed queue component based on the component parameters in the component parameter optimization set is calculated, and the component parameters corresponding to the shortest running solution time are obtained as the optimal configuration parameters.

[0150] In the above embodiments, according to the configuration parameter set of the distributed queue component, the Latin hypercube sampling method is used to obtain the training configuration parameters, thereby avoiding repeated extraction of the same training configuration parameters. When the throughput of the distributed queue component for the preset workload based on the training configuration parameters is higher than the preset standard throughput, the training configuration parameter set is obtained.

[0151] In the above embodiments, based on the network model constructed by the generator-discriminator, the generator captures hidden structures and hidden associations from the training configuration parameter set to generate a better configuration combination, and obtains the network configuration parameters after the discriminator can effectively distinguish them.

[0152] In the above embodiments, the generator updates parameters according to the second stochastic gradient function, and the discriminator updates parameters according to the first stochastic gradient function.

[0153] The beneficial effects of the above technology are: by non-repetitive sampling of the configuration parameter set of the distributed queue component, the training configuration parameters are obtained, and the throughput detection is performed based on the preset workload to obtain the training configuration parameter set, and the network model constructed by the generator-discriminator realizes the acquisition of network configuration parameters, and the component parameter optimization set constructed according to the training configuration parameters and the network configuration parameters is tested based on the distributed queue component to obtain the component parameters corresponding to the shortest running solution time as the optimal configuration parameters. The distributed queue component is configured according to the optimal configuration parameters, thereby effectively improving the storage performance of the distributed queue component.

[0154] Embodiment 7:

[0155] The embodiment of the present invention provides a client observation method based on log data, comprising the steps of: establishing an index directory based on log stream data stored in a distributed queue component; comprising:

[0156] The distributed queue component is built based on the service management node and counts the task load of the service management node;

[0157] The partition unit processes the decompressed data to obtain the log stream data, performs a backup operation to obtain the log stream backup data, and stores the log stream data and the log stream backup data in the partition unit;

[0158] According to the task load of the service management node in the distributed queue component, the partition units are intelligently allocated based on the service management node;

[0159] The partition unit generates a log summary based on the log stream data, obtains the node address of the server management node where the partition unit is located, and constructs log index data;

[0160] The index directory is constructed by obtaining the log index data of all partition units in the distributed queue component.

[0161] In the above embodiments, according to the task load of the service management node in the distributed queue component, the partition unit is intelligently allocated to the service management node with a smaller task load for storage.

[0162] In the above embodiment, the partition unit processes the decompressed data to obtain the log stream data, performs a backup operation to obtain the log stream backup data, and stores the log stream data and the log stream backup data.

[0163] In the above embodiment, the partition unit generates a log summary according to the log stream data, obtains the node address of the server management node where the partition unit is located, and constructs the log index data; and constructs the index directory by obtaining the log index data of all partition units in the distributed queue component.

[0164] In the above embodiments, the service management node and the partition unit are in a physical storage relationship.

[0165] The beneficial effects of the above technology are: by analyzing the log summary of the log stream data in the partition unit and the node address of the server management node where the partition unit is located, the log index data is obtained, and then an index directory is constructed, which is convenient for background staff to query the corresponding log stream data according to the index directory, and the storage of the partition unit is intelligently allocated according to the task load of the service management node in the distributed queue component, thereby improving the working storage performance of the distributed queue component.

[0166] Embodiment 8:

[0167] The embodiment of the present invention provides a client observation method based on log data, comprising the steps of: analyzing log stream data based on preset analysis indicators to obtain log analysis data; comprising:

[0168] Obtain log stream data in the distributed queue component according to the index directory;

[0169] Perform anomaly detection on log stream data. When no anomaly is detected in the log stream data, use the log stream data for indicator analysis.

[0170] Based on the preset analysis indicators, a data analysis template is constructed, and the log stream data is imported into the data analysis template for calculation to obtain the log analysis data; when the log stream data changes, the log analysis data is updated in real time.

[0171] In the above embodiments, log stream data is obtained in a distributed queue component according to an index directory, and anomaly detection is performed on the log stream data. When it is detected that there is no anomaly in the log stream data, a data analysis template is constructed based on preset analysis indicators, and the log stream data is imported into the data analysis template for calculation to obtain log analysis data.

[0172] In the above embodiments, the log analysis data is dynamically updated in real time according to the changes in the log stream data in the distributed queue component.

[0173] In the above embodiments, the staff sets the preset analysis indicators according to the requirements and verifies the generated data analysis template.

[0174] In the above embodiments, the staff can write a log analysis program as needed to connect to the log stream data to obtain log analysis data.

[0175] The beneficial effects of the above technology are: staff can realize intelligent analysis of log stream data by setting preset analysis indicators and building data analysis templates, obtain log analysis data to meet the log monitoring and log reporting needs of various application scenarios, and dynamically update log analysis data, thereby improving the real-time and accuracy of log analysis data.

[0176] Embodiment 9:

[0177] The embodiment of the present invention provides a client observation method based on log data, comprising the steps of: performing anomaly detection on log stream data, and when detecting that the log stream data has no anomaly, using the log stream data for indicator analysis; comprising:

[0178] Obtain log stream data for vectorization processing to generate log vector data;

[0179] According to the time vector elements in the log vector data, a time relationship vector network is constructed;

[0180] Analyze the influence of adjacent log vector data on log vector data in the temporal relationship vector network, and perform attention optimization to obtain preprocessed log vectors;

[0181]

[0182] Among them, T′ j For the j-th log vector data T j The preprocessed log vector obtained by processing is assumed to be the log vector data T j Neighborhood T(T j -δ,T j +δ) has N adjacent log vector data, σ is the first activation function, ∏() is the vector concatenation operator, θ(T i ,T j ,t′ i,j ) is the log vector data T j With adjacent log vector data T i The connection weight, θ(T i ,T j ,t′ i,j ) i is the log vector data T j With all adjacent log vector data T i The connection weight iteration, t′i,j is the log vector data T j With adjacent log vector data T i The normalized log time difference of is the linear transformation matrix;

[0183] θ(T i ,T j ,t′ i,j )=tanh[(T i F T ) T T j +F t t′ i,j ]

[0184] Among them, tanh is the second activation function, F T is the vector transformation matrix, F t is the time transformation matrix;

[0185] t i,j =t i -t j

[0186]

[0187] Among them, t i is the adjacent log vector data T i The time vector element, t j is the log vector data T j The time vector element, t i,j is the adjacent log vector data T i With log vector data T j The log time difference, t max is the maximum log time difference in the temporal relationship vector network, t min is the minimum log time difference in the time relationship vector network;

[0188] The preprocessed log vector is input into the anomaly detection model for anomaly detection to obtain the anomaly detection results of the log stream data.

[0189] In the above embodiments, the log stream data is obtained for vectorization processing to generate log vector data, and a time relationship vector network is constructed based on the time vector elements in the log vector data; the influence of adjacent log vector data on the log vector data in the time relationship vector network is analyzed, and attention optimization is performed to obtain a preprocessed log vector, and the preprocessed log vector is input into an anomaly detection model for anomaly detection to obtain anomaly detection results of the log stream data.

[0190] In the above embodiments, the preprocessed log vector is obtained by analyzing the influence of the connection weight between the log vector data and the adjacent log vector data on the log vector data; the connection weight between the log vector data and the adjacent log vector data is calculated by analyzing the semantic similarity and time difference between the log vector data and the adjacent log vector data.

[0191] In the above embodiments, the time vector elements of the log vector data and the adjacent log vector data may occur within a very short time interval, resulting in a log time difference approaching 0. The log time difference is converted to a fixed interval through normalization processing to obtain the normalized log time difference value.

[0192] In the above embodiments, the semantic information of log stream data is extracted through log vector data, and based on the influence of adjacent log vector data on the log vector data in the time relationship vector network, the time structure relationship between the log stream data is comprehensively considered to improve the accuracy of anomaly detection model detection.

[0193] The beneficial effects of the above technology are: by vectorizing the log stream data, the semantic information of the log stream data can be extracted, and by constructing a time relationship vector network, the influence of adjacent log vector data on the log vector data in the network can be analyzed, and the preprocessed log vector can be obtained for the anomaly detection model, thereby improving the accuracy of the anomaly detection model in detecting anomalies in the log stream data.

[0194] Embodiment 10:

[0195] The embodiment of the present invention provides a client observation system based on log data, referring to Figure 2 ,include:

[0196] The data collection module is used to collect the log data on the client device, compress it, and transmit the compressed log data to the server;

[0197] The data verification module is used by the server to decompress the compressed log data through the log service component and perform data format verification. After the verification passes, the decompressed data is transmitted to the distributed queue component;

[0198] The data encapsulation module is used to encapsulate the decompressed data through the distributed queue component and convert it into log stream data;

[0199] The data indexing module is used to create an index directory based on the log stream data stored in the distributed queue component;

[0200] The data analysis module is used to analyze the log stream data based on preset analysis indicators and obtain log analysis data.

[0201] In the above embodiments, the data acquisition module collects log data on the client device and compresses it to obtain compressed log data; the data verification module decompresses the compressed log data through the log service component in the server and performs data format verification. After the verification, the decompressed data is transmitted to the distributed queue component. The data encapsulation module encapsulates the decompressed data through the distributed queue component and converts it into log stream data; the data indexing module establishes an index directory based on the log stream data stored in the distributed queue component, and the data analysis module analyzes the log stream data based on preset analysis indicators to obtain log analysis data.

[0202] In the above embodiments, the client observation system based on log data can achieve good scalability and customization capabilities, and can meet various monitoring requirements on the terminal.

[0203] In the above embodiments, the client observation system based on log data can proactively solve the problems observed by the client, obtain the operation status and problems of the client, and thereby perform problem repair and optimization.

[0204] The beneficial effects of the above technology are: the client device collects log data on the other end, transmits the compressed log data to the server, the server decompresses it, and after checking the data format, transmits it to the distributed queue component to obtain log stream data, thereby realizing the collection and storage of log data for the client device; an index directory is established based on the log stream data, which is convenient for the staff and the query and acquisition of log stream data during data analysis, and by establishing preset analysis indicators, the system realizes intelligent analysis of log stream data and obtains log analysis data; compared with traditional technology, the server in the above system verifies the data format of the decompressed data and then transmits it to the distributed queue component to obtain log stream data for indicator analysis, which solves the problem that different types of log data cannot be effectively identified in traditional technology, and by establishing preset analysis indicators, the system realizes intelligent analysis of log stream data, and the staff can set preset analysis indicators according to business needs and access log stream data to support more detailed client indicators and business indicator scenarios, thereby meeting the customized observation needs of actual scenarios.

[0205] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A client observation method based on log data, characterized in that: include: Collect log data on the client side through the client device, compress it, and transmit the compressed log data to the server side; The server decompresses the compressed log data through the log service component and performs data format verification. After the verification passes, the decompressed data is transmitted to the distributed queue component; The distributed queue component encapsulates the decompressed data and converts it into log stream data; Establishing an index directory based on the log stream data stored in the distributed queue component; Based on preset analysis indicators, the log stream data is analyzed to obtain log analysis data.

2. The client observation method based on log data according to claim 1, characterized in that: The step of collecting log data on the client side through the client device, compressing the log data, and transmitting the compressed log data to the server side includes: Obtaining a network log by recording network request information of the client device; Obtaining a log by recording operation information of the user based on the client device; Obtaining an error log by recording operation error information of the client device; Obtaining an execution log by recording event operation information and event troubleshooting information of the client device; The log data includes: network log, dot log, error log and execution log; Acquire corresponding meta information data according to the log data, compress the log data and the meta information data, generate compressed log data and save it locally on the client device; The meta information data includes the application name, log type and log level corresponding to the log data. The client device detects the current network transmission rate, and when the network transmission rate is higher than a preset rate threshold, transmits the compressed log data to the server; when the network transmission rate is lower than the preset rate threshold, delays the transmission operation of the compressed log data.

3. The client observation method based on log data according to claim 1, characterized in that: The step of: analyzing the log stream data based on preset analysis indicators to obtain log analysis data; also includes: When effective analysis cannot be performed based on the log stream data, sending a detailed log acquisition request to the client device through the server; The client device obtains the detailed log data to the server according to the detailed log acquisition request and the user's request operation based on the client device; The server performs data format verification on the detailed log data, and after the verification is passed, the distributed queue component converts the detailed log data into detailed log stream data for indicator analysis to obtain log analysis data.

4. The client observation method based on log data according to claim 1, characterized in that: The step: the server decompresses the compressed log data through the log service component, performs data format verification, and transmits the decompressed data to the distributed queue component after the verification passes; including: The server performs decompression processing on the compressed log data transmitted by the client device through the log service component to obtain decompressed data; A format check is performed based on the decompressed data. When the data format of the decompressed data is a preset data format, the decompressed data is transmitted to the distributed queue component. When the data format of the decompressed data is not a preset data format, the compressed log data is refused to be received, and a prompt message is returned to the client device.

5. The client observation method based on log data according to claim 1, characterized in that: The step: the distributed queue component encapsulates the decompressed data and converts it into log stream data; including: Construct a distributed queue component, obtain a set of configuration parameters of the distributed queue component, perform throughput detection and operation resolution time test under a preset workload, and obtain the optimal configuration parameters of the distributed queue component for configuration; The distributed queue component stores the decompressed data in a partition unit under a corresponding topic subcomponent in the distributed queue component according to a log topic of the decompressed data; The partition unit obtains the application name, log name, log level and client IP information in the decompressed data, and generates metadata; and constructs log stream data according to the metadata and the decompressed data and stores it in the partition unit.

6. The client observation method based on log data according to claim 5, characterized in that: The steps of: constructing a distributed queue component, obtaining a set of configuration parameters of the distributed queue component, performing throughput detection and a running solution time test under a preset workload, and obtaining the optimal configuration parameters of the distributed queue component for configuration; include: Obtaining a set of configuration parameters of the distributed queue component; Dividing each configuration parameter in the configuration parameter set into a plurality of non-overlapping intervals, and constructing a cube parameter set according to the plurality of intervals of the configuration parameters; Based on the cube parameter set, only one sampling point is randomly selected in the interval of each configuration parameter to construct the training configuration parameter; The parameters of the distributed queue component are configured according to the training configuration parameters, and the throughput of the distributed queue component for a preset workload based on the training configuration parameters is calculated. After repeating the above steps for a preset number of times, the training configuration parameters with a throughput higher than a preset standard throughput are selected to construct a training configuration parameter set; The attention mechanism layer configures the training parameter set P = [p1, ..., p n ] to adjust the attention mechanism, input the generator built based on the feedforward neural network for processing, and output n sets of generator parameter sets Q = [q1,…,q n ]; Constructing a discriminator based on the first stochastic gradient function; in, is the gradient operator of the discriminator. There are n sets of training configuration parameters in the training configuration parameter set. i ) is the discriminator D based on the i-th group of training configuration parameters p in the training configuration parameter set P i Detection of G(q i ) is the generator G based on the training configuration parameter p i The resulting generator parameter q i ; The discriminator performs parameter detection according to the training configuration parameters and the generator parameters, and when the discriminator cannot effectively distinguish the training configuration parameters from the generator parameters, obtains the generator parameters as network configuration parameters; The generator is updated based on the second stochastic gradient function and the generator parameters are re-output; Repeat the above steps, and the discriminator detects several groups of generator parameters generated by the generator, obtains network configuration parameters, and constructs a network configuration parameter set; Constructing a component parameter optimization set based on the network configuration parameter set and the training configuration parameter set; Under a preset workload, the running solution time of the distributed queue component based on the component parameters in the component parameter optimization set is calculated, and the component parameters corresponding to the shortest running solution time are obtained as the optimal configuration parameters.

7. The client observation method based on log data according to claim 1, characterized in that: The step of establishing an index directory based on the log stream data stored in the distributed queue component includes: The distributed queue component is constructed based on the service management node, and counts the task load of the service management node; The partition unit processes the decompressed data to obtain log stream data, performs a backup operation to obtain log stream backup data, and stores the log stream data and the log stream backup data in the partition unit; According to the task load of the service management node in the distributed queue component, the partition unit is intelligently deployed based on the service management node; The partition unit generates a log summary according to the log stream data, obtains a node address of a server management node where the partition unit is located, and constructs log index data; An index directory is constructed by acquiring log index data of all partition units in the distributed queue component.

8. The client observation method based on log data according to claim 1, characterized in that: The step of analyzing the log stream data based on preset analysis indicators to obtain log analysis data includes: According to the index directory, obtaining log stream data in the distributed queue component; Performing anomaly detection on the log stream data, and when it is detected that the log stream data has no anomaly, using the log stream data for indicator analysis; Based on preset analysis indicators, a data analysis template is constructed, the log stream data is imported into the data analysis template for calculation, and log analysis data is obtained; when the log stream data changes, the log analysis data is updated in real time.

9. The client observation method based on log data according to claim 8, characterized in that: The step: performing anomaly detection on the log stream data, and when it is detected that the log stream data has no anomaly, using the log stream data for indicator analysis; include: Obtain log stream data for vectorization processing to generate log vector data; Constructing a time relationship vector network according to the time vector elements in the log vector data; Analyze the influence of adjacent log vector data on the log vector data in the temporal relationship vector network, and perform attention optimization to obtain a preprocessed log vector; Among them, T′ j For the j-th log vector data T j The preprocessed log vector obtained by processing is assumed to be the log vector data T j Neighborhood T(T j -δ,T j +δ) has N adjacent log vector data, σ is the first activation function, ∏() is the vector concatenation operator, θ(T i ,T j ,t′ i,j ) is the log vector data T j With adjacent log vector data T i The connection weight, θ(T i ,T j ,t′ i,j ) i is the log vector data T j With all adjacent log vector data T i The connection weight iteration, t′ i,j is the log vector data T j With adjacent log vector data T i The normalized log time difference of is the linear transformation matrix; θ(T i ,T j ,t′ i,j )=tanh[(T i F T ) T T j +F t t′ i,j ] Among them, tanh is the second activation function, F T is the vector transformation matrix, F t is the time transformation matrix; t i,j =t i -t j Among them, t i is the adjacent log vector data T i The time vector element, t j is the log vector data T j The time vector element, t i,j is the adjacent log vector data T i With log vector data T j The log time difference, t max is the maximum log time difference in the temporal relationship vector network, t min is the minimum log time difference in the time relationship vector network; The preprocessed log vector is input into the anomaly detection model to perform anomaly detection, and the anomaly detection result of the log stream data is obtained.

10. A client observation system based on log data, characterized in that: include: The data collection module is used to collect the log data on the client side through the client device, compress it, and transmit the compressed log data to the server side; A data verification module is used for the server to decompress the compressed log data through the log service component and perform data format verification, and transmit the decompressed data to the distributed queue component after the verification passes; A data encapsulation module, used to encapsulate the decompressed data through the distributed queue component and convert it into log stream data; A data indexing module, used to establish an index directory based on the log stream data stored in the distributed queue component; The data analysis module is used to analyze the log stream data based on preset analysis indicators to obtain log analysis data.

Images