Log data analysis method, device, storage medium and electronic device

Abstract

The embodiments of this specification disclose a log data analysis method, device, storage medium and electronic device. The log data analysis device includes at least one log analysis module, a log acquisition module and a report generation module. The log data corresponding to multiple distributed servers are acquired, the log data corresponding to each distributed server is analyzed and processed by at least one log analysis module, and an analysis report corresponding to each log data is generated according to the analysis result by the report generation module, so that the user can monitor the operating status of multiple distributed servers through the analysis report.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a log data analysis method, device, storage medium and electronic device. Background Art

[0002] Logs are data files used to record user operations, electronic equipment operating status and other parameters, which can be used by relevant personnel to understand the operation of electronic equipment and event processing. As the operation of electronic equipment continues, a large amount of log data is often generated. Especially nowadays when distributed server architecture is increasingly used, the source, volume and type of log data are becoming more complicated. How to achieve efficient and intelligent analysis of log data has always been the focus of computer technology. Summary of the invention

[0003] The embodiments of this specification provide a log data analysis method, device, storage medium and electronic device, which can realize the automation of log data analysis and improve the efficiency of log data analysis and collation. The technical solution is as follows:

[0004] In a first aspect, an embodiment of the present specification provides a log data analysis device, the device comprising:

[0005] At least one log analysis module, log acquisition module and report generation module;

[0006] The log acquisition module is used to acquire log data corresponding to multiple distributed servers;

[0007] The log analysis module is used to analyze the log data according to the analysis instruction to obtain the analysis result corresponding to the log data;

[0008] The report generating module is used to generate an analysis report corresponding to each log data according to at least one analysis result corresponding to the log data.

[0009] In a second aspect, an embodiment of the present specification provides a log data analysis method, the method comprising:

[0010] Get the log data corresponding to multiple distributed servers;

[0011] Analyze each of the log data according to at least one target analysis instruction to obtain at least one analysis result corresponding to each of the log data;

[0012] Generate an analysis report corresponding to the log data according to at least one analysis result corresponding to the log data.

[0013] In a third aspect, an embodiment of the present specification provides a computer storage medium, wherein the computer storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the above-mentioned method steps.

[0014] In a fourth aspect, an embodiment of the present specification provides a computer program product, wherein the computer program product stores a plurality of instructions, and the instructions begin with a processor loading and executing the above method steps.

[0015] In a fourth aspect, an embodiment of the present specification provides an electronic device, which may include: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the above-mentioned method steps.

[0016] The beneficial effects brought by the technical solutions provided by some embodiments of this specification include at least:

[0017] The present specification provides a log data analysis device, wherein a log acquisition module acquires log data corresponding to multiple distributed servers, at least one log analysis module analyzes and processes the log data corresponding to each distributed server, and a report generation module generates an analysis report corresponding to each log data. By providing at least one log analysis module for each distributed server, each log analysis module can provide log data analysis and processing capabilities to generate an analysis report for the log data of the distributed server, thereby avoiding the problem of difficulty in log data analysis caused by the complexity of the form and huge amount of log data due to the large number of server types and complex services in the distributed server architecture, and realizing intelligent analysis and efficient processing of log data in the distributed server architecture, so that users can conveniently and quickly monitor the operating status of multiple distributed servers through analysis reports. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a structural diagram of a log data analysis device provided in an embodiment of this specification;

[0020] Figure 2A-2B This is a schematic diagram of the architecture of a distributed server provided in an embodiment of this specification;

[0021] Figure 3 It is a structural diagram of a log data analysis device provided in an embodiment of this specification;

[0022] Figure 4 This is a schematic diagram of a scenario in which a user inputs a selection instruction provided in an embodiment of this specification;

[0023] Figure 5 It is a structural diagram of a log data analysis device provided in an embodiment of this specification;

[0024] Figure 6 It is a schematic diagram of the structure of a neural network provided in an embodiment of this specification;

[0025] Figure 7 It is a flow chart of a log data analysis method provided by an embodiment of this specification;

[0026] Figure 8 It is a structural schematic diagram of an electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0028] In the description of this specification, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In the description of this specification, it should be noted that, unless otherwise clearly specified and limited, "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. For those of ordinary skill in the art, the specific meanings of the above terms in this specification can be understood in specific circumstances. In addition, in the description of this specification, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are an "or" relationship.

[0029] The present specification is described in detail below with reference to specific embodiments.

[0030] See also Figure 1As shown, it is a structural schematic diagram of a log data analysis device provided in an embodiment of the present specification. The multiple modules included in the log data analysis device can be implemented as all or part of the device through software, hardware or a combination of both.

[0031] The log data analysis device 100 includes but is not limited to a mobile station (MS), a mobile terminal, a mobile phone, a handset, and portable equipment, etc. The log data analysis device 100 can communicate with one or more wide area network servers via a radio access network (RAN). For example, it can be a mobile phone (or a "cellular" phone), a computer with wireless communication capabilities, etc. It can also be a portable, pocket-sized, handheld, computer-built-in, or vehicle-mounted mobile device or equipment. In another embodiment, the log data analysis device 100 can also be connected to other electronic devices for communication via wired means.

[0032] In this embodiment, the log data analysis device 100 may also be equipped with a display device and a camera. The display device may be any device capable of realizing a display function, and the camera is used to collect video data; for example, the display device may be a cathode ray tube display (CR), a light-emitting diode display (LED), an electronic ink screen, a liquid crystal display (LCD), a plasma display panel (PDP), etc. The user may use the display device on the log data analysis device 100 to view displayed text, pictures, videos and other information, and send instructions to the log data analysis device 100 through an input device so that the log data analysis device 100 executes the action corresponding to the instruction.

[0033] See also Figure 1 The log data analysis device 100 includes: at least one log analysis module, a log acquisition module 101 and a report generation module 103, and the at least one log analysis module includes a log analysis module 1021, a log analysis module 1022, a log analysis module 1023, and a log analysis module 1024. It can be understood that there is no limit to the number of log analysis modules. Figure 1 Just an example.

[0034] The log acquisition module 101 is used to acquire log data corresponding to multiple distributed servers. A distributed server refers to a server system formed by multiple dispersed servers connected via an interconnected network. The processing and control functions of the distributed server are distributed on each server.

[0035] In one embodiment, the log data analysis device 100 further includes: a network interface. The log acquisition module 101 is specifically used to acquire log data of multiple distributed servers through the network interface. Log data can be understood as text data generated by the print output code embedded in the program by the program developer to assist in debugging, which is used to record variable information, program execution status, etc. when the program is running. Users can monitor the process status, service status, etc. of the server by viewing the log data.

[0036] Specifically, if Figure 2A As shown, it is a schematic diagram of the architecture of a distributed server provided in an embodiment of this specification, and the network interface of the log data analysis device 100 is connected to multiple distributed servers, and the multiple distributed servers include servers 301 to 309. It can be understood that the distribution and number of servers in the embodiments of this specification are only examples, and this embodiment does not impose any limitation on this.

[0037] In this embodiment, the log acquisition module 101 is specifically used to directly obtain the log data of multiple distributed servers from the multiple distributed servers through the network interface. For example, the log acquisition module 101 obtains the address corresponding to each distributed server, and obtains the log data of the distributed servers corresponding to each address through the network interface. In one embodiment, the log acquisition module 101 obtains the addresses of other distributed servers through the distributed servers with which the connection has been established, or obtains the addresses of the distributed servers through the instructions input by the user. The present application also includes other methods for obtaining the addresses of distributed servers. According to the preset frequency or receiving the log acquisition instruction input by the user, the log acquisition module 101 sends the log data acquisition instruction to the distributed servers corresponding to each address through the network interface, so that the distributed servers send log data to the log data analysis device. For another example, the log acquisition module 101 receives the log data sent by each distributed server through the network interface. Specifically, each distributed server reports log data to the log data analysis device based on the frequency corresponding to each distributed server. Among them, the log acquisition module 101 establishes connections with distributed servers 301 to 309 respectively through a local area network (LAN) and / or a wide area network (WAN). Protocols suitable for local area network communication include transmission control protocol (TCP), address resolution protocol (ARP), etc., and protocols suitable for wide area network communication include message queue telemetry transport protocol (MQTT), etc. A connection is established with each distributed server through a network interface corresponding to the network protocol, thereby obtaining log data corresponding to each distributed server.

[0038] In another embodiment, Figure 2B 1 is a schematic diagram of the architecture of a distributed server provided in an embodiment of this specification. The network interface of the log data analysis device 100 is connected to a switch, and the switch is connected to multiple distributed servers respectively. Figure 2A , which will not be described here. A switch can provide an exclusive electrical signal path for any two electronic devices connected to the switch, such as an Ethernet switch. In this embodiment, the log acquisition module 101 is specifically used to obtain log data of multiple distributed servers from the switch through a network interface. For example, the switch is a router, and the log data analysis device 100 establishes a local area network connection with the distributed server through the router, thereby obtaining log data corresponding to the distributed server.

[0039] Log analysis modules 1021 to 1024 are used to analyze log data according to the analysis instructions corresponding to each log analysis module to obtain analysis results corresponding to the log data. At least one analysis instruction includes at least the following instructions: data cleaning, data conversion, data integration, data source analysis, fault analysis, and operation status analysis. For example, data cleaning is to discover and correct identifiable errors in log data, including checking data consistency, processing invalid values ​​and missing values, data conversion is to convert the format statement of log data into other specified formats, and data source analysis is to determine the distributed server corresponding to the log data according to the identifier corresponding to the log data.

[0040] In one embodiment, the analysis instructions corresponding to each log analysis module are different from each other, that is, the processing methods of the log analysis module for each log data are different, and the analysis results obtained are also different. For example, the log analysis module 1021 performs data cleaning on the log data, the log analysis module 1022 performs fault analysis on the log data, the log analysis module 1023 performs data source analysis on the log data, and the log analysis module 1024 performs data cleaning and operation status analysis on the log data.

[0041] The report generation module 103 is used to generate an analysis report corresponding to each log data according to at least one analysis result corresponding to the log data. Figure 1 As shown, the log acquisition module 101 acquires the log data corresponding to each distributed server, and further at least one log analysis module analyzes each log data. For example, the log analysis module 1021 obtains a first analysis result after analyzing the log data, the log analysis module 1022 obtains a second analysis result after analyzing the log data, the log analysis module 1023 obtains a third analysis result after analyzing the log data, and the log analysis module 1024 obtains a fourth analysis result after analyzing the log data. The report generation module 103 generates an analysis report corresponding to the log data according to the first analysis result, the second analysis result, the third analysis result and the fourth analysis result.

[0042] In one embodiment, when the report generation module 103 detects that there is fault information in the analysis result, it outputs fault prompt information. For example, the fault information is highlighted in the analysis report, and an alarm is sounded through the speaker unit of the log data analysis device. By prompting the fault information, the user can detect whether the distributed server has an operating fault and make timely adjustments.

[0043] The present specification provides a log data analysis device, wherein a log acquisition module acquires log data corresponding to multiple distributed servers, at least one log analysis module analyzes and processes the log data corresponding to each distributed server, and a report generation module generates an analysis report corresponding to each log data. By providing at least one log analysis module for each distributed server, each log analysis module can provide log data analysis and processing capabilities to generate an analysis report for the log data of the distributed server, thereby avoiding the problem of difficulty in log data analysis caused by the complexity of the form and huge amount of log data due to the large number of server types and complex services in the distributed server architecture, and realizing intelligent analysis and efficient processing of log data in the distributed server architecture, so that users can conveniently and quickly monitor the operating status of multiple distributed servers through analysis reports.

[0044] See also Figure 3 As shown, it is a structural schematic diagram of another log data analysis device provided in an embodiment of this specification. The multiple modules included in the log data analysis device can be implemented as all or part of the device through software, hardware or a combination of both.

[0045] The log data analysis device 200 includes: at least one log analysis module, a log acquisition module 101, a report generation module 103, and a target selection module 201. The at least one log analysis module includes a log analysis module 1021, a log analysis module 1022, a log analysis module 1023, and a log analysis module 1024. It can be understood that there is no limit to the number of log analysis modules, and FIG. 2 is only an example.

[0046] The working principles of the log analysis module, the log acquisition module 101 and the report generation module 103 are shown in the above Figure 1 , I will not go into details here.

[0047] The target selection module is used to receive a user's selection instruction for a target log analysis module corresponding to the target log data, so that the target log analysis module analyzes the target log data according to the selection instruction; wherein at least one log analysis module includes a target log analysis module, and multiple log data include target log data. For example, for target log data provided by a distributed server, the user specifies the target log analysis modules as log analysis module 1021 and log analysis module 1022 for the target log data, so as to generate analysis results of log analysis module 1021 and log analysis module 1022 corresponding to the log data.

[0048] like Figure 4As shown, it is a schematic diagram of a scenario in which a user inputs a selection instruction provided by an embodiment of this specification. The selection instruction can be a selection instruction of multiple controls displayed by the user through the display interface of the log data analysis device 200, for example, selecting the log data corresponding to the target distributed server under the "server" control, selecting the target log analysis module under the "log analysis module" control, or inputting instructions in a blank input box, for example, entering the address corresponding to the target distributed server under the "server" control, and entering the number of the target log analysis module under the "log analysis module" control. It can be understood that the selection instruction also includes any other form, and this specification does not impose any limitation on this.

[0049] In another embodiment, the log analysis module is specifically used to determine whether the log data needs to be analyzed according to the identification of the log data. If yes, the log data is analyzed according to the analysis instruction to obtain the analysis result corresponding to the log data. If no, the log data is not processed. For example, when the distributed server sends log data to the log data analysis device 200, it adds an identification to the log data, and the identification is used to specify the log analysis module 1021 and the log analysis module 1022 to process the log data, so that when the log acquisition module 101 acquires the log data and sends the log data to the log analysis module 1021, the log analysis module 1022, the log analysis module 1023 and the log analysis module 1024, each log analysis module determines whether the log data needs to be analyzed according to the identification of the log data, wherein the log analysis module 1021 and the log analysis module 1022 analyze the log data and generate the analysis result, and the log analysis module 1023 and the log analysis module 1024 do not process the log data. The identifier can be a number corresponding to the log analysis module, or a code associated with the target log analysis module that needs to analyze the log data, or any other possible identifier. For example, when log analysis module 1022 is designated as the target log analysis module, the identifier "Log analysis-1022" is added to a preset position in the log data. This embodiment sends log data to all log analysis modules so that the log analysis modules determine whether the log data needs to be analyzed based on the identifier of the log data. Each log analysis module stores the log data. When the non-target log analysis module corresponding to the log data receives a selection instruction for the log data, it can quickly respond to the selection instruction, analyze the log data and generate analysis results for users to view, thereby improving the flexibility and response speed of log data analysis.

[0050] In another embodiment, the log acquisition module 101 is used to acquire the target log analysis module corresponding to the log data according to the identifier of the log data, and further send the log data to the target log analysis module. For example, when the distributed server sends log data to the log data analysis device 200, it adds an identifier to the log data, and the identifier is used to specify the log analysis module 1021 and the log analysis module 1022 to process the log data. After the log acquisition module 101 acquires the log data and the identifier corresponding to the log data, the log data is sent to the log analysis module 1021 and the log analysis module 1022, so that the log analysis module 1021 and the log analysis module 1022 analyze the log data and generate analysis results. In this embodiment, the log acquisition module determines the target log analysis module corresponding to the log data, and only sends the log data to the target log analysis module corresponding to the log data, thereby reducing the waste of storage resources of the log data analysis device.

[0051] The present specification provides a log data analysis device, wherein a log acquisition module acquires log data corresponding to multiple distributed servers, and by selecting instructions or identifiers carried by log data, target log data is only analyzed and processed by a target log analysis module, so as to obtain analysis results and even analysis reports specified by the target log data. By providing at least one log analysis module for each distributed server, each log analysis module can provide log data analysis and processing capabilities to generate analysis reports for the log data of the distributed server, thereby avoiding the problem of log data analysis difficulties caused by the complexity of the form and huge amount of log data due to the large variety of servers and complex services in the distributed server architecture, and realizing intelligent analysis and efficient processing of log data in the distributed server architecture, so that users can conveniently and quickly monitor the operating status of multiple distributed servers through analysis reports, and obtain analysis reports corresponding to the target data analysis modules specified by log data, so as to meet users' different analysis needs for multiple log data and cope with complex working environments.

[0052] See also Figure 5 As shown, it is a structural schematic diagram of another log data analysis device provided in an embodiment of this specification. The multiple modules included in the log data analysis device can be implemented as all or part of the device through software, hardware or a combination of both.

[0053] The log data analysis device 300 includes: at least one log analysis module, a log acquisition module 101 and a report generation module 103, and an associated storage module 301. The at least one log analysis module includes a log analysis module 1021, a log analysis module 1022, a log analysis module 1023, and a log analysis module 1024. It is understandable that there is no limit to the number of log analysis modules. Figure 5 Just an example.

[0054] The working principles of the log analysis module, the log acquisition module 101 and the report generation module 103 are shown in the above Figure 1 , I will not go into details here.

[0055] The associated storage module 301 is used to associate and store the log data and the analysis report corresponding to the log data. For example, by adding an identifier with an associated relationship to the log data and the report corresponding to the log data, or setting a file name with an associated relationship for the log data and the report corresponding to the log data, the log data and the analysis report corresponding to the log data are associated and stored in the storage unit of the log data analysis device 300. In this embodiment, the log data and the analysis report corresponding to the log data are associated and stored so that the user can view the analysis report and call back the log data at any time.

[0056] In one embodiment, the log analysis module is specifically used to perform transfer learning on the historical log data and the analysis reports corresponding to the historical log data stored in the associated storage module 301, and to update the analysis instructions corresponding to the log analysis module based on preset conditions, so that the log analysis module analyzes the log data according to the updated analysis instructions.

[0057] Among them, the preset conditions include at least one or more of the following: receiving an update instruction from the user, the historical log data stored in the log data analysis device exceeds a preset threshold, and a preset update frequency. For example, receiving an update instruction from the user is receiving an update instruction for a designated target log analysis module input by the user through an input device. It is detected that the amount of historical log data stored in the log data analysis device exceeds a first preset threshold, and all or part of the log analysis modules are updated. After a period of time after the update is completed, it is detected that the amount of historical log data stored in the log data analysis device exceeds a second preset threshold, and all or part of the log analysis modules are updated. The preset update frequency can be any one of a time period or a number of log data reported by a distributed server as a period, and can be set as needed by relevant technical personnel.

[0058] The log data analysis module uses a neural network architecture to transfer historical log data and the analysis reports corresponding to the historical log data, thereby updating the analysis instructions corresponding to the log analysis module. Figure 6 As shown, it is a structural schematic diagram of a neural network provided in an embodiment of the present application, which adopts a hidden Markov model based on a deep neural network, namely, a DNN-HMM model, and introduces an error back propagation algorithm to optimize the existing neural network model to improve the recognition accuracy of the neural network model.

[0059] A deep neural network consists of an input layer, a hidden layer, and an output layer. Figure 6 As shown, the input layer is used to calculate the output value of the hidden layer unit input to the bottom layer according to the analysis report corresponding to the log data input to the deep neural network. The input layer usually includes multiple input units, and the input unit is used to calculate the output value of the hidden layer unit input to the bottom layer according to the input analysis report. After the analysis report is input to the input unit, the input unit calculates the output value output to the bottom layer hidden layer according to its own weighted value using the analysis report input to the input unit.

[0060] There are usually multiple hidden layers, each of which includes multiple hidden layer units. The hidden layer units receive input values ​​from the hidden layer units in the next hidden layer. The input values ​​from the hidden layer units in the next hidden layer are weighted and summed according to the weighted values ​​of this layer, and the result of the weighted sum is used as the output value of the hidden layer units output to the previous hidden layer.

[0061] The output layer includes multiple output units, which receive input values ​​from hidden layer units in the top hidden layer, perform weighted summation on the input values ​​from the hidden layer units in the top hidden layer according to the weighted values ​​of this layer, calculate the actual output value according to the result of the weighted summation, and adjust the connection weights and thresholds of each layer along the output path based on the error between the expected output value and the actual output value.

[0062] Specifically, in this embodiment, a DNN-HMM model that introduces an error back propagation algorithm is used to create an initial model. After extracting the log data and the analysis report corresponding to the log data, the analysis report is input into the neural network model. The training process of the neural network model usually consists of two parts: forward propagation and back propagation. In the forward propagation process, the analysis report is transmitted from the input layer of the neural network model to the output layer after the transfer function (also called activation function, conversion function) of the hidden layer neurons (also called nodes) is calculated, wherein each layer of neuron state affects the next layer of neuron state, and the actual output value-analysis report is calculated in the output layer, and the expected error between the actual output value and the expected output value is calculated. The parameters of the neural network model are adjusted based on the expected error, and the parameters include the weight value and threshold of each layer. After the training is completed, a neural network including a new analysis instruction is generated, and the transfer learning of the log data and the analysis report corresponding to the log data in the storage unit is completed by the neural network.

[0063] The present specification provides a log data analysis device, wherein a log acquisition module acquires log data corresponding to multiple distributed servers, at least one log analysis module analyzes and processes the log data corresponding to each distributed server, and a report generation module generates an analysis report corresponding to each log data, by configuring at least one analysis instruction for each distributed server, each analysis instruction can provide analysis and processing capabilities for log data, so as to generate an analysis report for the log data of the distributed server, thereby avoiding the problem of difficulty in log data analysis caused by the complexity of the form and huge amount of log data due to the large variety of servers and complex services in the distributed server architecture, and realizing intelligent analysis and efficient processing of log data in the distributed server architecture, so that users can conveniently and quickly monitor the operating status of multiple distributed servers through analysis reports, and update the analysis instructions corresponding to the log analysis module, so that the log analysis module analyzes and processes the log data according to the updated analysis instructions, and continuously update and improve the log analysis module, thereby improving the analysis capability of the log data analysis device for log data.

[0064] It should be noted that the multiple functional modules included in the log data analysis device provided in the above embodiment are only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0065] In one embodiment, Figure 7 The figure is a flow chart of a log data analysis method provided in this specification, which can be implemented by a computer program and can be run on the above Figures 1 to 6 The computer program can be integrated into an application or run as an independent tool application.

[0066] Specifically, the log data analysis method includes:

[0067] S101. Obtain log data corresponding to multiple distributed servers.

[0068] Specifically, the log data of multiple distributed servers are directly obtained from the multiple distributed servers through the network interface. For example, the address corresponding to each distributed server is obtained, and the log data of the distributed servers corresponding to each address is obtained through the network interface. For another example, the log data sent by each distributed server is received through the network interface.

[0069] S102: Analyze each log data according to at least one target analysis instruction to obtain at least one analysis result corresponding to each log data.

[0070] Obtain the selection instruction corresponding to each log data, and determine at least one target analysis instruction corresponding to the log data based on the selection instruction corresponding to each log data. For example, obtain the selection instruction input by the user for each log data; or, according to the identifier of each log data, obtain the selection instruction corresponding to each log data. See above Figure 3-Figure 4 As shown, no further description is given here.

[0071] At least one target analysis instruction includes at least the following instructions: data cleaning, data conversion, data integration, data source analysis, fault analysis, and operation status analysis. For example, data cleaning is to discover and correct identifiable errors in log data, including checking data consistency, processing invalid values ​​and missing values, data conversion is to convert the format statement of log data into other specified formats, and data source analysis is to determine the distributed server corresponding to the log data according to the identifier corresponding to the log data.

[0072] This embodiment selects instructions or identifiers to analyze and process the target log data only based on the target analysis instructions, and obtains the analysis results and analysis reports specified by the target log data to meet the user's different analysis requirements for multiple log data and cope with complex working environments.

[0073] S103: Generate an analysis report corresponding to the log data according to at least one analysis result corresponding to the log data.

[0074] The analysis report is used to characterize the operating status of the distributed server corresponding to the log data. For example, the first analysis result is obtained after analyzing the log data through the first target analysis instruction, and the second analysis result is obtained after analyzing the log data through the second target analysis instruction, and the analysis report corresponding to the log data is generated according to the first analysis result and the second analysis result.

[0075] After generating an analysis report corresponding to the log data according to at least one analysis result corresponding to the log data, the method further includes: storing the log data and the analysis report corresponding to the log data in association with each other. For example, by adding an identifier with an association relationship to the log data and the report corresponding to the log data, or setting a file name with an association relationship to the log data and the report corresponding to the log data, the log data and the analysis report corresponding to the log data are stored in association with each other in a storage unit of the log data analysis device.

[0076] In one embodiment, transfer learning is performed on the associated stored historical log data and the analysis report corresponding to the historical log data, and the target analysis instruction is updated based on the preset conditions, so that the log data is analyzed according to the updated target analysis instruction. The preset conditions include at least one or more of the following: receiving an update instruction from the user, the historical log data stored in the log data analysis device exceeds a preset threshold, and a preset update frequency. This embodiment can update the analysis instruction corresponding to the log analysis module so that the log analysis module analyzes and processes the log data according to the updated analysis instruction, and continuously updates and improves the log analysis module to improve the log data analysis device's ability to analyze the log data.

[0077] In addition, the log data analysis method and the log data analysis device embodiment provided in the above embodiment belong to the same concept, and the implementation process thereof is detailed in the method embodiment, which will not be repeated here.

[0078] The present specification provides a log data analysis method, which obtains log data corresponding to multiple distributed servers, analyzes and processes the log data corresponding to each distributed server according to at least one target analysis instruction, and generates an analysis report corresponding to each log data. By providing at least one log analysis module for each distributed server, each log analysis module can provide log data analysis and processing capabilities to generate an analysis report for the log data of the distributed server, thereby avoiding the problem of difficulty in log data analysis caused by the complexity of the form and huge amount of log data due to the large number of server types and complex services in the distributed server architecture, and realizing intelligent analysis and efficient processing of log data in the distributed server architecture, so that users can conveniently and quickly monitor the operating status of multiple distributed servers through analysis reports.

[0079] The serial numbers of the embodiments of this specification are for description only and do not represent the advantages or disadvantages of the embodiments.

[0080] The present specification also provides a computer storage medium, which can store multiple instructions, and the instructions are suitable for being loaded and executed by a processor as described above. Figure 7 The log data analysis method of the embodiment shown in the figure can be found in the specific execution process. Figure 7 The specific description of the illustrated embodiment will not be repeated here.

[0081] The present specification also provides a computer program product, which stores at least one instruction, and the at least one instruction is loaded and executed by the processor as described above. Figure 7 The log data analysis method of the embodiment shown in the figure can be found in the specific execution process. Figure 7 The specific description of the illustrated embodiment will not be repeated here.

[0082] See also Figure 8 , is a schematic diagram of the structure of an electronic device provided in the embodiment of this specification. Figure 8 As shown, the electronic device 800 may include: at least one processor 801 , at least one network interface 804 , a user interface 803 , a memory 805 , and at least one communication bus 802 .

[0083] The communication bus 802 is used to realize the connection and communication between these components.

[0084] The user interface 803 may include a display screen (Display) and a camera (Camera), and the optional user interface 803 may also include a standard wired interface and a wireless interface.

[0085] The network interface 804 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0086] Among them, the processor 801 may include one or more processing cores. The processor 801 uses various interfaces and lines to connect various parts within the entire server 800, and executes various functions and processes data of the server 800 by running or executing instructions, programs, code sets or instruction sets stored in the memory 805, and calling data stored in the memory 805. Optionally, the processor 801 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 801 can be a combination of one or more of an integrated processor (Central Processing Unit, CPU), an image processor (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 801, and it can be implemented separately through a chip.

[0087] Among them, the memory 805 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 805 includes a non-transitory computer-readable storage medium. The memory 805 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 805 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 805 may also be optionally at least one storage device located away from the aforementioned processor 801. As Figure 8 As shown, the memory 805 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a log data analysis application.

[0088] exist Figure 8 In the electronic device 800 shown, the user interface 803 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 801 can be used to call the log data analysis application stored in the memory 805, and specifically perform the following operations:

[0089] Get the log data corresponding to multiple distributed servers;

[0090] Analyze each of the log data according to at least one target analysis instruction to obtain at least one analysis result corresponding to each of the log data;

[0091] Generate an analysis report corresponding to the log data according to at least one analysis result corresponding to the log data.

[0092] In one embodiment, before the processor 801 executes the analysis of each log data according to at least one target analysis instruction to obtain at least one analysis result corresponding to each log data, it further executes:

[0093] Get the selection instruction corresponding to each log data;

[0094] The at least one target analysis instruction corresponding to the log data is determined based on the selection instruction corresponding to each log data.

[0095] In one embodiment, the processor 801 executes the selection instruction corresponding to each log data, specifically performing:

[0096] Obtaining the selection instruction input by the user for each log data;

[0097] Or, according to the identifier of each log data, a selection instruction corresponding to each log data is obtained.

[0098] In one embodiment, the at least one target analysis instruction includes at least the following instructions: data cleaning, data conversion, data integration, data source analysis, fault analysis, and operation status analysis.

[0099] In one embodiment, after the processor 801 generates an analysis report corresponding to the log data according to at least one analysis result corresponding to the log data, it further executes:

[0100] The log data and the analysis report corresponding to the log data are associated and stored.

[0101] In one embodiment, after the processor 801 executes the storing of associating the log data with the analysis report corresponding to the log data, it further executes:

[0102] By performing transfer learning on the associated stored historical log data and the analysis report corresponding to the historical log data, the target analysis instruction is updated based on preset conditions, so that the log data is analyzed according to the updated target analysis instruction.

[0103] In one embodiment, the preset conditions include at least one or more of the following: receiving an update instruction from a user, the historical log data stored in the log data analysis device exceeds a preset threshold, and a preset update frequency.

[0104] The electronic device provided in this specification can obtain log data corresponding to multiple distributed servers, analyze and process the log data corresponding to each distributed server according to at least one target analysis instruction, and generate an analysis report corresponding to each log data. By providing at least one log analysis module for each distributed server, each log analysis module can provide analysis and processing capabilities for log data to generate an analysis report for the log data of the distributed server, thereby avoiding the problem of difficulty in log data analysis caused by the complexity of the form and huge amount of log data due to the large number of server types and complex services in the distributed server architecture, and realizing intelligent analysis and efficient processing of log data in the distributed server architecture, so that users can quickly and easily monitor the operating status of multiple distributed servers through analysis reports.

[0105] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0106] The above disclosure is only the preferred embodiment of this specification, which certainly cannot be used to limit the scope of rights of this specification. Therefore, equivalent changes made according to the claims of this specification are still within the scope covered by this specification.

Claims

1. A log data analysis device, the device comprising: At least one log analysis module, log acquisition module and report generation module; The log acquisition module is used to acquire log data corresponding to multiple distributed servers; The log analysis module is used to analyze the log data according to the analysis instruction to obtain the analysis result corresponding to the log data; The report generating module is used to generate an analysis report corresponding to each log data according to at least one analysis result corresponding to the log data; The device also includes: an associated storage module; The associated storage module is used to associate and store the log data with the analysis report corresponding to the log data; The log analysis module is specifically used to perform transfer learning on the historical log data stored in the associated storage module and the analysis report corresponding to the historical log data, and to update the analysis instructions corresponding to the log analysis module based on preset conditions, so that the log analysis module analyzes the log data according to the updated analysis instructions.

2. The log data analysis device according to claim 1, further comprising: Network interface; The log acquisition module is specifically used to acquire the log data of the multiple distributed servers through the network interface.

3. The log data analysis device according to claim 2, wherein the network interface of the log data analysis device is connected to a switch, and the switch is respectively connected to the plurality of distributed servers; The log acquisition module is specifically configured to acquire the log data of the plurality of distributed servers from the switch through the network interface.

4. The log data analysis device according to claim 2, wherein the network interface of the log data analysis device is connected to the plurality of distributed servers; The log acquisition module is specifically configured to directly acquire the log data of the multiple distributed servers from the multiple distributed servers through the network interface.

5. The log data analysis device according to claim 4, wherein the log acquisition module is specifically used to acquire the log data of the distributed servers corresponding to each address through the network interface according to the address corresponding to each distributed server.

6. According to the log data analysis device of claim 4, the log acquisition module is specifically used to receive the log data sent by each of the distributed servers through the network interface.

7. The log data analysis device according to claim 1, further comprising: Target selection module; The target selection module is used to receive a user's selection instruction for a target log analysis module corresponding to target log data, so that the target log analysis module analyzes the target log data according to the selection instruction; wherein, the at least one log analysis module includes the target log analysis module, and the multiple log data include the target log data.

8. The log data analysis device according to claim 1, wherein the analysis instructions corresponding to each of the log analysis modules are different from each other; The log analysis module is specifically used to determine whether the log data needs to be analyzed based on the identification of the log data. If yes, the log data is analyzed according to the analysis instruction to obtain the analysis result corresponding to the log data. If no, the log data is not processed.

9. The log data analysis device according to claim 1, wherein the preset conditions include at least one or more of the following: receiving an update instruction from a user, the historical log data stored in the log data analysis device exceeding a preset threshold, and a preset update frequency.

10. According to any one of claims 1 to 9, the at least one analysis instruction comprises at least the following instructions: data cleaning, data conversion, data integration, data source analysis, fault analysis, and operation status analysis.

11. A log data analysis method, the method being applicable to the log data analysis device according to any one of claims 1 to 10, the method comprising: Get the log data corresponding to multiple distributed servers; Analyze each of the log data according to at least one target analysis instruction to obtain at least one analysis result corresponding to each of the log data; generating an analysis report corresponding to the log data according to at least one analysis result corresponding to the log data; After generating the analysis report corresponding to the log data according to at least one analysis result corresponding to the log data, the method further includes: storing the log data and the analysis report corresponding to the log data in association with each other; After the log data and the analysis report corresponding to the log data are stored in association, the method further includes: By performing transfer learning on the associated stored historical log data and the analysis report corresponding to the historical log data, the target analysis instruction is updated based on preset conditions, so that the log data is analyzed according to the updated target analysis instruction.

12. The method according to claim 11, before analyzing each of the log data according to at least one target analysis instruction to obtain at least one analysis result corresponding to each of the log data, further comprising: Get the selection instruction corresponding to each log data; The at least one target analysis instruction corresponding to the log data is determined based on the selection instruction corresponding to each log data.

13. The method according to claim 12, wherein obtaining the selection instruction corresponding to each log data comprises: Obtaining the selection instruction input by the user for each log data; Or, according to the identifier of each log data, a selection instruction corresponding to each log data is obtained.

14. According to the method of claim 11, the at least one target analysis instruction includes at least the following instructions: data cleaning, data conversion, data integration, data source analysis, fault analysis, and operation status analysis.

15. The method according to claim 11, wherein the preset conditions include at least one or more of the following: receiving an update instruction from a user, the historical log data stored in the log data analysis device exceeding a preset threshold, and a preset update frequency.

16. A computer storage medium storing a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the method steps according to any one of claims 11 to 15.

17. A computer program product, wherein the computer program product stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the method steps according to any one of claims 11 to 15.

18. An electronic device, comprising: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the method steps as claimed in any one of claims 11 to 15.

Images