Real-time index analysis method, system and equipment, medium and program product

Abstract

The invention discloses an index real-time analysis method, system and device, a medium and a program product. The method comprises the steps that firstly, a log collection tool Logagent is used for collecting service logs of a server; the service log is stored in a kafka cluster; and finally, based on a preset storm topology, according to a preset configuration rule, performing index analysis processing on each index in the business log to obtain an index analysis result. According to the method, non-invasive index real-time analysis is realized, existing system codes do not need to be changed on a large scale, a user can flexibly configure data analysis rules according to own requirements, and the efficiency and accuracy of index data real-time analysis are effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of data analysis, and in particular to a method, system, device, medium and program product for real-time analysis of indicators. Background Art

[0002] In an online production environment, it is often necessary to quickly provide data such as the magnitude, trend, year-on-year and month-on-month changes of certain dimensional indicators of a certain interface service. Therefore, how to quickly, non-invasively and in real time provide this data, and how to display dimensional indicators based on flexible configurations have become issues that need to be urgently addressed in this field.

[0003] The existing traditional approach requires intrusive writing of a large amount of non-functional code in the business code to meet the need to view certain indicators and their dimensional trends after the business is launched. Therefore, the traditional approach often makes the business project more bloated. Whether using the aspect approach or adding similar codes in certain interfaces, they will become lengthy and distorted logic units.

[0004] In addition, the existing technology lacks flexibility, has many business fields, complex business angles, and cannot take all aspects into consideration in the business code, resulting in cumbersome operations and low efficiency.

[0005] Therefore, it is urgent to invent a real-time indicator analysis method to solve the problems of low analysis efficiency and accuracy of existing indicator analysis methods and the use of intrusive codes leading to bloated business projects. Summary of the invention

[0006] In view of this, embodiments of the present invention provide a method, system, device, medium and program product for real-time analysis of indicators, which at least partially solve the problems existing in the prior art.

[0007] Other features and advantages of the present invention will become apparent from the following detailed description, or may be learned in part by practice of the present invention.

[0008] In order to achieve the above purpose, the embodiment of the present invention provides the following technical solutions:

[0009] According to a first aspect of an embodiment of the present invention, a method for real-time analysis of indicators is provided, the method comprising:

[0010] Get the service log of the server;

[0011] The business log is stored in the kafka cluster;

[0012] Based on the preset storm topology, each indicator in the business log is analyzed and processed according to the preset configuration rules to obtain an indicator analysis result.

[0013] Furthermore, obtain the service log of the server, including:

[0014] Use the log collection tool Logagent to collect service logs of the server.

[0015] Furthermore, based on the preset storm topology, each indicator in the business log is analyzed and processed according to the preset configuration rules to obtain an indicator analysis result, including:

[0016] Performing random grouping processing on the business logs in the kafka cluster to obtain first group data;

[0017] For each of the first grouped data, based on a preset configuration rule, an indicator analysis process is performed on each indicator in the first grouped data to obtain an indicator analysis result.

[0018] Further, based on a preset configuration rule, an indicator analysis process is performed on each indicator in the first grouped data to obtain an indicator analysis result, including:

[0019] Performing field grouping processing on the first grouped data according to the preset grouping field in the preset configuration rule to obtain second grouped data;

[0020] For each of the second grouped data, counting the number of occurrences of each indicator in the second grouped data according to a preset period to obtain an indicator statistical result, wherein the indicator is an object to be analyzed that is customized in the preset configuration rule;

[0021] Indicator data analysis is performed according to the indicator statistical results to obtain indicator analysis results.

[0022] Further, based on the preset storm topology, each indicator in the business log is analyzed and processed according to the preset configuration rules to obtain the indicator analysis result, which also includes:

[0023] According to the preset cycle, the indicator analysis results are stored in the MongoDB database using the upsert method.

[0024] Furthermore, the preset configuration rule is an indicator analysis rule configured according to JSONpath or js script;

[0025] The preset grouping fields include business line, AppName, interface name, indicator name, dimension name, and dimension value;

[0026] The indicator data analysis includes dimensional analysis, magnitude analysis, trend analysis, and year-on-year and month-on-month analysis.

[0027] According to a second aspect of an embodiment of the present invention, a real-time indicator analysis system is provided, the system comprising:

[0028] Log collection module, used to obtain the service logs of the server;

[0029] An intermediate storage module is used to store the business log in the Kafka cluster;

[0030] The indicator operation module is used to perform indicator analysis and processing on each indicator in the business log based on a preset storm topology and according to preset configuration rules to obtain an indicator analysis result.

[0031] According to a third aspect of an embodiment of the present invention, there is provided a device for real-time analysis of indicators, the device comprising: a processor and a memory;

[0032] The memory is used to store one or more program instructions;

[0033] The processor is used to run one or more program instructions to execute the steps of a real-time indicator analysis method as described in any one of the above items.

[0034] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of a real-time indicator analysis method as described in any one of the above items are implemented.

[0035] According to a fifth aspect of an embodiment of the present invention, a computer program product is provided, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, which, when executed by a computer, enable the computer to implement the steps of a real-time indicator analysis method as described in any one of the above items.

[0036] The embodiments of the present invention have the following advantages:

[0037] Real-time monitoring and early warning: Through real-time data collection and processing, real-time monitoring of interface service indicators can be achieved, which facilitates timely detection of abnormal situations and triggers early warning mechanisms, helping users to quickly respond to and handle problems.

[0038] Flexible data analysis: Users can flexibly configure data analysis rules according to their own needs, including dimensions, magnitude, trends, year-on-year and month-on-month comparisons, so that they can quickly conduct multi-dimensional data analysis and discover potential patterns and trends in indicator data.

[0039] Non-invasive integration: It realizes non-invasive data extraction and analysis, and can be seamlessly integrated with existing systems without the need for large-scale changes to existing system codes, thus reducing the complexity and risk of integration. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0041] Figure 1 A flowchart of a method for real-time analysis of indicators provided by an embodiment of the present invention;

[0042] Figure 2 One of the principle schematic block diagrams of a method for real-time analysis of indicators provided by an embodiment of the present invention;

[0043] Figure 3 A schematic diagram of an indicator configuration based on JSONpath provided in an embodiment of the present invention;

[0044] Figure 4 A schematic diagram of an indicator configuration based on a js script method provided by an embodiment of the present invention;

[0045] Figure 5 A schematic diagram for comparing script engines provided in an embodiment of the present invention;

[0046] Figure 6 A second schematic block diagram of the principle of a real-time indicator analysis method provided by an embodiment of the present invention;

[0047] Figure 7 A schematic diagram of the overall architecture corresponding to a real-time indicator analysis method provided by an embodiment of the present invention;

[0048] Figure 8 One of the schematic diagrams of the real-time analysis results of the visualization indicators provided by the embodiment of the present invention;

[0049] Fig. 9 The second schematic diagram of the real-time analysis results of the visualization indicators provided in the embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0051] In addition, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or elements is not necessarily limited to those steps or elements explicitly listed, but may include other steps or elements not explicitly listed or inherent to such process, method, product, or apparatus.

[0052] Figure 1 The figure shows a flow chart of a method for real-time analysis of indicators according to an embodiment of the present invention.

[0053] like Figure 1 As shown, the real-time indicator analysis method according to an embodiment of the present invention may include step S100, step S200 and step S300.

[0054] refer to Figure 2 and Figure 6 In step S100, the service log of the server is obtained.

[0055] Specifically, the above steps include:

[0056] Use the log collection tool Logagent to collect service logs printed by the server.

[0057] Next, in step S200, the business log is stored in the Kafka cluster.

[0058] Specifically, the above steps include:

[0059] In the Kafka cluster, synchronize a copy of the business log data to mapi-kafka, and use the data in mapi-kafka to perform subsequent processing.

[0060] The embodiment of the present invention collects and stores server business data through a streaming transmission method to ensure the real-time performance of data analysis.

[0061] In step S300, based on the preset storm topology and according to the preset configuration rules, an indicator analysis process is performed on each indicator in the business log to obtain an indicator analysis result.

[0062] Specifically, the above steps include:

[0063] First, the logSpout data sending node in the preset storm topology is used to write the business log data to the resolveBolt node. Then, the resolveBolt node is used to perform random grouping (shuffleGrouping) on ​​the business logs stored in the kafka cluster to obtain several first grouped data. In addition, the grouped data can also be used for other business nodes (appBizLogBolt).

[0064] For each first grouped data, based on the metricGroupingBolt node in the preset storm topology, field grouping processing is performed on the first grouped data according to the preset grouping field in the preset configuration rule of the node to obtain a plurality of second grouped data.

[0065] Among them, the above preset grouping fields include business line, AppName, interface name, indicator name, dimension name, and dimension value.

[0066] The above-mentioned preset configuration rules are indicator analysis rules configured according to JSONpath or js scripts. The embodiment of the present invention uses java's ScriptEngine script engine to parse the script language and supports dynamic code execution. After the code is modified, there is no need to restart the JVM process and it can be directly executed by parsing or compiling, which is very convenient.

[0067] Specifically, by comparing Rhino, Nashorn, J2V8, and Groovy scripting engines, and considering the performance and current JDK version, Nashorn scripting engine is preferred. If performance problems are found later, consider switching to j2v8 engine. The scripting engine comparison diagram is shown below. Figure 5 shown.

[0068] In order to ensure that the dimensional indicators of different interfaces can be configured at any time, the embodiment of the present invention chooses to use the configuration method to construct the indicator analysis rules. Since the interface input and output parameter headers are recorded in JSON format, the JSONpath method is preferred. At the same time, in order to support complex business matching and processing, configuration in a scripting manner is also supported.

[0069] Then, for each second group data, the number of occurrences of each indicator in the second group data is counted according to a preset period (1 minute) to obtain an indicator statistical result, wherein the indicator is an object to be analyzed that is customized in the preset configuration rule. Figure 3 The figure shows a schematic diagram of indicator configuration based on JSONpath. Figure 4 The figure shows a schematic diagram of indicator configuration based on js script.

[0070] Then, the indicator data analysis is performed based on the indicator statistical results to obtain the indicator analysis results, wherein the above indicator data analysis includes dimension analysis, magnitude analysis, trend analysis, and year-on-year and month-on-month analysis.

[0071] Finally, the realTimeCalculationBolt node in the preset storm topology is used to aggregate the data within the preset period (1 minute) using the delay queue method, and then the upsert method is used to accumulate the minute-level data of each aggregation into the MongoDB database to ensure that the upload time is accurate and without delay.

[0072] Finally, the indicator analysis results are visualized on the Tiny front end. Figure 8 and Fig. 9 The figure shows a schematic diagram of the visualization effect of the indicator analysis results.

[0073] in addition, Figure 7 The figure shows an overall architecture diagram corresponding to a real-time indicator analysis method provided by an embodiment of the present invention.

[0074] The embodiment of the present invention uses real-time data acquisition technology to instantly capture the data flow of the interface service, and uses streaming processing technology to process the data in real time, thereby ensuring the timeliness and accuracy of the data, so that users can obtain the latest indicator data and indicator analysis results.

[0075] The embodiment of the present invention also provides a flexible rule configuration engine, allowing users to customize data analysis rules as needed, such as dimension, magnitude, trend, year-on-year and month-on-month changes, etc. Such a design allows users to dynamically configure data analysis rules according to different needs without modifying the system code.

[0076] The embodiments of the present invention minimize the intrusion into the existing system to ensure that the impact of the data collection and analysis processing on the existing system is minimized, and achieves non-intrusive method for extracting and analyzing indicator data, thereby avoiding modification of the existing code.

[0077] The analyzed and processed indicator data is stored in an appropriate data storage system, and an efficient query interface is provided so that users can retrieve and analyze the data as needed. At the same time, a visual interface is provided to users, and the indicator trend data of the interface service is intuitively displayed in the form of charts, reports, etc., to help users quickly understand the data and make decisions.

[0078] In addition, the embodiment of the present invention further provides a real-time indicator analysis system, which specifically includes:

[0079] Log collection module, used to obtain the service logs of the server;

[0080] The intermediate storage module is used to store business logs in the Kafka cluster;

[0081] The indicator operation module is used to analyze and process each indicator in the business log based on the preset storm topology and according to the preset configuration rules to obtain the indicator analysis results.

[0082] In addition, an embodiment of the present invention also provides a device, which includes: a processor and a memory; the memory is used to store one or more program instructions; the processor is used to run one or more program instructions to execute the steps of a real-time indicator analysis method as described above.

[0083] In addition, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned method for real-time analysis of indicators are implemented.

[0084] In addition, an embodiment of the present invention further provides a computer program product, which includes computer program instructions, and when the computer program instructions are executed by a processor, the steps of the above-mentioned real-time indicator analysis method are implemented.

[0085] The embodiments of the present invention have the following advantages:

[0086] Real-time monitoring and early warning: Through real-time data collection and processing, real-time monitoring of interface service indicators can be achieved, which facilitates timely detection of abnormal situations and triggers early warning mechanisms, helping users to quickly respond to and handle problems.

[0087] Flexible data analysis: Users can flexibly configure data analysis rules according to their own needs, including dimensions, magnitude, trends, year-on-year and month-on-month comparisons, so that they can quickly conduct multi-dimensional data analysis and discover potential patterns and trends in indicator data.

[0088] Non-invasive integration: It realizes non-invasive data extraction and analysis, and can be seamlessly integrated with existing systems without the need for large-scale changes to existing system codes, thus reducing the complexity and risk of integration.

[0089] Intelligent decision support: The indicator trend data of the interface service is displayed through a visual interface, so that users can intuitively understand the data to make more intelligent decisions and optimize business processes and resource allocation.

[0090] Efficient data storage and query: The processed data is stored in an appropriate data storage system, and an efficient query interface is provided so that users can quickly retrieve and analyze historical data, supporting the analysis and comparison of historical trends.

[0091] In summary, the embodiments of the present invention can help users quickly obtain specific interface service indicator data in a production environment, realize real-time monitoring, flexible data analysis, non-invasive integration, intelligent decision support, and efficient data storage and query, thereby improving business operation efficiency and assisting user decision-making.

[0092] In the embodiment of the present invention, the processor may be an integrated circuit chip having the ability to process signals. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, or discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of the present invention may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present invention may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The processor reads the information in the storage medium and completes the steps of the above method in combination with its hardware. The storage medium may be a memory, for example, a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct RAM bus random access memory (DRRAM).The storage medium described in the embodiment of the present invention is intended to include but is not limited to these and any other suitable types of memory. Those skilled in the art should be aware that in one or more of the above examples, the functions described in the present invention can be implemented by a combination of hardware and software. When the software is applied, the corresponding function can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication medium includes any medium that is convenient for transmitting a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access. Although the present invention has been described in detail above with general descriptions and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to it on the basis of the present invention. Therefore, these modifications or improvements made on the basis of not departing from the spirit of the present invention all belong to the scope of protection claimed in the present invention.

Claims

1. A real-time indicator analysis method, characterized in that: The method comprises: Get the service log of the server; The business log is stored in the kafka cluster; Based on the preset storm topology, each indicator in the business log is analyzed and processed according to the preset configuration rules to obtain an indicator analysis result.

2. A real-time indicator analysis method as claimed in claim 1, characterized in that: Get the service logs of the server, including: Use the log collection tool Logagent to collect service logs of the server.

3. A real-time indicator analysis method as claimed in claim 1, characterized in that: Based on the preset storm topology, each indicator in the business log is analyzed and processed according to the preset configuration rules to obtain the indicator analysis results, including: Performing random grouping processing on the business logs in the kafka cluster to obtain first group data; For each of the first grouped data, based on a preset configuration rule, an indicator analysis process is performed on each indicator in the first grouped data to obtain an indicator analysis result.

4. A real-time indicator analysis method as claimed in claim 3, characterized in that: Based on the preset configuration rules, each indicator in the first grouped data is subjected to indicator analysis processing to obtain an indicator analysis result, including: Performing field grouping processing on the first grouped data according to the preset grouping field in the preset configuration rule to obtain second grouped data; For each of the second grouped data, counting the number of occurrences of each indicator in the second grouped data according to a preset period to obtain an indicator statistical result, wherein the indicator is an object to be analyzed that is customized in the preset configuration rule; Indicator data analysis is performed according to the indicator statistical results to obtain indicator analysis results.

5. A real-time indicator analysis method as claimed in claim 3, characterized in that: Based on the preset storm topology, each indicator in the business log is analyzed and processed according to the preset configuration rules to obtain the indicator analysis results, which also include: According to the preset cycle, the indicator analysis results are stored in the MongoDB database using the upsert method.

6. A real-time indicator analysis method as claimed in claim 4, characterized in that: The preset configuration rules are indicator analysis rules configured according to JSONpath or js script; The preset grouping fields include business line, AppName, interface name, indicator name, dimension name, and dimension value; The indicator data analysis includes dimensional analysis, magnitude analysis, trend analysis, and year-on-year and month-on-month analysis.

7. A real-time indicator analysis system, characterized in that: The system comprises: Log collection module, used to obtain the service logs of the server; An intermediate storage module is used to store the business log in the Kafka cluster; The indicator operation module is used to perform indicator analysis and processing on each indicator in the business log based on a preset storm topology and according to preset configuration rules to obtain an indicator analysis result.

8. A real-time indicator analysis device, characterized in that: The device comprises: a processor and a memory; The memory is used to store one or more program instructions; The processor is used to run one or more program instructions to execute the steps of a real-time indicator analysis method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a real-time indicator analysis method as described in any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product includes computer program instructions, which, when executed by a processor, implement the steps of a real-time indicator analysis method as described in any one of claims 1 to 6.

Images