A method and system for analyzing business call log relationships based on memory cache

Abstract

The present invention provides a method and system for analyzing business call log relationships based on memory cache, which relates to the technical field of distributed data relationship processing, including: connecting the log stream data of the business service call process to a stream processing platform; saving it in the memory cache, continuing to circulate the unified serial number as the stream data, and assigning a time window grouping mark to the stream data; judging whether the time window corresponding to the stream data is open according to the time window grouping mark, if it is open, then appending the stream data to the corresponding time window, otherwise creating a new time window; closing the time window after reaching the opening time, triggering the analysis of all stream data in the time window; and saving the analysis results. The beneficial effects of the invention are: using the memory cache technology to analyze the business service call process log after cloudification and containerization, using a very small time window for online data flow analysis, obtaining and storing the business call relationship results in quasi-real time, and providing an efficient means for quickly discovering business system failures.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed data relationship processing, and in particular to a method and system for analyzing business call log relationships based on memory cache. Background Art

[0002] After the system is cloud-based and containerized, the amount of system service call relationship log data output becomes huge and the relationships become complex, making it difficult to locate problems caused by the business service call process in a timely manner.

[0003] Based on this, the present invention designs an analysis method for business call log relationships based on memory cache, which analyzes problems in log data online with low latency and realizes timely positioning of problems generated in the business service process. Summary of the invention

[0004] In response to the above problems, the present invention provides a method and system for analyzing business call log relationships based on memory cache, which can accurately and quickly display business information in a variety of business scenarios by matching parameters of business types in the database.

[0005] To achieve the above object, the present invention provides a method for analyzing business call log relationships based on memory cache, comprising:

[0006] Connect the log stream data of the business service call process to the stream processing platform;

[0007] Filter out data that does not meet the requirements in the log stream data;

[0008] The filtered log stream data is stored in a memory cache, the unified serial number is used as the stream data to continue to flow, and a time window grouping mark is assigned to the stream data according to the unified serial number;

[0009] When the stream data arrives at the time window, it is determined whether the time window corresponding to the stream data is opened according to the time window grouping flag. If it is opened, the stream data is appended to the corresponding time window. If it is not opened, a new time window is created.

[0010] The time window is closed after reaching the opening time, and the analysis of all the flow data in the time window is triggered at the same time;

[0011] Save the analysis results.

[0012] As a further improvement of the present invention, the data in the log stream that does not meet the requirements includes:

[0013] The format of the unified serial number is incorrect;

[0014] The difference between the date and time in the unified serial number and the log output timestamp exceeds 8 hours;

[0015] The operation codes of business services are not within the scope of analysis;

[0016] Duplicate logs.

[0017] As a further improvement of the present invention, the step of storing the filtered log stream data in a memory cache comprises:

[0018] In the cache information, the primary key is the unified flow field of the log stream data, and the value is the stream data log structure of the collection type.

[0019] As a further improvement of the present invention, a time window grouping mark is allocated to the stream data according to a unified serial number; comprising:

[0020] The unified serial number is divided by the number of time windows, and the remainder is used as the time window grouping mark.

[0021] As a further improvement of the present invention, the time window is preset with an opening duration, and is automatically closed after the opening duration is reached.

[0022] As a further improvement of the present invention, the analysis of all the flow data within the time window includes:

[0023] By using the unified serial number, the corresponding complete log stream data is matched and obtained in the memory cache;

[0024] The business call relationship is obtained according to the call relationship field, the execution information field and the result status field in the complete log stream data.

[0025] As a further improvement of the present invention, when saving the analysis result, a unified serial number is used as the analysis result ID, and the analysis result is saved on the ElasticSearch cluster.

[0026] As a further improvement of the present invention, the analysis results can be queried in the ElasticSearch cluster by unifying the serial number.

[0027] As a further improvement of the present invention, the process of querying and analyzing the results in the ElasticSearch cluster includes:

[0028] Obtain link node summary information and business link node information through a unified serial number;

[0029] Analyze the call relationship according to the business link node information;

[0030] The summary information and the call relationship are presented on a page.

[0031] The present invention also discloses a business call log relationship analysis system based on memory cache, comprising: a data access module, a data filtering module, a data cache module, a data screening module, a data analysis module and a data storage module;

[0032] The data access module is used to:

[0033] Connect the log stream data of the business service call process to the stream processing platform;

[0034] The data filtering module is used to:

[0035] Filter out data that does not meet the requirements in the log stream data;

[0036] The data cache module is used to:

[0037] The filtered log stream data is stored in a memory cache, the unified serial number is used as the stream data to continue to flow, and a time window grouping mark is assigned to the stream data according to the unified serial number;

[0038] The data screening module is used for:

[0039] When the stream data arrives at the time window, it is determined whether the time window corresponding to the stream data is opened according to the time window grouping flag. If it is opened, the stream data is appended to the corresponding time window. If it is not opened, a new time window is created.

[0040] The data analysis module is used to:

[0041] The time window is closed after reaching the opening time, and the analysis of all the flow data in the time window is triggered at the same time;

[0042] The data storage module is used to:

[0043] Save the analysis results.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] The present invention uses cloud-based and containerized business service call process logs, utilizes memory caching technology, and uses a very small time window to perform online data flow analysis, obtains and stores business call relationship results in quasi-real time, and provides an efficient means to quickly discover business system failures, reduce the time spent on manual problem troubleshooting and locating, save costs, and improve labor productivity.

[0046] The present invention filters the log stream data during the data stream processing to obtain effective data to simplify the data volume. At the same time, when the stream data is circulated, the amount of circulating data is simplified, unnecessary information flow is reduced, and resource consumption is reduced.

[0047] The present invention stores the results on an ElasticSearch cluster, and can provide a monitoring platform with the functions of problem alarm and problem cause analysis result presentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A flow chart of a method for analyzing business call log relationships based on memory cache disclosed in an embodiment of the present invention;

[0049] Figure 2 A schematic diagram of a business call log relationship analysis system based on memory cache disclosed in an embodiment of the present invention;

[0050] Figure 3 A flowchart of querying and presenting results stored in an ElasticSearch cluster disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0053] Example:

[0054] like Figure 1 As shown, the present invention provides a method for analyzing business call log relationships based on memory cache, comprising:

[0055] S1. Connect the log stream data of the business service call process to the stream processing platform;

[0056] Among them, the stream processing platform can be selected from flink or storm stream processing platform.

[0057] S2, filter out the data in the log stream that does not meet the requirements;

[0058] Among them, include:

[0059] The format of the unified serial number (Trace_ID) is incorrect;

[0060] The difference between the date and time in the unified serial number (Trace_ID) and the log output timestamp exceeds 8 hours. This indicates that the data corresponding to the unified serial number is backlog data. Therefore, it is not necessary to obtain its call relationship to determine whether there is a problem.

[0061] The OP code of the business service is not within the scope of analysis. This means that there is no need to divide the calling relationship of the business into stages.

[0062] Duplicate logs do not need to be analyzed repeatedly.

[0063] S3, save the filtered log stream data in the memory cache, continue to circulate the unified serial number as the stream data, and assign a time window grouping mark to the stream data according to the unified serial number;

[0064] in,

[0065] In the cache information, the primary key is the unified flow field of the log stream data, and the value is the stream data log structure of the collection type;

[0066] By continuing to circulate the unified serial number as flow data, unnecessary information can be reduced during the circulation process, thus reducing resource consumption;

[0067] Divide the unified serial number by the number of time windows, and use the remainder as the time window grouping mark. For example, if the unified serial number is 202103051007 and the number of time windows is 5, the window grouping mark of the unified serial number is 202103051007%5=2.

[0068] S4, when the stream data arrives at the time window, it is determined whether the time window corresponding to the stream data is opened according to the time window grouping flag. If it is opened, the stream data is appended to the corresponding time window. If it is not opened, a new time window is created;

[0069] in,

[0070] For example, if there are five time windows, which are numbered 1, 2, 3, 4, and 5, the corresponding window grouping marks are serial numbers 1, 2, 3, 4, and 0, respectively.

[0071] further,

[0072] Still taking the unified serial number of 202103051007 as an example, when it is the first serial number that the stream data platform just accesses for processing, all time windows are not open at this time, so window No. 2 is not opened, and the corresponding window No. 2 is opened;

[0073] The preset time window is opened for 15 seconds each time. If window No. 2 is opened for 10 seconds, serial number 202103051012 arrives at the time window, and its window grouping flag is also 2, then when it arrives, it is judged that the corresponding time window has been opened, and the serial number 202103051012 is added to time window No. 2.

[0074] S5: The time window is closed after reaching the opening time, and the analysis of all flow data in the time window is triggered at the same time;

[0075] in,

[0076] Through the unified serial number (Trace_ID), the corresponding complete log stream data is matched and obtained in the memory cache, including link node summary information (business service) and business link node information (merged input and output parameter logs);

[0077] The business call relationship is obtained according to the call relationship field, execution information field and result status field in the complete log stream data.

[0078] S6. Save the analysis results.

[0079] in

[0080] When saving, the unified serial number is used as the analysis result ID, and the analysis result is saved on the ElasticSearch cluster.

[0081] like Figure 3 As shown, the present invention can query and analyze the results in the ElasticSearch cluster through a unified serial number, and the process includes:

[0082] Obtain link node summary information and business link node information through a unified serial number;

[0083] Analyze the call relationship based on the business link node information;

[0084] Present summary information and call relationships to the page.

[0085] like Figure 2 As shown, a business call log relationship analysis system based on memory cache includes: a data access module, a data filtering module, a data cache module, a data screening module, a data analysis module and a data storage module;

[0086] Data access module, used for:

[0087] Connect the log stream data of the business service call process to the stream processing platform;

[0088] Data filtering module for:

[0089] Filter out the data in the log stream that does not meet the requirements;

[0090] Data cache module, used for:

[0091] The filtered log stream data is saved in the memory cache, the unified serial number is used as the stream data to continue to flow, and a time window grouping mark is assigned to the stream data according to the unified serial number;

[0092] Data screening module, used for:

[0093] When the stream data arrives at the time window, it is determined whether the time window corresponding to the stream data is open according to the time window grouping flag. If it is open, the stream data is appended to the corresponding time window. If it is not open, a new time window is created.

[0094] Data analysis module for:

[0095] The time window is closed after reaching the opening duration, and the analysis of all flow data in the time window is triggered at the same time;

[0096] Data storage module, used for:

[0097] Save the analysis results.

[0098] Advantages of the present invention:

[0099] (1) The cloud-based and containerized business service call process logs are analyzed online using memory caching technology and a very small time window. The business call relationship results are acquired and stored in near real time, providing an efficient means to quickly discover business system failures, reducing the time spent on manual problem troubleshooting and location, saving costs, and improving labor productivity.

[0100] (2) During the data stream processing, the log stream data is filtered to obtain valid data in order to simplify the data volume. At the same time, when the stream data is circulated, the amount of circulating data is simplified, unnecessary information flow is reduced, and resource consumption is reduced.

[0101] (3) The results are stored in the ElasticSearch cluster, which can provide the monitoring platform with problem alerts and problem cause analysis results.

[0102] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for analyzing business call log relationships based on memory cache, characterized in that: include: Connect the log stream data of the business service call process to the stream processing platform; Filter out data that does not meet the requirements in the log stream data; The filtered log stream data is stored in a memory cache, the unified serial number is used as the stream data to continue to circulate, and a time window grouping mark is assigned to the stream data according to the unified serial number; wherein the filtered log stream data is stored in a memory cache; including: in the cache information, the primary key is the unified serial field of the log stream data, and the value is a set type stream data log structure; according to the unified serial number, a time window grouping mark is assigned to the stream data; including: dividing the unified serial number by the number of time windows, and the remainder is used as the time window grouping mark; When the stream data arrives at the time window, it is determined whether the time window corresponding to the stream data is opened according to the time window grouping flag. If it is opened, the stream data is appended to the corresponding time window. If it is not opened, a new time window is created. The time window is closed after reaching the opening time, and the analysis of all the flow data in the time window is triggered at the same time; wherein the analysis of all the flow data in the time window includes: matching and obtaining the corresponding complete log flow data in the memory cache through the unified serial number; obtaining the business call relationship according to the call relationship field, execution information field and result status field in the complete log flow data; Save the analysis results.

2. The method for analyzing business call log relationships according to claim 1, characterized in that: The data in the log stream that does not meet the requirements includes: The format of the unified serial number is incorrect; The difference between the date and time in the unified serial number and the log output timestamp exceeds 8 hours; The operation codes of business services are not within the scope of analysis; Duplicate logs.

3. The method for analyzing business call log relationships according to claim 1, characterized in that: The time window is preset with an opening duration, and is automatically closed after the opening duration is reached.

4. The method for analyzing business call log relationships according to claim 1, characterized in that: When saving the analysis result, a unified serial number is used as the analysis result ID, and the analysis result is saved on the ElasticSearch cluster.

5. The method for analyzing business call log relationships according to claim 4, characterized in that: By using a unified serial number, the analysis results can be queried in the ElasticSearch cluster.

6. The method for analyzing business call log relationships according to claim 4, characterized in that: The process of querying and analyzing the results of the ElasticSearch cluster includes: Obtain link node summary information and business link node information through a unified serial number; Analyze the call relationship according to the business link node information; The summary information and the call relationship are presented on a page.

7. A system for implementing the method for analyzing business call log relationships based on memory cache according to any one of claims 1 to 6, characterized in that: include: Data access module, data filtering module, data cache module, data screening module, data analysis module and data storage module; The data access module is used to: Connect the log stream data of the business service call process to the stream processing platform; The data filtering module is used to: Filter out data that does not meet the requirements in the log stream data; The data cache module is used to: The filtered log stream data is stored in a memory cache, the unified serial number is used as the stream data to continue to flow, and a time window grouping mark is assigned to the stream data according to the unified serial number; The data screening module is used for: When the stream data arrives at the time window, it is determined whether the time window corresponding to the stream data is opened according to the time window grouping flag. If it is opened, the stream data is appended to the corresponding time window. If it is not opened, a new time window is created. The data analysis module is used to: The time window is closed after reaching the opening time, and the analysis of all the flow data in the time window is triggered at the same time; The data storage module is used to: Save the analysis results.

Images