Distributed high-performance service calling link real-time monitoring method and system

Abstract

The invention discloses a distributed high-performance service calling link real-time monitoring method and a distributed high-performance service calling link real-time monitoring system, and provides a distributed calling information acquisition and analysis method based on buffering, acquisition, cold and hot storage and real-time processing. A cache queue is constructed by utilizing a distributed cache to serve as a buffer layer for delivering calling information, and the calling information of the buffer layer is transferred to a storage cluster through an acquisition cluster at regular time, so that normal reading and writing of the storage cluster under a high-concurrency request are ensured, and lossless acquisition of the calling information is realized; the method comprises the following steps of: performing regular self-inspection in a storage cluster, extracting invalid historical calling information from real-time storage, compressing the invalid historical calling information into a file, and storing the file into distributed storage, so that hot storage of real-time data and cold backup of historical data are realized, and the problem of access performance deterioration caused by continuous expansion of the real-time storage along with time is solved; by constructing a joint index of real-time storage and distributed storage, efficient retrieval and real-time processing of mass data are realized.

Description

Technical Field

[0001] The present invention relates to the field of big data analysis and processing, and in particular to a distributed high-performance service call link real-time monitoring method and system. Background Art

[0002] This section merely provides background information related to the present disclosure and is not necessarily prior art.

[0003] In the information age, raw data often needs to go through various service nodes, such as extraction, transformation, and outlier handling, before it can be transformed into final, user-friendly data with a certain level of usability. The order in which these different service nodes are called is the call chain for the raw data. When final data cannot be obtained normally or does not meet expectations, it is necessary to obtain the call chain corresponding to the raw data to observe the data flow, analyze, and address the fault points.

[0004] Although the currently proposed call link monitoring methods can capture information before and after service calls, they do not consider how to ensure real-time acquisition and non-loss of call link information in high-concurrency scenarios, resulting in the call link information being unable to timely and accurately reflect the current data flow situation. They also do not consider the storage problem of massive call link information, resulting in the continuous expansion of the storage occupied by the call link monitoring service during its continuous operation, reducing the performance of the storage system and placing a huge burden on the maintenance of the software system. Summary of the Invention

[0005] In response to the above-mentioned defects or improvement needs of the existing technology, a distributed high-performance service call link real-time monitoring method and system are provided. The method specifically includes the following steps:

[0006] Step 1: Based on the configuration, determine whether the current call link needs to be monitored. If monitoring is required, encapsulate the call information and asynchronously deliver it to the call information buffer queue.

[0007] Step 2: extract data from the call information buffer queue regularly according to the configured time interval, and store the extracted data into real-time storage through the call information storage module.

[0008] Step 3: Regularly check the validity of data in real-time storage, store the valid data in real-time storage, compress the invalid data into files and store them in distributed storage, and build a joint index of real-time storage and distributed storage.

[0009] Step 4: Analyze the data in real-time storage and distributed storage based on the joint index built in Step 3, generate statistical indicators, trigger abnormal alarms in real time, restore the full-link topology, and calculate the service value score based on the rules.

[0010] Furthermore, step 1 includes:

[0011] Step 1-1: Determine whether the call link needs to be monitored based on the service configuration. If no monitoring is required, the service is called normally.

[0012] Step 1-2: Encapsulate the call link to be monitored into call information in JSON format. The call information includes the call object, call type (query, delete, update and modify), whether the call is successful, call duration, call start time and call end time.

[0013] Step 1-3: asynchronously deliver the call information obtained in step 1-2 to the call information buffer queue.

[0014] Furthermore, step 2 includes:

[0015] Step 2-1: Create a transmission channel to extract data from the call information buffer queue obtained in step 1 according to the configured time interval.

[0016] Step 2-2: The data extracted from the call information buffer queue is stored in the real-time storage through the call information storage module.

[0017] Furthermore, step 3 includes:

[0018] Step 3-1: Configure the valid time range of the data, such as within three months, within one year, etc.

[0019] Step 3-2: regularly detect the data in real-time storage to determine whether it is within the valid time range. If it is within the valid time range, it is determined to be valid; otherwise, it is invalid.

[0020] In step 3-3, the expired data is compressed into a file, and then divided into several fragments according to the file size and stored in distributed storage respectively.

[0021] Steps 3-4: Build indexes for real-time storage and distributed storage.

[0022] Furthermore, step 4 includes:

[0023] Statistical analysis: Obtain statistical information such as the total number of system calls, success rate, and average access latency. Abnormal alerts: Obtain information about abnormal service status, service offline status, and service duration. Link tracking: Obtain detailed information about the service's call link. Value assessment: Calculate the service's value score based on custom evaluation rules.

[0024] A distributed, high-performance, real-time monitoring system for service call links comprises a call information delivery module, a call information collection module, a call information storage module, and a call information processing module. The call information delivery module detects the current call link and encapsulates the call information; the call information collection module retrieves the call information provided by the call information delivery module; the call information storage module stores the data obtained by the call information collection module in real-time storage; and the data support processing module performs real-time analysis based on the data provided by the call information storage module.

[0025] Preferably, the call information delivery module also includes a call status monitoring unit, a call information sending unit and a call information cache queue; the call status detection unit is used to detect whether the current call link needs to deliver call information; the call information sending unit is used to encapsulate the basic information of the current call link and asynchronously deliver it to the call information cache queue; the call information buffer queue is used to temporarily store the call information delivered by the call information sending unit. The so-called asynchronous delivery means that the task of delivering call information is independent of the service call and does not interfere with the normal call of the service.

[0026] Preferably, the call information collection module also includes a collection task scheduling unit, a collection data transmission unit and a collection data transfer unit; the collection task scheduling unit is used to manage the scheduling tasks of the collection module, and regularly collect the call information in the call information buffer queue according to the configuration scheduling information; the collection data transmission unit is used to establish a data transmission channel between the call information buffer queue, the call information collection module and the call information storage module to realize the flow of collected data; the collection data transfer unit is used to store the data collected from the call information buffer queue into real-time storage through the call information storage module.

[0027] Preferably, the call information storage module also includes a data aging detection unit, a real-time data storage unit, a data compression unit, a distributed storage unit and a joint index construction unit; the data aging detection unit is used to perform periodic detection on the data stored in the call information storage module according to the configured effective time, and filter out real-time data and historical data; the real-time data storage unit is used to store the real-time data filtered out by the data aging detection unit; the data compression unit is used to compress the historical data filtered out by the data aging detection unit into a file; the distributed storage unit is used to store the file compressed by the data compression unit; the joint index construction unit is used to perform joint index construction on the data in the real-time data storage unit and the distributed storage unit.

[0028] Preferably, the call information processing module also includes a statistical analysis unit, an abnormal alarm unit, a link tracking unit and a value evaluation unit; the statistical analysis unit is used to process the data stored in the real-time data storage unit and the distributed storage unit, and provide statistical data such as the total number of calls, call success rate and average call delay; the abnormal alarm unit is used to analyze the real-time data in the real-time data storage unit, and generate alarm information such as service abnormal offline, slow service response and service abnormal cancellation; the link tracking unit is used to analyze the data stored in the real-time data storage unit and the distributed storage unit, and generate a call link based on the call relationship; the value evaluation unit is used to analyze the data stored in the real-time data storage unit and the distributed storage unit, normalize the number of users, visits and activity of the calls, and add up to calculate the value score, where the sum of the coefficients of the number of users, visits and activity is equal to 1.

[0029] In general, the method of the present invention can achieve the following beneficial effects compared with the existing technical solutions:

[0030] The most outstanding technical feature of the present invention is that a call information buffer queue based on distributed cache is adopted between the call information and the storage cluster, and the call information is persisted from the buffer queue to the storage cluster through a separate acquisition module, thereby ensuring that the call information is not lost and significantly improving the anti-concurrency capability of the storage cluster.

[0031] The present invention realizes the hot and cold isolation of call information storage. The more recent data is stored in real-time storage, and the historical data completion files are stored in distributed storage. In the scenario of massive data, the storage performance is significantly improved.

[0032] The present invention realizes the joint index creation of real-time storage and distributed storage, and significantly improves the performance of data retrieval and processing in massive data scenarios.

[0033] The present invention realizes a statistical analysis service based on the stored call information, activates the call information, and provides a powerful auxiliary means for the healthy and stable operation of the monitored software system.

[0034] This paper proposes a distributed call information collection and analysis method based on buffering, collection, hot and cold storage, and real-time processing. This method uses distributed cache to build a cache queue as a buffer layer for call information delivery. The collection module cluster periodically transfers the call information from the buffer layer to the storage cluster, ensuring normal reading and writing of the storage cluster under high-concurrency requests, thereby achieving lossless collection of call information. The storage cluster performs regular self-inspections internally, extracting qualified historical call information from real-time storage and compressing it into files for storage in distributed storage. This achieves hot storage of real-time data and cold backup of historical data, resolving the performance issues associated with the continuous expansion of real-time storage over time. By constructing a joint index for real-time and distributed storage, efficient retrieval and real-time processing of massive data are achieved. This method implements distributed, high-performance monitoring of call information in high-concurrency scenarios, significantly improving the collection efficiency and storage performance of call information and providing a powerful auxiliary means for the healthy and stable operation of software systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more apparent.

[0036] Figure 1 A distributed, high-performance service call link real-time monitoring method and system architecture diagram disclosed in an embodiment of the present invention.

[0037] Figure 2 A distributed, high-performance service call link real-time monitoring method and system composition diagram disclosed in an embodiment of the present invention.

[0038] Figure 3 This is a diagram of the composition of the call information delivery module of the system of the present invention.

[0039] Figure 4 This is a diagram showing the composition of the call information collection module of the system of the present invention.

[0040] Figure 5 This is a diagram of the call information storage module composition of the system of the present invention.

[0041] Figure 6 This is a diagram of the call information processing module composition of the system of the present invention.

[0042] Figure 7 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0043] like Figure 1-2 The figure shows a distributed high-performance service call link real-time monitoring method and system architecture disclosed in an embodiment of the present invention, showing the data flow and hierarchical relationship of the method, such as Figure 2The figure shows a composition diagram of a distributed high-performance service call link real-time monitoring system disclosed in an embodiment of the present invention, including the following modules: a call information delivery module 1, a call information collection module 2, a call information storage module 3, a call information processing module 4, a call status detection unit 5, a call information sending unit 6, a call information buffer queue 7, a collection task scheduling unit 8, a collection data storage unit 9, a collection data transfer unit 10, a data timeliness detection unit 11, a real-time data storage unit 12, a data compression unit 13, a distributed storage unit 14, a joint index construction unit 15, a statistical analysis unit 16, an abnormal alarm unit 17, a link tracking unit 18 and a value evaluation unit 19.

[0044] Among them Figure 3 As shown, the call information delivery module 1 includes a call status detection unit 5, a call information sending unit 6, and a call information buffer queue 7. During the service call process, the call status detection unit first determines whether call monitoring is required. If call monitoring is required, it encapsulates the relevant information of the call process, including the caller, call type, call duration, call start time, and call end time, and then asynchronously delivers the encapsulated call information to the call information buffer queue.

[0045] like Figure 4 As shown, the call information collection module 2 includes a collection task scheduling unit 8, a collection task transmission unit 9, and a collection data transfer unit 11. The collection task scheduling unit creates a data collection channel through the collection data transmission unit according to the configured time interval, regularly reads the caller information from the call information buffer queue, and stores the data in the real-time storage through the call information storage module via the collection data transfer unit.

[0046] like Figure 5 As shown, the call information storage module 3 includes a data aging detection unit 11, a real-time data storage unit 12, a data compression unit 13, a distributed storage unit 14, and a joint index construction unit 15. The data aging detection unit periodically detects the data in the real-time data storage unit according to a specified aging range. When the data exceeds the validity period, the data is compressed into a file and stored in the distributed storage unit. When the data does not exceed the validity period, the data is still stored in the real-time storage unit. Finally, the joint index construction unit simultaneously constructs an index for the data in the real-time data storage unit and the distributed storage unit.

[0047] like Figure 6As shown, the call information processing module 4 includes a statistical analysis unit 16, an abnormal alarm unit 17, a link tracking unit 18, and a value evaluation unit 19. The statistical analysis unit is used to process the data in the real-time data storage unit and the distributed storage unit and obtain statistical data such as the total number of calls, success rate, and average access delay; the abnormal alarm unit is used to process the data in the real-time data storage unit and output corresponding alarm information according to the alarm rules; the link tracking unit is used to process the real-time data storage unit and the distributed storage unit and generate the call link corresponding to each request; the value evaluation unit is used to process the real-time data storage unit and the distributed storage unit and generate the value score of the corresponding service according to the value evaluation rules.

[0048] like Figure 7 As shown, a distributed high-performance service call link real-time monitoring method includes the following steps:

[0049] Step 1: Start monitoring and asynchronous delivery. Based on the configuration, determine whether the current call link needs to be monitored. If monitoring is required, encapsulate the call information and asynchronously deliver it to the call information buffer queue.

[0050] Step 2: regularly transfer the call information, extract data from the call information buffer queue regularly according to the configured time interval, and store the extracted data into the real-time storage through the call information storage module.

[0051] Step 3: Data validity maintenance and joint index construction. Regularly check the validity of data in real-time storage, store valid data in real-time storage, compress expired data into files and store them in distributed storage, and build a joint index for real-time storage and distributed storage.

[0052] Step 4: Data analysis and optimization: Analyze the data in real-time storage and distributed storage based on the joint index built in step 3, generate statistical indicators, trigger abnormal alarms in real time, restore the full-link topology, and calculate the service value score based on rules.

[0053] The workflow of the present invention is as follows:

[0054] When in use, the service automatically determines whether to monitor the call link during the call process. If monitoring is required, the corresponding caller, call type, call start time, call end time and other information will be asynchronously delivered to the call information buffer queue. The call collection module will extract the data in the call information buffer queue at regular intervals according to the configured time interval and store it in real-time storage. The storage cluster will automatically and regularly detect the timeliness of the stored data. If the data is not within the valid time, the data will be compressed into a file and stored in distributed storage. Finally, an index will be jointly built for the real-time storage and distributed storage.

[0055] When the user opens the call link monitoring software page, he first clicks on the statistical analysis module to view the overall call status of the system, such as the total number of calls, call success rate and average access delay. When the call success rate is found to be low or the average access delay is large, you can click on the abnormal alarm module to view alarms such as abnormal service offline, access time and status abnormality, and determine the priority of abnormal handling based on the service value score calculated by the value assessment module. When handling service abnormalities, you can click on the link tracking unit to view the data flow of each node in the service call link, identify the abnormal nodes and repair them. Finally, the system's call success rate increases and the average access delay decreases.

[0056] The present invention provides a distributed, high-performance method and system for real-time monitoring of service call links. While there are numerous methods and approaches for implementing this technical solution, the foregoing description represents only a preferred embodiment of the present invention. It should be noted that those skilled in the art may make improvements and modifications without departing from the principles of the present invention, and such improvements and modifications are also within the scope of protection of the present invention. Any components not specified in this embodiment may be implemented using existing technologies.

Claims

1. A distributed high-performance service call link real-time monitoring method, characterized in that: The steps include: Step 1: Determine whether the current call link needs to be monitored based on the configuration. If monitoring is required, encapsulate the call information and asynchronously send it to the call information buffer queue; Step 2: extract data from the call information buffer queue regularly according to the configured time interval, and store the extracted data into real-time storage through the call information storage module; Step 3: Regularly check the validity of data in real-time storage, store the valid data in real-time storage, compress the invalid data into files and store them in distributed storage, and build a joint index of real-time storage and distributed storage; Step 4: Analyze the data in real-time storage and distributed storage based on the joint index built in Step 3, generate statistical indicators, trigger abnormal alarms in real time, restore the full-link topology, and calculate the service value score based on the rules.

2. The distributed high-performance service call link real-time monitoring method according to claim 1 is characterized in that: Step 1 includes: Step 1-1: Determine whether the call link needs to be monitored based on the service configuration. If not, the service is called normally. Step 1-2: Encapsulate the call link to be monitored into call information; Step 1-3: asynchronously deliver the call information obtained in step 1-2 to the call information buffer queue.

3. The distributed high-performance service call link real-time monitoring method according to claim 2 is characterized in that: Step 2 includes: Step 2-1: Create a transmission channel to extract data from the call information buffer queue obtained in step 1 according to the configured time interval; Step 2-2, the data extracted from the call information buffer queue is stored in the real-time storage through the call information storage module; 4. The distributed high-performance service call link real-time monitoring method according to claim 3 is characterized in that: Step 3 includes: Step 3-1, configure the valid time range of the data; Step 3-2, regularly checking whether the data in the real-time storage is within the valid time range, if it is within the valid time range, it is set to be valid, otherwise it is invalid; Step 3-3: compress the invalid data into files and store them in distributed storage; Step 3-4: Jointly build the index of real-time storage and distributed storage.

5. The distributed high-performance service call link real-time monitoring method according to claim 4 is characterized in that: Step 4 includes: Statistical analysis to obtain statistical information such as the total number of system calls, success rate, and average access latency; Abnormal alarm, obtain abnormal information such as abnormal service status, abnormal service offline and abnormal service time consumption of the system; Link tracking, obtaining detailed information about the service call link; Value assessment: calculate the value score of the service based on customized assessment rules.

6. A distributed high-performance service call link real-time monitoring system, characterized in that: A distributed high-performance service call link real-time monitoring method according to any one of claims 1 to 5 is adopted, comprising a call information delivery module (1), a call information collection module (2), a call information storage module (3) and a call information processing module (4); the call information delivery module (1) detects the current call link and encapsulates the call information, the call information collection module (2) calls the call information provided by the call information delivery module (1), the call information storage module (3) saves the data obtained by the call information collection module (2) into real-time storage, and the data support processing module (4) realizes real-time analysis through the data provided by the call information storage module (3).

7. The distributed high-performance service call link real-time monitoring system according to claim 6 is characterized in that: The call information delivery module (1) includes a call status monitoring unit (5), a call information sending unit (6) and a call information cache queue (7); the call status detection unit (5) detects whether the current call link needs to deliver call information; the call information sending unit (6) encapsulates the information of the current call link and asynchronously delivers it to the call information cache queue (7); the call information buffer queue (7) temporarily stores the call information delivered by the call information sending unit (6); the asynchronous delivery means that the task of delivering the call information is independent of the service call and does not interfere with the normal call of the service.

8. The distributed high-performance service call link real-time monitoring system according to claim 7 is characterized in that: The call information acquisition module (2) comprises an acquisition task scheduling unit (8), an acquisition data transmission unit (9) and an acquisition data transfer unit (10); the acquisition task scheduling unit (8) manages the scheduling tasks of the call information acquisition module (2) and regularly collects the call information in the call information buffer queue (7) according to the configuration scheduling information; the acquisition data transmission unit (9) establishes a data transmission channel between the call information buffer queue (7), the call information acquisition module (2) and the call information storage module (3) to realize the flow of the collected data; the acquisition data transfer unit (10) stores the data collected from the call information buffer queue (7) into the real-time storage through the call information storage module (2).

9. The distributed high-performance service call link real-time monitoring system according to claim 8 is characterized in that: The call information storage module (3) comprises a data aging detection unit (11), a real-time data storage unit (12), a data compression unit (13), a distributed storage unit (14) and a joint index construction unit (15); the data aging detection unit (11) performs a periodic detection on the data stored in the call information storage module (3) according to the configured effective time, and filters out real-time data and historical data; the real-time data storage unit (12) stores the real-time data filtered out by the data aging detection unit (11); the data compression unit (13) compresses the historical data filtered out by the data aging detection unit (11) into a file; the distributed storage unit (14) stores the file compressed by the data compression unit (13); and the joint index construction unit (15) performs a joint index construction on the data in the real-time data storage unit (12) and the distributed storage unit (14).

10. The distributed high-performance service call link real-time monitoring system according to claim 9 is characterized in that: The call information processing module (4) includes a statistical analysis unit (16), an abnormal alarm unit (17), a link tracking unit (18) and a value evaluation unit (19); the statistical analysis unit (16) processes the data stored in the real-time data storage unit (12) and the distributed storage unit (14), and provides statistical data such as the total number of calls, call success rate and average call delay; the abnormal alarm unit (17) analyzes the real-time data in the real-time data storage unit (12) and generates alarm information such as service abnormal offline, service slow response and service abnormal cancellation; the link tracking unit (18) analyzes the data stored in the real-time data storage unit (12) and the distributed storage unit (14), and generates a call link according to the call relationship; the value evaluation unit (19) analyzes the data stored in the real-time data storage unit (12) and the distributed storage unit (14), normalizes the number of users, the number of visits and the activity of the calls, and adds them up to calculate the value score, wherein the sum of the sum coefficients of the number of users, the number of visits and the activity is equal to 1.

Images