A quality assurance method for master-slave redundancy automatic switching

Abstract

The application relates to a quality guarantee method for master-backup redundancy automatic switching, which comprises the following steps: establishing a master cache cluster and a backup cache cluster, wherein the master cache cluster and the backup cache cluster contain the same finished product data; checking the data integrity, data timeliness and data accuracy of the master cache cluster respectively, extracting the finished product data from the master cache cluster if all the three times of checking are passed, extracting the finished product data from the backup cache cluster if at least one time of checking fails; and assembling and outputting the finished product data.

Description

Technical Field

[0001] This invention relates to the field of data quality assurance, and in particular to a quality assurance method for automatic switching between primary and backup redundancy. Background Technology

[0002] Currently, some applications in the real-time data monitoring and assurance category have high requirements for data quality and functional reliability. They require the underlying big data platform to provide redundant backup functions for data collection, distribution, and processing, and to be able to quickly switch over in case of data processing failures.

[0003] However, due to the characteristics of real-time processing clusters, if the running application tasks fail and affect data processing, on the one hand, it is difficult to perform data aggregation and processing like batch processing; on the other hand, it will immediately affect the stability of real-time monitoring applications. Summary of the Invention

[0004] The purpose of this invention is to provide a quality assurance method for automatic switching of primary and backup redundancy. It aims to address the problems of multiple links in the existing real-time big data processing and the difficulty of timely troubleshooting. It uses the primary and backup redundancy of real-time computing clusters as data support and provides automatic detection methods to achieve automatic switching function at the second level, thereby improving the stability and effectiveness of real-time monitoring data quality and ensuring the integrity, reliability, stability and timeliness of the data consumed by upper-layer applications.

[0005] The technical solution adopted by this invention to solve the above-mentioned technical problems is to provide a quality assurance method for automatic switching of primary and backup redundancy, comprising:

[0006] Establish a primary cache cluster and a backup cache cluster, wherein the primary cache cluster and the backup cache cluster contain the same finished product data;

[0007] The data integrity, timeliness, and accuracy of the main cache cluster are verified respectively. If all three verifications pass, the finished data is extracted from the main cache cluster; if at least one verification fails, the finished data is extracted from the backup cache cluster.

[0008] The finished product data is assembled and output.

[0009] In one possible implementation, establishing the primary cache cluster and the backup cache cluster includes:

[0010] Obtain business signaling data;

[0011] The primary processing cluster and the backup processing cluster process the service signaling data respectively to generate finished product data;

[0012] Establish a primary cache cluster and a backup cache cluster to store the finished data generated by the primary processing cluster and the backup processing cluster, respectively.

[0013] In one possible implementation, after establishing the primary cache cluster and the standby cache cluster, the method further includes:

[0014] Receive finished product data requests from upper-layer applications.

[0015] In one possible implementation, assembling and outputting the finished product data includes:

[0016] The finished product data is assembled based on the requirements of the upper-layer application;

[0017] Output the assembled finished product data to an FTP server or a real-time data stream.

[0018] In one possible implementation, the step of verifying the data integrity, timeliness, and accuracy of the main cache cluster includes:

[0019] The integrity of the data in the main cache cluster is verified according to the preset integrity verification rules.

[0020] The timeliness of data in the main cache cluster is verified according to the preset timeliness verification rules.

[0021] The accuracy of the data in the main cache cluster is verified according to the preset accuracy verification rules.

[0022] In one possible implementation, the primary cache cluster and the backup cache cluster are built using Redis.

[0023] In one possible implementation, the service signaling data includes at least 2G service signaling, 4G service signaling, VoLTE service signaling, and 5G service signaling data.

[0024] In one possible implementation, the primary processing cluster and the backup processing cluster are built using the Apache Flink framework.

[0025] The quality assurance method for automatic switching between primary and backup redundancy proposed in this invention has the following advantages:

[0026] 1. Rapid automatic failover of primary and backup redundancy: Based on the data interface service, the function of automatic detection and automatic failover of abnormal data of primary and backup is realized. On the one hand, the impact of abnormal data of the underlying layer on the upper layer application is shielded, so that the redundancy failover is transparent to the upper layer application. On the other hand, the efficiency of primary and backup redundancy failover is improved.

[0027] 2. Configurable anomaly data verification rules: By making anomaly data judgment rules configurable, various custom verification mechanisms such as anomaly data integrity verification, volatility verification, and numerical verification can be implemented. Attached Figure Description

[0028] Figure 1 This is a flowchart of a quality assurance method for automatic switching of primary and backup redundancy disclosed in an embodiment of the present invention;

[0029] Figure 2 This is a schematic diagram of the process for verifying data integrity, timeliness, and accuracy as disclosed in an embodiment of the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] To facilitate understanding of the embodiments of the present invention, further explanations and descriptions will be provided below with reference to the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0032] Figure 1 This is a flowchart illustrating a quality assurance method for automatic switching between primary and backup redundancy, as disclosed in an embodiment of the present invention. Figure 1 The method includes at least the following steps:

[0033] Step S103: Establish a primary cache cluster and a backup cache cluster, wherein the primary cache cluster and the backup cache cluster contain the same finished product data; Step S105: Verify the data integrity, timeliness, and accuracy of the primary cache cluster respectively. If all three verifications pass, extract the finished product data from the primary cache cluster; if at least one verification fails, extract the finished product data from the backup cache cluster; Step S106: Assemble and output the finished product data.

[0034] First, in step S103, a primary cache cluster and a backup cache cluster are established, wherein the primary cache cluster and the backup cache cluster contain the same finished product data.

[0035] Specifically, a unified Redis data caching cluster is built, and dual cache groups Topic A and Topic B are created. Topic A corresponds to the primary cache cluster, and Topic B corresponds to the backup cache cluster.

[0036] In some possible implementations, prior to step S103, the method further includes:

[0037] Step S101: Obtain service signaling data.

[0038] Specifically, the original signaling data of various services is the basic data support for this method. This method collects various service signaling data reported by core network devices in real time. The service signaling data includes at least: 2G service signaling, 4G service signaling, VoLTE service signaling, and 5G service signaling data.

[0039] In step S102, the main processing cluster and the backup processing cluster process the service signaling data respectively to generate finished product data.

[0040] Specifically, based on the DPI (Deep Packet Inspection) signaling data of various services collected, two Apache Flink (hereinafter referred to as Flink) real-time processing clusters (primary and backup) read the same service signaling data and deploy the same real-time data statistics task. This task can complete the real-time statistical analysis function of relevant indicators required by the upper-layer real-time monitoring and protection application.

[0041] Accordingly, step S103 can be replaced by: establishing a primary cache cluster and a backup cache cluster to store the finished data generated by the primary processing cluster and the backup processing cluster, respectively.

[0042] Specifically, the primary cache cluster and the backup cache cluster store the finished product data of the guarantee index calculation of the two Flink real-time processing clusters, respectively, for data output.

[0043] Then, in step S105, the data integrity, timeliness and accuracy of the main cache cluster are verified respectively. If all three verifications pass, the finished data is extracted from the main cache cluster; if at least one verification fails, the finished data is extracted from the backup cache cluster.

[0044] Specifically, the data integrity of the main cache cluster is verified according to the preset integrity verification rules.

[0045] For example, taking 4G DPI as an example, the monitorable indicators include at least: page response success rate, game login success rate, TCP connection establishment success rate, HTTP download success rate, HTTP service response success rate, and HTTP service response latency, with the completeness judgment based on whether the value is empty.

[0046] The timeliness of data in the main cache cluster is verified according to the preset timeliness verification rules.

[0047] For example, taking 4G DPI as an example, the monitorable indicators include at least: page response success rate, game login success rate, TCP connection establishment success rate, HTTP download success rate, HTTP service response success rate, HTTP service response latency, etc. The timeliness judgment is based on the difference between the interface service layer reading time and the data generation timestamp being less than 5 minutes.

[0048] The accuracy of the data in the main cache cluster is verified according to the preset accuracy verification rules.

[0049] For example, taking 4G DPI as an example, the monitorable indicators include at least: page response success rate, game login success rate, TCP connection establishment success rate, HTTP download success rate, HTTP service response success rate, HTTP service response latency, etc., among which the success rate indicators are not less than 98% and the latency indicators are not higher than 200ms as the basis for judging accuracy.

[0050] In some possible implementations, step S105 can be replaced with:

[0051] Verify the data integrity of the primary cache cluster; if this fails, retrieve the finished data from the backup cache cluster.

[0052] If it passes, continue to verify the timeliness of the data in the main cache cluster; if it fails, extract the finished data from the standby cache cluster.

[0053] If it passes, continue to verify the accuracy of the data in the main cache cluster; if it fails, extract the finished data from the standby cache cluster.

[0054] If successful, retrieve the finished data from the main cache cluster.

[0055] In this implementation, if any verification fails, the finished data is retrieved directly from the backup cache cluster, without further verification, thus improving overall operational efficiency. The process is as follows: Figure 2 As shown.

[0056] In some possible implementations, prior to step S105, the method further includes:

[0057] Step S104: Receive the finished product data request from the upper layer application.

[0058] Finally, in step S106, the finished product data is assembled and output.

[0059] Specifically, the data interface service assembles the finished data according to the data requirements of the upper-layer protection scenario, generating real-time monitoring and protection indicator data files or data output streams of various dimensions. The assembled finished data is then output to an FTP server or a real-time data stream.

[0060] By using the technical solution described in this invention, when an upper-layer application requests real-time protection indicator data, the data interface service can perform abnormal data judgment and automatic switching of primary and backup redundancy, thus shielding the impact of lower-layer abnormal data on upper-layer real-time monitoring applications.

[0061] The data interface service extracts finished data from Topic A of the Redis data cache cluster and provides data output externally through data files or real-time interfaces. When a major network failure occurs in the Flink master cluster, causing abnormal task execution on the master cluster nodes, the data interface service can automatically monitor the data anomalies in Topic A of the Redis cluster's data cache and determine whether to automatically switch to reading data from Topic B of the Redis cluster's data cache, thereby achieving seamless switching of shared data when the Flink master cluster fails.

[0062] When the data interface service reads data for the next period, it continues to check whether the data in cache group Topic A is abnormal. If it is abnormal, it continues to read the data in cache group Topic B until the Flink main cluster recovers from the failure.

[0063] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0064] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0065] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A quality assurance method for master-backup redundancy automatic switching, characterized in that, The method comprises: acquiring service signaling data; a main processing cluster and a backup processing cluster respectively process the service signaling data to generate finished product data; the same real-time data statistical task is deployed in the main processing cluster and the backup processing cluster to generate the same finished product data; the main processing cluster and the backup processing cluster are real-time processing clusters; a main cache cluster and a backup cache cluster are established to respectively store the finished product data generated by the main processing cluster and the backup processing cluster; the data integrity, data timeliness and data accuracy of the main cache cluster are respectively verified, if all the three verifications are passed, the finished product data is extracted from the main cache cluster, if at least one verification fails, the finished product data is extracted from the backup cache cluster; the finished product data is assembled and output; the verification of the data integrity, data timeliness and data accuracy of the main cache cluster comprises: the data integrity of the main cache cluster is verified according to the preset integrity verification rule; the data timeliness of the main cache cluster is verified according to the preset timeliness verification rule; the data accuracy of the main cache cluster is verified according to the preset accuracy verification rule.

2. The method of claim 1, wherein, After the main cache cluster and the backup cache cluster are established, the method further comprises: receiving a finished product data request from an upper-layer application.

3. The method of claim 1, wherein, the assembling and outputting of the finished product data comprises: the finished product data is assembled according to the demand information of the upper-layer application; the assembled finished product data is output to an FTP server or a real-time data stream.

4. The method of claim 1, wherein, The main cache cluster and the backup cache cluster are constructed using Redis.

5. The method of claim 1, wherein, The service signaling data at least comprises 2G service signaling, 4G service signaling, VoLTE service signaling and 5G service signaling data.

6. The method of claim 1, wherein, The main processing cluster and the backup processing cluster are constructed using Apache Flink framework.

Images