Core network hidden danger detection method, device, electronic device and storage medium
By preprocessing the real-time alarm flow of the core network and inputting the abnormal detection model, combined with kernel density estimation, the problem of difficulty in discovering hidden faults in the existing technology is solved, the timeliness and accuracy of fault detection is improved, and the cost.
Patent Information
- Application Number
- CN202110387834.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-04-09
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2041-04-09
AI Technical Summary
It is difficult to effectively detect hidden faults in the core network in the existing technology. The service dial-up method has lag, high cost, large resource usage, poor threshold adaptability of the performance index method and the failure cause cannot be checked.
By accessing real-time alarm flow for preprocessing, the alarm characteristic data is obtained and the abnormal detection model is input, and the abnormality detection model is corrected in combination with kernel density estimation, and the abnormal positioning results are output to detect hidden faults.
It improves the timeliness and accuracy of fault detection, reduces the duration of network failures, reduces costs, and realizes timely detection of hidden dangers in the core network.
Smart Images

Figure CN115209452B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of network maintenance, and in particular to a method, device, electronic equipment and storage medium for troubleshooting hidden dangers in a core network. Background Art
[0002] The communication network is divided into core network, wireless network, transmission network and data network. The core network is the brain of the entire network, which is responsible for the control and distribution of the overall business. With the development of technologies such as NFV and 5G, the core network architecture has become increasingly complex, and core network failures have gradually changed from the traditional dominant dominant to a combination of dominant and recessive distribution. Compared with explicit failures, which can be directly detected by observing alarms, recessive failures are more hidden, similar to diseases that are still in the latent stage, and can only be discovered through a series of analytical and diagnostic methods.
[0003] At present, there is a large amount of operation data in the existing core network, which reflects the operation status of the network from different dimensions. Therefore, for the hidden dangers and faults of the core network, it is necessary to start with these network operation data and find the abnormal points. There are two main methods for discovering hidden dangers and faults in the core network, the service dialing method and the performance indicator method:
[0004] The service dialing method is the mainstream method at present. This method simulates users using network services. Under normal network conditions, the dialing test will be completed smoothly. When a fault occurs, the service dialing test will fail and prompt the maintenance personnel to discover the fault. The performance indicator method sets thresholds for various service KPI indicators in network operation. When the indicator is lower than the threshold, a relevant alarm will be generated and the maintenance personnel will be prompted to discover the fault.
[0005] It can be found that the business dialing method has the following disadvantages:
[0006] 1) Network faults discovered by simulating user behavior cannot be strictly called the discovery of hidden faults, because such faults have actually occurred in the network and caused some network services to fail. Therefore, the service dialing method has a certain lag;
[0007] 2) It takes up a lot of dial-up card resources and needs to frequently execute real business, which will put pressure on some networks with smaller capacity;
[0008] 3) Actual business charges will be generated, and the cost expenditure is relatively high;
[0009] 4) The network operation data introduced is too little to conduct an overview analysis of the overall operation of the network.
[0010] The performance index method has the following disadvantages:
[0011] 1) Faults are discovered by setting thresholds. However, fixed thresholds make this method less adaptable. For example, the decrease in business volume at night will increase the fluctuation of performance indicators, increasing the probability of exceeding the threshold. However, fixed thresholds cannot take this factor into account.
[0012] 2) Since the network is developing dynamically, fixed performance thresholds cannot always represent the changing network, so the thresholds need to be manually analyzed and corrected frequently;
[0013] 3) Although network failures can be discovered, the cause of the failure cannot be further verified. Summary of the invention
[0014] The present invention provides a core network hidden danger troubleshooting method, device, electronic equipment and storage medium, which are used to solve the defect that only relatively obvious network faults can be troubleshooted in the prior art.
[0015] In a first aspect, the present invention provides a method for troubleshooting core network risks, comprising:
[0016] Accessing a real-time alarm stream, preprocessing the real-time alarm stream, and obtaining alarm characteristic data;
[0017] Acquire the data to be determined, input the data to be determined and the alarm feature data into an anomaly detection model, and obtain an initial anomaly determination result;
[0018] The initial abnormality determination result is corrected based on kernel density estimation, and an abnormality positioning result is output.
[0019] In one embodiment, the acquiring of the real-time alarm stream and preprocessing of the real-time alarm stream to obtain the alarm characteristic data specifically includes:
[0020] Access the Kafka real-time alarm stream interface, obtain the full alarm field of the preset core network, perform denoising, standardization and time slicing on the full alarm field, and obtain pre-processed alarm data;
[0021] Vectorizing the preprocessed alarm data, extracting the vectorized preprocessed alarm data according to a preset characteristic field, and obtaining the alarm characteristic data having the same dimension as the preset characteristic field;
[0022] The alarm vectorization data is classified according to performance data, dialing data, engineering data and resource data to obtain the alarm characteristic data.
[0023] In one embodiment, the access to the Kafka real-time alarm stream interface, obtaining the full alarm field of the preset standard core network, and performing denoising, standardization and time slicing processing on the full alarm field to obtain pre-processed alarm data specifically includes:
[0024] Removing the noise alarm corresponding to the empty field or abnormal field in the full alarm field to obtain the alarm text;
[0025] Unifying different expressions in the same field in the alarm text and outputting a standardized alarm text;
[0026] The standardized alarm text is sliced and sampled according to a preset slice duration to obtain the pre-processed alarm data.
[0027] In one embodiment, the acquiring of the data to be determined and inputting the data to be determined and the alarm feature data into an anomaly detection model to obtain an initial anomaly determination result specifically includes:
[0028] Acquire the data to be determined with preset dimensions, and classify them in combination with the alarm feature data to obtain model input data;
[0029] The model input data is input into the anomaly detection model, and after negative removal, anomaly detection and voting, the initial anomaly determination result is obtained.
[0030] In one embodiment, the obtaining of the to-be-determined data with preset dimensions and combining the alarm feature data for classification to obtain the model input data specifically includes:
[0031] Acquire performance data, dial test data, engineering data and resource data respectively to form the data to be determined;
[0032] Clean the alarm feature data to generate a new input field set, where the input field set includes size, diff1day, diff7day and pool_diff;
[0033] Filling the missing data in the data to be determined with 0 to obtain the filled data to be determined;
[0034] The input field set and the padded data to be determined are classified according to the signaling plane and the user plane to obtain the model input data.
[0035] In one embodiment, the correcting the initial abnormality determination result based on kernel density estimation and outputting the abnormality location result specifically includes:
[0036] Preprocessing the multidimensional data corresponding to the initial abnormality determination result to obtain preprocessing result data;
[0037] Input the preprocessing result data into the kernel density KDE model for density fitting to obtain a kernel density probability estimate;
[0038] The kernel density probability estimate is corrected to obtain the maximum value of the change degree in the data dimension, which is the anomaly positioning result.
[0039] In one embodiment, the preprocessing of the multidimensional data corresponding to the initial abnormality determination result to obtain preprocessing result data specifically includes:
[0040] For the signaling plane performance index, the signaling plane preprocessing result data is obtained by subtracting the original index value from 1;
[0041] For the user plane performance index, the user plane preprocessing result data is obtained based on the original value of the index, the maximum value of the index and the minimum value of the index;
[0042] For alarm indicators, based on the original value of the indicator, the maximum value of the indicator and the minimum value of the indicator, Gaussian distribution transformation is performed to obtain the alarm preprocessing result data.
[0043] In a second aspect, the present invention further provides a core network hidden danger troubleshooting device, comprising:
[0044] A preprocessing module, used to access the real-time alarm stream, preprocess the real-time alarm stream, and obtain alarm feature data;
[0045] A processing module, used for obtaining data to be determined, inputting the data to be determined and the alarm feature data into an anomaly detection model, and obtaining an initial anomaly determination result;
[0046] The correction module is used to correct the initial abnormality determination result based on kernel density estimation and output the abnormality positioning result.
[0047] In a third aspect, the present invention further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the core network hidden danger troubleshooting methods described above are implemented.
[0048] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the core network hidden danger troubleshooting methods described above.
[0049] The core network hidden danger troubleshooting method, device, electronic device and storage medium provided by the present invention can timely discover relatively hidden faults in the network and improve the judgment speed and accuracy by preprocessing the alarm flow, then performing network alarm vectorization, inputting the abnormal detection model for detection, and finally performing correction through kernel density estimation. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0051] Figure 1 It is a flow chart of the core network hidden danger troubleshooting method provided by the present invention;
[0052] Figure 2 It is a schematic diagram of the alarm data processing process provided by the present invention;
[0053] Figure 3 is a calculation flow chart of the anomaly detection model provided by the present invention;
[0054] Figure 4 It is a kernel density estimation flow chart provided by the present invention;
[0055] Figure 5 It is a kernel density probability correction and output flow chart provided by the present invention;
[0056] Figure 6 It is a structural schematic diagram of a core network hidden danger troubleshooting device provided by the present invention;
[0057] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0058] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0059] Figure 1 It is a flowchart of the core network hidden danger troubleshooting method provided by the present invention, such as Figure 1 As shown, including:
[0060] 101, accessing a real-time alarm stream, preprocessing the real-time alarm stream, and obtaining alarm characteristic data;
[0061] 102, obtaining data to be determined, inputting the data to be determined and the alarm feature data into an anomaly detection model to obtain an initial anomaly determination result;
[0062] 103. Correct the initial abnormality determination result based on kernel density estimation and output an abnormality positioning result.
[0063] Specifically, based on the network alarm stream, the present invention pre-processes the real-time alarm stream and proposes an alarm vectorization method, which converts the alarm from a text file into a vector model that can be applied to mathematical calculations. In order to discover the necessity of network hidden dangers and failures as soon as possible, an anomaly detection model is designed, and a method for processing model input data is proposed. In addition, through the setting of voting logic, it is expected to reduce the instability caused by the anomaly detection algorithm alone. In order to find the main hidden danger points in the hidden dangers, a method for finding the root cause of the hidden danger through kernel density estimation is proposed, and a method for processing the input data of the kernel density estimation model is designed. Finally, a data post-correction method based on expert experience is also combined. This method can shield the negative impact of certain specific scenarios on the accuracy in data analysis.
[0064] It is understandable that the more timely the hidden dangers and faults are discovered, the less time the network will be blocked and the less impact on users' business. By shortening the time it takes to discover hidden dangers and faults, the duration of network failures can be reduced, achieving the goal of reducing costs and increasing efficiency. At present, according to the average monthly failure volume and business losses in a certain place, if the hidden danger discovery time is shortened by 10 minutes, the average annual cost can be saved by about 1.4 million yuan.
[0065] The present invention pre-processes the alarm flow, then performs network alarm vectorization, inputs the abnormal detection model for detection, and finally corrects it through kernel density estimation, so as to timely discover relatively hidden faults in the network and improve the judgment speed and accuracy.
[0066] Based on the above embodiment, step 101 in the method specifically includes:
[0067] Access the Kafka real-time alarm stream interface, obtain the full alarm field of the preset core network, perform denoising, standardization and time slicing on the full alarm field, and obtain pre-processed alarm data;
[0068] Vectorizing the preprocessed alarm data, extracting the vectorized preprocessed alarm data according to a preset characteristic field, and obtaining the alarm characteristic data having the same dimension as the preset characteristic field;
[0069] The alarm vectorization data is classified according to performance data, dialing data, engineering data and resource data to obtain the alarm characteristic data.
[0070] Among them, the access to the Kafka real-time alarm stream interface, obtaining the full alarm field of the preset standard core network, and performing denoising, standardization and time slicing processing on the full alarm field to obtain pre-processed alarm data, specifically includes:
[0071] Removing the noise alarm corresponding to the empty field or abnormal field in the full alarm field to obtain the alarm text;
[0072] Unifying different expressions in the same field in the alarm text and outputting a standardized alarm text;
[0073] The standardized alarm text is sliced and sampled according to a preset slice duration to obtain the pre-processed alarm data.
[0074] Specifically, the first step is to access, standardize and preprocess the real-time alarm stream:
[0075] Connect to the Kafka real-time alarm stream interface of the fault management system, access the alarms of the core networks of the three domains 5GC, NFV, and EPC, and the related full alarm fields; remove noise alarms with empty or abnormal field values. The significance of this step is to reduce the number of alarms to be calculated and reduce the computing pressure; perform alarm standardization operations, and standardize the different expressions that may exist in the same field in the alarm text. For example, if the computer room information contains two expressions, the computer room code and the Chinese name of the computer room, they are uniformly modified to the Chinese name form; perform 5-minute slice preprocessing operations on the access alarms, and perform sampling every 5 minutes according to the alarm occurrence time. For example, alarms with an alarm occurrence time of 0:00-0:05 will form a set, with a total of 12*24=288 sets throughout the day. Figure 2 shown.
[0076] Then comes alarm vectorization. Since alarm data is continuous text, in order to facilitate system operations, the alarm needs to be reconstructed into a computable vector model. At the same time, since the alarm contains many fields, some of which (such as manufacturer alarm ID, work order number) have no value for data analysis and calculation, it is necessary to select characteristic fields that can represent the characteristics of the alarm from all fields and construct them into an N-dimensional vector to achieve fast and efficient calculation.
[0077] The following feature fields are extracted:
[0078] Alarm occurrence time, first-level specialty, second-level specialty, equipment type, province, city, manufacturer, alarm title, network element name, alarm location object name, circuit name, alarm unique identifier, alarm clearing status, network element alias, alarm elimination time, monitoring level, alarm level, peer network element, and business system.
[0079] Each of the above 19 feature fields represents an attribute of the alarm. All fields are constructed into a 19-dimensional vector to form an absolutely unique alarm for subsequent calculations.
[0080] Next, access the performance data, dial test data, engineering data, and resource data respectively:
[0081] Connect to the real-time performance data interface of the performance management platform and receive performance files via FTP, including 5-minute performance data of 4 / 5G signaling plane and user plane network element granularity. The performance data is shown in Table 1:
[0082] Table 1
[0083]
[0084] The dial test alarm data interface of the fault management system is connected, and the data is consumed through Kafka, including the 4G EPC domain dial test alarm data;
[0085] Connect to the engineering cutover data interface of the network operation and maintenance management platform, and receive files via FTP, including engineering information in the EPC, vEPC, and 5GC network fields;
[0086] The network topology resource interface that connects to the resource management system receives files via FTP, including EPC MME / SAEGW pool information and NFV / 5GC vertical correlation resource information.
[0087] Based on any of the above embodiments, step 102 in the method specifically includes:
[0088] Acquire the data to be determined with preset dimensions, and classify them in combination with the alarm feature data to obtain model input data;
[0089] The model input data is input into the anomaly detection model, and after negative removal, anomaly detection and voting, the initial anomaly determination result is obtained.
[0090] The step of obtaining the to-be-determined data with preset dimensions and classifying the data in combination with the alarm feature data to obtain the model input data specifically includes:
[0091] Acquire performance data, dial test data, engineering data and resource data respectively to form the data to be determined;
[0092] Clean the alarm feature data to generate a new input field set, where the input field set includes size, diff1day, diff7day and pool_diff;
[0093] Filling the missing data in the data to be determined with 0 to obtain the filled data to be determined;
[0094] The input field set and the padded data to be determined are classified according to the signaling plane and the user plane to obtain the model input data.
[0095] Specifically, the alarm data input above needs to be cleaned and processed to form new input fields before entering the anomaly detection model. The input fields are:
[0096] 1) Size: This field represents the number of alarms in the current 5-minute time slice. The field is processed by grouping the alarm set obtained through the slice preprocessing operation according to the characteristic field 'Network Element Name'. Alarms with the same 'Network Element Name' form a set, and delete alarms with 'Alarm Level' not 1, 2, or 3. Then, it is determined whether each alarm in the time slice has a clearing time. That is, if the alarm is cleared within the slice time, the corresponding alarm is deleted. Finally, the total number of alarms in the set is calculated and assigned to size;
[0097] 2) diff1day: This field represents the difference between the alarm volume of the current 5-minute time slice and the alarm volume of the 5-minute time slice of the same time period yesterday. The calculation formula is:
[0098] diff1day=size today -size yesterday (1)
[0099] Where size today is the current size value, size yesterday The size value at the same time yesterday;
[0100] 3) diff7day: This field represents the difference between the alarm volume of the current 5-minute time slice and the alarm volume of the 5-minute time slice of the same period last week. The calculation formula is:
[0101] diff7day=size today -size lastweek (2)
[0102] size yesterday The size value at the same time last week;
[0103] 4) pool_diff: This field represents the difference between the size value of this network element and the average size value in the pool. The calculation formula is:
[0104]
[0105] pool_diff=size today -size poolava (4)
[0106] Where size poolava is the average size in the pool, n is the number of network elements in the pool, ∑ poolThe size is the sum of the sizes of all network elements in the pool.
[0107] In addition, for the performance data input in the aforementioned embodiment, there is a situation where data files are lost, and the missing data is supplemented by filling with 0 values.
[0108] The processed data are classified according to the signaling plane and the user plane, and 9-dimensional input data of the signaling plane and 8-dimensional input data of the user plane are obtained, as shown in Table 2:
[0109] Table 2
[0110]
[0111] The above data is input into the anomaly detection model for calculation. The data will pass through the following modules:
[0112] 1) Negative removal module: Since diff1day and diff7day represent the difference between the current alarm volume and the historical value, and a negative value means that the alarm volume is decreasing, indicating that the network situation is improving, it should not be set as an exception at this time. When the data passes through the negative removal module, the negative values of diff1day, diff7day, and pool_diff will be set to 0;
[0113] 2) Anomaly detection module: The PYOD tool library is used, and the selected algorithms include the following 12 specific anomaly detection algorithms, as shown in Table 3:
[0114] Table 3
[0115]
[0116] After the model calculation, a 12-dimensional variable will be output, and each dimension is an abnormal result determined by an anomaly detection algorithm;
[0117] 3) Voting module: This module will vote on the results output by the anomaly detection module. When more than two anomaly detection algorithms consider this data to be abnormal, the voting module will eventually judge this data as abnormal. Figure 3 shown.
[0118] Based on any of the above embodiments, step 103 in the method specifically includes:
[0119] Preprocessing the multidimensional data corresponding to the initial abnormality determination result to obtain preprocessing result data;
[0120] Input the preprocessing result data into the kernel density KDE model for density fitting to obtain a kernel density probability estimate;
[0121] The kernel density probability estimate is corrected to obtain the maximum value of the change degree in the data dimension, which is the anomaly positioning result.
[0122] The preprocessing of the multidimensional data corresponding to the initial abnormality determination result to obtain preprocessing result data specifically includes:
[0123] For the signaling plane performance index, the signaling plane preprocessing result data is obtained by subtracting the original index value from 1;
[0124] For the user plane performance index, the user plane preprocessing result data is obtained based on the original value of the index, the maximum value of the index and the minimum value of the index;
[0125] For alarm indicators, based on the original value of the indicator, the maximum value of the indicator and the minimum value of the indicator, Gaussian distribution transformation is performed to obtain the alarm preprocessing result data.
[0126] Specifically, the present invention performs anomaly detection on input data and outputs anomaly judgment. However, due to the voting mechanism, it is not certain at this time which data dimension is the main cause of the equipment anomaly, so it is also necessary to judge the main abnormal dimension, and the multi-dimensional data will be pre-processed for subsequent dimensional anomaly analysis.
[0127] 1) For the signaling plane performance indicators 'TAU success rate', 'attachment success rate', 'service request success rate', 'X2 handover success rate', 'paging success rate', and the user plane performance indicator 'HTTP session response success rate greater than 500KB', formula (5) is used for preprocessing calculation:
[0128] p′=1-p (5)
[0129] Where p is the original value, and p′ is the input value of the subsequent model after preprocessing;
[0130] 2) For the user plane performance indicators 'HTTP session average response delay greater than 500KB (ms)', 'HTTP download rate greater than 500KB (Kbps)', and 'HTTP500 downstream traffic (MB)', formula (6) is used for normalization:
[0131]
[0132] Where x is the original value, x min is the minimum value of this dimension within 7 days, x max is the maximum value within 7 days, and x′ is the input value of the subsequent model after preprocessing;
[0133] 3) For the alarm data 'size', 'diff1day', 'diff7day', and 'pool_diff', normalization is first performed:
[0134]
[0135] Where a is the original value, a min is the minimum value of this dimension within 7 days, a max is the maximum value within 7 days, and a′ is the normalized value;
[0136] Considering the business meaning of size, that is, the number of network element alarms satisfies the Poisson distribution, square root processing is performed to transform these indicators into Gaussian distribution, and the signs are retained:
[0137]
[0138] Wherein a′ is the normalized value obtained by formula 7, and a″ is the value after Gaussian transformation.
[0139] The above preprocessed data is input into the kernel density KDE model for density fitting. The input data time length is nearly 7 days, with a total of 7*12*24=2016 input data. Since the data has been transformed into a Gaussian distribution, the kernel density function of the Gaussian kernel is selected to continue the calculation:
[0140]
[0141] K(x) is the kernel density function, and the Gaussian kernel density function is used here. x is the input data, and h is the bandwidth, which is another parameter for model training. Formula (10) is used here to determine it:
[0142]
[0143] Where R(k), m2(k) satisfies:
[0144] R(k)=∫K(x) 2 dx (11)
[0145] m2(k)=∫x 2 K(x) 2 dx (12)
[0146] n is the number of input data, that is, 2016.
[0147] At this point, we can use formula (9) to perform kernel density estimation and calculate the change degree of the input data, that is, the probability in the kernel density Process such as Figure 4 shown.
[0148] Since kernel density estimation is pure data analysis, it is necessary to correct it in combination with solidification experience when outputting the results to increase the accuracy of the results.
[0149] 1) For the signaling plane performance indicators 'TAU success rate', 'attachment success rate', 'service request success rate', 'X2 handover success rate', 'paging success rate', and the user plane performance indicators 'HTTP session response success rate greater than 500KB' and 'HTTP download rate greater than 500KB (Kbps)', when the original values of these 7 indicators are greater than the historical 7-day average, the change degree is set to 0;
[0150] 2) For the user plane performance indicator 'HTTP session average response delay greater than 500KB (ms)', when the original value of the indicator is less than the historical 7-day average, the corresponding change degree is set to 0;
[0151] 3) For the user plane performance indicator 'HTTP500 downstream traffic (MB)', let f now is today's traffic, f yesterday is the traffic volume at the same time yesterday, then when When , the change degree is set to 0.
[0152] After correction, compare the change degrees of each data dimension. The dimension with the largest value is the main factor causing the anomaly, that is, the main potential fault point. The correction process is as follows: Figure 5 shown.
[0153] The core network hidden danger troubleshooting device provided by the present invention is described below. The core network hidden danger troubleshooting device described below and the core network hidden danger troubleshooting method described above can be referenced to each other.
[0154] Figure 6 Schematic diagram of the structure of the core network hidden danger detection device provided by the present invention. Figure 6 As shown, it includes: a pre-processing module 61, a processing module 62 and a correction module 63; wherein:
[0155] The preprocessing module 61 is used to access the real-time alarm stream, preprocess the real-time alarm stream, and obtain alarm feature data; the processing module 62 is used to obtain the data to be determined, input the data to be determined and the alarm feature data into the anomaly detection model, and obtain the initial anomaly determination result; the correction module 63 is used to correct the initial anomaly determination result based on kernel density estimation, and output the anomaly location result.
[0156] The present invention pre-processes the alarm flow, then performs network alarm vectorization, inputs the abnormal detection model for detection, and finally corrects it through kernel density estimation, so as to timely discover relatively hidden faults in the network and improve the judgment speed and accuracy.
[0157] Based on any of the above embodiments, the preprocessing module 61 includes: a first preprocessing submodule 611, a vectorization submodule 612 and a classification submodule 613, wherein:
[0158] The first preprocessing submodule 611 is used to access the Kafka real-time alarm stream interface, obtain the full alarm field of the preset standard core network, and perform denoising, standardization and time slicing processing on the full alarm field to obtain preprocessed alarm data; the vectorization submodule 612 is used to vectorize the preprocessed alarm data, extract the vectorized preprocessed alarm data according to the preset feature field, and obtain the alarm feature data having the same dimension as the preset feature field; the classification submodule 613 is used to classify the alarm vectorization data according to performance data, dialing data, engineering data and resource data to obtain the alarm feature data.
[0159] Based on any of the above embodiments, the first preprocessing submodule 611 is specifically used for:
[0160] Remove the noise alarms corresponding to the empty fields or abnormal fields in the full alarm field to obtain the alarm text; unify the different expressions in the same field in the alarm text and output the standardized alarm text; slice and sample the standardized alarm text according to the preset slice length to obtain the preprocessed alarm data.
[0161] Based on any of the above embodiments, the processing module 62 includes: a second preprocessing submodule 621 and a processing submodule 622, wherein:
[0162] The second preprocessing submodule 621 is used to obtain the data to be judged with preset dimensions, and classify it in combination with the alarm feature data to obtain model input data; the processing submodule 622 is used to input the model input data into the anomaly detection model, and obtain the initial anomaly judgment result through negative removal, anomaly detection and voting.
[0163] Based on any of the above embodiments, the second preprocessing submodule 621 is specifically used for:
[0164] Acquire performance data, dial test data, engineering data and resource data respectively to constitute the data to be determined; clean the alarm feature data to generate a new input field set, the input field set includes size, diff1day, diff7day and pool_diff; fill the missing data in the data to be determined with 0 to obtain the padded data to be determined; classify the input field set and the padded data to be determined according to the signaling plane and the user plane to obtain the model input data.
[0165] Based on any of the above embodiments, the correction module 63 includes: a third preprocessing submodule 631, an estimation submodule 632 and a correction submodule 633, wherein:
[0166] The third preprocessing submodule 631 is used to preprocess the multidimensional data corresponding to the initial abnormality judgment result to obtain preprocessing result data; the estimation submodule 632 is used to input the preprocessing result data into the kernel density KDE model for density fitting to obtain the kernel density probability estimation value; the correction submodule 633 is used to correct the kernel density probability estimation value to obtain the maximum value of the change degree in the data dimension, which is the abnormality positioning result.
[0167] Based on any of the above embodiments, the third preprocessing submodule 631 is specifically used for:
[0168] For signaling plane performance indicators, the signaling plane preprocessing result data is obtained based on 1 minus the original value of the indicator; for user plane performance indicators, the user plane preprocessing result data is obtained based on the original value of the indicator, the maximum value of the indicator and the minimum value of the indicator; for alarm indicators, the alarm preprocessing result data is obtained based on the original value of the indicator, the maximum value of the indicator and the minimum value of the indicator, and Gaussian distribution conversion is performed.
[0169] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the core network hidden danger troubleshooting method, which includes: accessing the real-time alarm stream, preprocessing the real-time alarm stream, and obtaining alarm feature data; obtaining the data to be determined, inputting the data to be determined and the alarm feature data into the anomaly detection model, and obtaining the initial anomaly determination result; correcting the initial anomaly determination result based on the kernel density estimation, and outputting the anomaly location result.
[0170] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0171] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the core network hidden danger troubleshooting method provided by the above methods, and the method includes: accessing a real-time alarm stream, preprocessing the real-time alarm stream, and obtaining alarm feature data; obtaining data to be determined, inputting the data to be determined and the alarm feature data into an anomaly detection model to obtain an initial anomaly determination result; correcting the initial anomaly determination result based on kernel density estimation, and outputting an anomaly location result.
[0172] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the above-mentioned core network hidden danger troubleshooting method, the method comprising: accessing a real-time alarm stream, preprocessing the real-time alarm stream, and obtaining alarm feature data; obtaining data to be determined, inputting the data to be determined and the alarm feature data into an anomaly detection model to obtain an initial anomaly determination result; correcting the initial anomaly determination result based on kernel density estimation, and outputting an anomaly location result.
[0173] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0174] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0175] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for troubleshooting core network risks, characterized in that: include: Accessing a real-time alarm stream, preprocessing the real-time alarm stream, and obtaining alarm characteristic data; Acquire the data to be determined, input the data to be determined and the alarm feature data into an anomaly detection model, and obtain an initial anomaly determination result; Correcting the initial abnormality determination result based on kernel density estimation and outputting an abnormality location result; The accessing of the real-time alarm stream and preprocessing of the real-time alarm stream to obtain alarm characteristic data specifically include: Access the Kafka real-time alarm stream interface, obtain the full alarm field of the preset standard core network, perform denoising, standardization and time slicing processing on the full alarm field to obtain pre-processed alarm data; the time slicing processing is a text slicing sampling processing according to the preset slicing duration; Vectorizing the preprocessed alarm data, extracting the vectorized preprocessed alarm data according to a preset characteristic field, and obtaining alarm vectorized data having the same dimension as the preset characteristic field; Classifying the alarm vectorization data according to performance data, dialing data, engineering data and resource data to obtain the alarm feature data; The step of obtaining the data to be determined and inputting the data to be determined and the alarm feature data into an anomaly detection model to obtain an initial anomaly determination result specifically includes: Acquire the data to be determined with preset dimensions, and classify them in combination with the alarm feature data to obtain model input data; Inputting the model input data into the anomaly detection model, and obtaining the initial anomaly determination result through negative elimination, anomaly detection and voting; The step of obtaining the to-be-determined data with preset dimensions and classifying the data in combination with the alarm feature data to obtain the model input data specifically includes: Acquire performance data, dial test data, engineering data and resource data respectively to form the data to be determined; The alarm feature data is cleaned to generate a new input field set, which includes size, diff1day, diff7day and pool_diff; wherein the size field represents the alarm quantity in the current 5-minute time slice, the diff1day field represents the difference between the alarm quantity in the current 5-minute time slice and the alarm quantity in the 5-minute time slice in the same time period yesterday, the diff7day field represents the difference between the alarm quantity in the current 5-minute time slice and the alarm quantity in the 5-minute time slice in the same time period last week, and the pool_diff field represents the difference between the size value of the network element and the average size value in the pool; Filling the missing data in the data to be determined with 0 to obtain the filled data to be determined; Classifying the input field set and the padded data to be determined according to the signaling plane and the user plane to obtain the model input data; The step of inputting the model input data into the anomaly detection model, performing negative elimination, anomaly detection and voting, and obtaining the initial anomaly determination result specifically includes: The model input data is input into the anomaly detection model for calculation, and the model input data is processed by the following modules: a negative removal module, used to set the negative values of diff1day, diff7day, and pool_diff to 0; an anomaly detection module, used to use multiple anomaly detection algorithms selected from the PYOD tool library to perform anomaly detection on the results output by the negative removal module to obtain a multidimensional variable, each dimension of the multidimensional variable is a result of whether it is abnormal determined by an anomaly detection algorithm; a voting module, used to vote on the results output by the anomaly detection module, and when more than two anomaly detection algorithms determine that the current model input data is abnormal, the current model input data is determined to be abnormal.
2. The method for troubleshooting core network risks according to claim 1, characterized in that: The access to the Kafka real-time alarm stream interface obtains the full alarm field of the preset standard core network, performs denoising, standardization and time slicing processing on the full alarm field to obtain pre-processed alarm data, specifically including: Removing the noise alarm corresponding to the empty field or abnormal field in the full alarm field to obtain the alarm text; Unifying different expressions in the same field in the alarm text and outputting a standardized alarm text; The standardized alarm text is sliced and sampled according to a preset slice duration to obtain the pre-processed alarm data.
3. The method for troubleshooting core network risks according to claim 1, characterized in that: The correcting the initial abnormality determination result based on kernel density estimation and outputting the abnormality positioning result specifically includes: Preprocessing the multidimensional data corresponding to the initial abnormality determination result to obtain preprocessing result data; Input the preprocessing result data into the kernel density KDE model for density fitting to obtain a kernel density probability estimate; The kernel density probability estimate is corrected to obtain the maximum value of the change degree in the data dimension, which is the anomaly positioning result.
4. The method for troubleshooting core network risks according to claim 3 is characterized in that: The preprocessing of the multidimensional data corresponding to the initial abnormality determination result to obtain preprocessing result data specifically includes: For the signaling plane performance index, the signaling plane preprocessing result data is obtained by subtracting the original index value from 1; For the user plane performance index, the user plane preprocessing result data is obtained based on the original value of the index, the maximum value of the index and the minimum value of the index; For alarm indicators, based on the original value of the indicator, the maximum value of the indicator and the minimum value of the indicator, Gaussian distribution transformation is performed to obtain the alarm preprocessing result data.
5. A core network hidden danger detection device, characterized in that: include: A preprocessing module, used to access the real-time alarm stream, preprocess the real-time alarm stream, and obtain alarm feature data; A processing module, used for obtaining data to be determined, inputting the data to be determined and the alarm feature data into an anomaly detection model, and obtaining an initial anomaly determination result; A correction module, used to correct the initial abnormality determination result based on kernel density estimation and output an abnormality positioning result; Wherein, the preprocessing module includes: The first preprocessing submodule is used to access the Kafka real-time alarm stream interface, obtain the full alarm field of the preset standard core network, perform denoising, standardization and time slicing processing on the full alarm field, and obtain preprocessed alarm data; the time slicing processing is a text slicing sampling processing according to the preset slicing duration; A vectorization submodule, used to vectorize the preprocessed alarm data, extract the vectorized preprocessed alarm data according to a preset feature field, and obtain alarm vectorized data having the same dimension as the preset feature field; A classification submodule, for classifying the alarm vectorization data according to performance data, dialing data, engineering data and resource data to obtain the alarm feature data; Wherein, the processing module includes: A second preprocessing submodule is used to obtain the to-be-determined data with a preset dimension, and classify it in combination with the alarm feature data to obtain model input data; A processing submodule, used for inputting the model input data into the anomaly detection model, and obtaining the initial anomaly determination result through negative elimination, anomaly detection and voting; Among them, the second preprocessing submodule is specifically used to: respectively obtain performance data, dial test data, engineering data and resource data to constitute the data to be determined; perform data cleaning on the alarm feature data to generate a new input field set, the input field set includes size, diff1day, diff7day and pool_diff; wherein the size field represents the alarm amount in the current 5-minute time slice, the diff1day field represents the difference between the alarm amount in the current 5-minute time slice and the alarm amount in the 5-minute time slice in the same time period yesterday, the diff7day field represents the difference between the alarm amount in the current 5-minute time slice and the alarm amount in the 5-minute time slice in the same time period last week, and the pool_diff field represents the difference between the size value of the network element and the average size value in the pool; fill the missing data in the data to be determined with 0 to obtain the filled data to be determined; classify the input field set and the filled data to be determined according to the signaling plane and the user plane to obtain the model input data; Among them, the processing submodule is specifically used to: input the model input data into the anomaly detection model for calculation, and process the model input data through the following modules: a negative removal module, used to set the negative values of diff1day, diff7day, and pool_diff to 0; an anomaly detection module, used to use multiple anomaly detection algorithms selected from the PYOD tool library to perform anomaly detection on the results output by the negative removal module to obtain a multidimensional variable, each dimension of the multidimensional variable is a result of whether it is abnormal determined by an anomaly detection algorithm; a voting module, used to vote on the results output by the anomaly detection module, and when more than two anomaly detection algorithms determine that the current model input data is abnormal, the current model input data is determined to be abnormal.
6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the core network hidden danger troubleshooting method as described in any one of claims 1 to 4 are implemented.
7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the core network hidden danger troubleshooting method as described in any one of claims 1 to 4 are implemented.
