A communication device data cleaning method, system and storage medium

Abstract

This invention discloses a method, system, and storage medium for cleaning data of communication equipment. Based on a two-layer data feature model of time segmentation and device type, this invention performs two-layer matching and filtering on the data, which has higher accuracy and reliability compared to traditional one-time cleaning.

Description

TECHNICAL FIELD

[0001] The application relates to a communication device data cleaning method and system and a storage medium, and belongs to the technical field of machine learning. BACKGROUND

[0002] With the development of Internet technology and the popularity of network applications, communication network IP (Internet Protocol) has become a development direction. In the 5G communication era, the key technologies of the 5G core network, SDN (Software Defined Network), NFV (Network Functions Virtualization) and MEC (Mobile Edge Computing), are all based on IP technology. Based on 5G technology, various new types of services can be supported to develop rapidly. Operators need to manage and monitor multi-vendor equipment and different types of equipment for public network and private line service network to ensure the stability of network services and the increasing requirements of users for application experience. At the same time, for enterprise networks based on 5G or other IP communication network technologies, users also need to monitor the operation of network equipment in real time to ensure the stability and efficient operation of enterprise networks.

[0003] In a communication network, to ensure real-time monitoring of network equipment, the collection of various data information of communication equipment is the basis, and the accuracy and reliability of the collected data must be ensured. If the collected data is based on errors, the stability and efficient operation of the network cannot be guaranteed. Therefore, the data needs to be cleaned before use. However, the current data cleaning algorithm mainly performs one-time cleaning based on a data feature model, and the cleaning accuracy and reliability are poor. SUMMARY

[0004] The application provides a communication device data cleaning method, system, storage medium and computing device, which solves the problems disclosed in the background art.

[0005] To solve the above technical problems, the technical scheme adopted by the application is:

[0006] A communication device data cleaning method, comprising:

[0007] obtaining a standard data feature model pre-trained and matched with a communication device type; wherein the standard data feature model comprises features of standard total data of the communication device and features of time period data in the standard total data;

[0008] According to the collected communication device data, the features of the collected total data and the features of the time period data in the collected total data are calculated; wherein the collected total data is all the collected communication device data;

[0009] if the feature of the collected total data does not match the feature of the standard total data, discarding the collected communication device data;

[0010] if the feature of the collected total data matches the feature of the standard total data, and the feature of the time period data in the collected total data does not match the feature of the corresponding time period data in the standard total data, discarding the collected communication device data of the unmatched time period.

[0011] The standard data feature model is:

[0012] P dt =(P dt0 ,P dt1 ,...,P dtn )

[0013] wherein P dt is the standard data feature model matched with the dt type communication device, P dt0 is the standard total data feature of the dt type communication device, and P dti is the feature of the i-th time period data of the dt type communication device, 1≤i≤n, and n is the total number of time periods set in a day.

[0014] The training of the standard data feature model comprises:

[0015] obtaining standard data of various types of communication devices;

[0016] using the standard data to calculate the feature of the standard total data and the feature of the time period data in the standard total data of various types of communication devices, and constructing the standard data feature model of various types of communication devices.

[0017] Further comprising, if the feature of the collected total data matches the feature of the standard total data, and the feature of all time period data in the collected total data matches the feature of the corresponding time period data in the standard total data, using the feature of the collected total data and the feature of the time period data in the collected total data to update the corresponding standard data feature model.

[0018] The feature of the standard total data, the feature of the time period data in the standard total data, the feature of the collected total data, and the feature of the time period data in the collected total data all include data mean, data range, and data standard deviation.

[0019] A communication device data cleaning system comprises:

[0020] a standard model acquisition module for acquiring a standard data feature model pre-trained and matched with the type of communication device; wherein the standard data feature model comprises the feature of the standard total data and the feature of the time period data in the standard total data of the communication device;

[0021] The real-time calculation module calculates the feature of the total collected data and the feature of the time period data in the total collected data according to the collected communication device data; wherein the total collected data is all the collected communication device data;

[0022] The first matching module discards the collected communication device data if the feature of the total collected data does not match the feature of the standard total data.

[0023] The second matching module discards the collected communication device data of the time period that does not match if the feature of the total collected data matches the feature of the standard total data and the feature of the time period data in the total collected data does not match the feature of the time period data in the corresponding standard total data.

[0024] In the standard model acquisition module, the standard data feature model is:

[0025] P dt =(P dt0 ,P dt1 ,...,P dtn )

[0026] Wherein P dt is the standard data feature model matched with the dt type communication device, P dt0 is the standard total data feature of the dt type communication device, and P dti is the feature of the i-th time period data of the dt type communication device, 1≤i≤n, and n is the total number of time periods set in a day.

[0027] The training module further includes a training module, and the process of training the standard data feature model includes:

[0028] The standard data of various types of communication devices is obtained.

[0029] The standard total data feature and the feature of the time period data in the standard total data of various types of communication devices are calculated by using the standard data, and the standard data feature model of various types of communication devices is constructed.

[0030] The updating module further includes an updating module; the updating module updates the corresponding standard data feature model by using the feature of the total collected data and the feature of the time period data in the total collected data if the feature of the total collected data matches the feature of the standard total data and the feature of all the time period data in the total collected data matches the feature of the time period data in the corresponding standard total data.

[0031] A computer readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by a computing device, cause the computing device to perform a communication device data cleaning method.

[0032] The application has the advantages that: the application is based on the double-layer data feature model of time segmentation and device type, and performs double-layer matching filtering on data, so that the accuracy and reliability are higher than those of traditional one-time cleaning. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 a flowchart of the cleaning method of the application;

[0034] Figure 2 a flowchart of the matching method of the application. DETAILED DESCRIPTION

[0035] The application will be further described below with reference to the drawings. The following examples are only used to more clearly illustrate the technical solutions of the application, and cannot be used to limit the protection scope of the application.

[0036] As shown in Figure 1 a communication device data cleaning method, comprising the following steps:

[0037] Step 1: obtaining a standard data feature model pre-trained and matched with the type of communication device; wherein the standard data feature model comprises the features of standard total data of the communication device and the features of time-segmented data in the standard total data.

[0038] Step 2: calculating the features of the collected total data and the features of time-segmented data in the collected total data according to the collected communication device data; wherein the collected total data is all the collected communication device data.

[0039] Step 3: if the features of the collected total data do not match the features of the standard total data, discarding the collected communication device data.

[0040] If the features of the collected total data match the features of the standard total data, and the features of time-segmented data in the collected total data do not match the features of time-segmented data in the corresponding standard total data, discarding the collected communication device data of the unmatched time-segmented data.

[0041] The above method is based on the double-layer data feature model of time segmentation and device type, and performs double-layer matching filtering on data, so that the accuracy and reliability are higher than those of traditional one-time cleaning; wherein the time segmentation refers to that the data to be filtered has obvious time-segmented features, that is, the data has obviously different features in different time periods; the device type refers to that the data features of different types of devices are obviously different.

[0042] The communication device data mainly includes device alarm data, device configuration data, device state data and device capability data. The device alarm data has time distribution characteristics, i.e. has a lower occurrence overview in a certain section and a higher occurrence overview in another time period. The device configuration data has relative stability and complies with the threshold range limit based on a template, and can be quickly matched and filtered through the data characteristics of a large number of typical configuration files. The device state data includes data flow information and port state information. Since the communication device is closely related to the daily behavior of people, the data characteristics of the state information of the communication device have strong time correlation. The device capability data is the hardware basic information of the communication device and has stability, and the data characteristics thereof are basically unchanged within a certain period of time.

[0043] Based on the above analysis, the standard data characteristic model of various communication devices can be preselected and trained, and a double-layer data characteristic model is constructed based on time and device type.

[0044] For each type of communication device, a standard data set containing time information, i.e. a training sample set, is prepared, and the time section is set by a person. The time section is divided by hours within a day, and the typical time section within a day is divided as follows: 0-7, 7-9, 9-12, 12-14, 14-18, 18-22, 22-24.

[0045] The basic data characteristics are P, P is a set, the set includes data mean X, data range S and data standard deviation R, i.e. P=(X, S, R), i.e. all characteristics include data mean, data range and data standard deviation.

[0046] The standard total data characteristics and the standard time section data characteristics are calculated for each type of device, and P dt0 and P dti are calculated; wherein P dt0 is the standard total data characteristics of the dt type communication device, the total data is the sum of the time section data, and P dti is the data characteristics of the i-th time section of the dt type communication device, 1≤i≤n, and n is the total number of time sections set within a day.

[0047] Therefore, after the training is completed, each type of standard data characteristic model can be represented as:

[0048] P dt =(P dt0 ,P dt1 ,...,P dtn )

[0049] Wherein, P dt is the standard data characteristic model matched with the dt type communication device.

[0050] Therefore, the above training process can be as follows:

[0051] 1) Obtain standard data of various types of communication equipment;

[0052] 2) Using the standard data, calculate the features of the standard total data of various types of communication equipment and the features of the time period data in the standard total data, and build a standard data feature model of various types of communication equipment.

[0053] Collect communication equipment data (support api read data or direct access to database), according to the collected communication equipment data, calculate the features of the collected total data and the features of the time period data in the collected total data; wherein the collected total data is all the collected communication equipment data.

[0054] As shown in Figure 2 , using the standard data feature model matched with the type of communication equipment, the features of the collected total data and the features of the time period data in the collected total data are filtered by double layer matching, that is, the features of the collected total data are matched with the features of the standard total data, and the feature matching can be realized by setting the error threshold range between them; if the features of the collected total data and the features of the standard total data do not match, the collected communication equipment data is discarded, and a log is recorded; if the features of the collected total data and the features of the standard total data match, further matching is performed, that is, the features of the time period data in the collected total data are matched with the features of the time period data in the corresponding standard total data.

[0055] If the features of the time period data in the collected total data and the features of the time period data in the corresponding standard total data do not match, the collected communication equipment data of the non-matching time period is discarded, and a log is recorded; if the features of the time period data in the collected total data and the features of the time period data in the corresponding standard total data match, the features of the collected total data and the features of the time period data in the collected total data can be used to update the corresponding standard data feature model, so as to perform subsequent data filtering.

[0056] The filtered data can be stored in the memory and provided directly through the standard API interface to the IP communication network for network equipment monitoring and management; it can be stored in the communication equipment data information database in real time for communication network management; it can also be pushed to the third party service through the API interface in real time.

[0057] In the above method, the model training is simple, the model structure is adjustable, the algorithm complexity is low, the double layer filtering takes into account both long-term data features and time period data features, and is suitable for data filtering and cleaning with obvious time period distribution characteristics, and the matching algorithm is simple, the training model and the matching method have strong scalability.

[0058] Based on the same technical scheme, the application further discloses a software system of the method, a communication equipment data cleaning system, which comprises:

[0059] a training module for training a standard data feature model;

[0060] The specific process comprises the following steps: obtaining standard data of various types of communication equipment; using the standard data to calculate features of standard total data of various types of communication equipment and features of time period data in the standard total data, and constructing a standard data feature model of various types of communication equipment.

[0061] a standard model acquisition module for acquiring a standard data feature model that is pre-trained and matched with the type of the communication equipment; wherein the standard data feature model comprises features of standard total data of the communication equipment and features of time period data in the standard total data.

[0062] In the standard model acquisition module, the standard data feature model is as follows:

[0063] P dt =(P dt0 ,P dt1 ,...,P dtn )

[0064] wherein P dt is a standard data feature model matched with the dt type of communication equipment, P dt0 is a feature of standard total data of the dt type of communication equipment, and P dti is a feature of the i-th time period data of the dt type of communication equipment, 1≤i≤n, and n is the total number of set time periods in a day.

[0065] a real-time calculation module for calculating features of collected total data and features of time period data in the collected total data according to collected communication equipment data; wherein the collected total data is all collected communication equipment data.

[0066] a first matching module for discarding the collected communication equipment data if the features of the collected total data do not match the features of the standard total data.

[0067] a second matching module for discarding the collected communication equipment data of the time period that does not match if the features of the collected total data match the features of the standard total data and the features of time period data in the collected total data do not match the features of corresponding time period data in the standard total data.

[0068] an updating module for updating the corresponding standard data feature model using the features of the collected total data and the features of time period data in the collected total data if the features of the collected total data match the features of the standard total data and the features of all time period data in the collected total data match the features of corresponding time period data in the standard total data.

[0069] Based on the same technical solution, the application further discloses a computer readable storage medium storing one or more programs, the one or more programs including instructions which, when executed by a computing device, cause the computing device to perform the communication device data cleaning method.

[0070] Based on the same technical solution, the application further discloses a computing device including one or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and configured to be executed by the one or more processors, and the one or more programs include instructions for performing the communication device data cleaning method.

[0071] Those skilled in the art will understand that embodiments of the application can be provided as methods, systems, or computer program products. Therefore, the application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer usable program code.

[0072] The application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the flow Figure 1 The function specified in one or more flows and / or blocks Figure 1 The function specified in one or more flows and / or blocks

[0073] These computer program instructions can also be stored in a computer readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including instruction devices that implement the flow Figure 1 The function specified in one or more flows and / or blocks Figure 1 The function specified in one or more flows and / or blocks

[0074] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are generated to realize the computer-implemented processes in the computer or other programmable devices, and the instructions executed in the computer or other programmable devices provide operational steps for implementing the functions specified in the flowchart Figure 1 one flow or a plurality of flows and / or the functions specified in the block Figure 1 one flow or a plurality of flows and / or the functions specified in the block

[0075] The above merely illustrates the embodiments of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall fall within the scope of the claims of the present application.

Claims

1. A method for cleaning communication device data, the method comprising: The method comprises the following steps: obtaining a standard data feature model pre-trained and matched with the type of the communication device; wherein the standard data feature model comprises features of standard total data of the communication device and features of time-period data in the standard total data; calculating features of the collected total data and features of time-period data in the collected total data according to the collected communication device data; wherein the collected total data is all the collected communication device data; if the features of the collected total data do not match the features of the standard total data, discarding the collected communication device data; if the features of the collected total data match the features of the standard total data and the features of time-period data in the collected total data do not match the features of time-period data in the corresponding standard total data, discarding the collected communication device data of the unmatched time period.

2. The method of claim 1, wherein, The standard data feature model comprises: P dt = (P dt0 ,P dt1 ,...,P dtn ) P dt Standard data feature model matched with dt type communication device, P dt0 Standard total data feature of dt type communication device, P dti Data feature of dt type communication device in ith period, 1≤i≤n, n is total number of set periods in a day.

3. The method of claim 1, wherein, The training of the standard data feature model comprises: obtaining standard data of various types of communication devices; using the standard data to calculate features of standard total data of various types of communication devices and features of time-period data in the standard total data, and constructing standard data feature models of various types of communication devices.

4. The method of claim 1, wherein, Further, if the features of the collected total data match the features of the standard total data and the features of all time-period data in the collected total data match the features of time-period data in the corresponding standard total data, using the features of the collected total data and the features of time-period data in the collected total data to update the corresponding standard data feature model.

5. The method of claim 1, wherein, The features of the standard total data, the features of time-period data in the standard total data, the features of the collected total data and the features of time-period data in the collected total data all comprise data mean, data range and data standard deviation.

6. A communication device data cleansing system characterized by, The method comprises the following steps: a standard model obtaining module for obtaining a standard data feature model pre-trained and matched with the type of the communication device; wherein the standard data feature model comprises features of standard total data of the communication device and features of time-period data in the standard total data; a real-time calculation module for calculating features of the collected total data and features of time-period data in the collected total data according to the collected communication device data; wherein the collected total data is all the collected communication device data; a first matching module for discarding the collected communication device data if the features of the collected total data do not match the features of the standard total data; a second matching module for discarding the collected communication device data of the unmatched time period if the features of the collected total data match the features of the standard total data and the features of time-period data in the collected total data do not match the features of time-period data in the corresponding standard total data.

7. The communication device data cleansing system of claim 6, wherein, In the standard model obtaining module, the standard data feature model comprises: P dt = (P dt0 ,P dt1 ,...,P dtn ) P dt Standard data feature model matched with dt type communication device, P dt0 Standard total data feature of dt type communication device, P dti Data feature of dt type communication device in ith period, 1≤i≤n, n is total number of set periods in a day.

8. The communication device data cleansing system of claim 6, wherein, The training module further comprises the following steps: obtaining standard data of various types of communication devices; using the standard data to calculate features of standard total data of various types of communication devices and features of time-period data in the standard total data, and constructing standard data feature models of various types of communication devices.

9. The communication device data cleansing system of claim 6, wherein, The application also comprises an updating module; if the features of the collected total data match the features of the standard total data, and the features of all the time period data in the collected total data match the features of the corresponding time period data in the standard total data, the features of the collected total data and the features of the time period data in the collected total data are used to update the corresponding standard data feature model.

10. A computer-readable storage medium storing one or more programs, the one or more programs comprising instructions for: The one or more programs include instructions that when executed by a computing device, cause the computing device to perform any of the methods of claims 1-5.

Images