Telecommunication program resource allocation method, system, device and medium based on online feature selection

Abstract

The present invention discloses a telecommunication program resource allocation method, system, device and medium based on online feature selection, including obtaining historical feature data associated with script program running resource consumption, obtaining different historical data sampling subsets; clustering on different historical data sampling subsets using a k-means clustering algorithm based on feature similarity; constructing multiple feature groups for multiple historical data clustering results using a method based on a mutual correlation matrix; for newly flowing stream features in online stream features, calculating their correlation with each of the feature groups, and adding them to the feature group with the largest correlation; eliminating redundant features in the feature group to form a final feature subset; according to the final feature subset, using a KNN classifier to determine the resource interval of each script program, and determining the operation of the script program according to resource allocation. The present invention makes full use of historical data and stream feature data to effectively solve the problem of online feature selection in telecommunication network data scenarios.

Description

Technical Field

[0001] The present invention relates to the field of software development and information technology, and in particular to a telecommunication program resource allocation method, system, equipment and medium based on online feature selection. Background Art

[0002] The data types, data source components, and specialties of telecommunications networks are diverse, resulting in data structures and business logics that vary greatly. Data types include performance data, business indicator data, configuration data, resource data, etc. Data sources include Kafka, files, databases, etc. Specialties include metropolitan area networks, core networks, access networks, etc. When performing unified data aggregation, there are generally two approaches: (i) For each data storage scenario, a customized parsing storage program is developed; this solution will result in many small programs, difficult maintenance, a long development cycle for scenario implementation, and difficulty in demand changes. However, since the program logic is completely mastered by the platform, stability is controllable. (ii) A task scheduling platform is built, and the demand side develops corresponding python scripts for different data scenarios. The script content is generally connecting to the data source -> obtaining data -> data cleaning -> data storage, running on the task scheduling platform for scheduled scheduling, and realizing continuous data aggregation; this solution has the following advantages: the demand side knows the data logic and configures the script by itself, without the need for a version release process, and the scenario landing block; the demand change page is convenient and efficient; the platform operation and maintenance pressure is small, and only the operation and maintenance scheduling platform is required, without paying attention to each demand.

[0003] However, due to the openness of the scheduling platform, the content of user scripts is uncertain, which leads to reduced controllability of the platform, resulting in the following problems: the more scripts the platform runs, the more resources are needed, and there are high-load time points. Resources include CPU and memory; user script content is uncontrollable, and there may be a large resource occupation, causing the platform to crash; each script has different resource requirements, too much allocation will waste resources, too little allocation will cause program execution failure, and it is impossible to preset uniformly.

[0004] At present, machine learning algorithms are widely used in resource allocation. Machine learning can predict load patterns and identify resource usage trends by analyzing historical data, and make resource allocation strategies accordingly, thereby improving resource utilization efficiency. Feature selection is the process of removing useless information from the data set to obtain the optimal features. It can effectively reduce the feature dimension, thereby improving the prediction performance of subsequent unknown samples. Therefore, it is one of the important steps in data preprocessing in machine learning. However, in the current learning of resource allocation, batch learning is used, feature selection is performed offline, and all data of training samples are given in advance. Therefore, the batch learning method has limitations when the entire feature space of the training data is unknown or the cost of obtaining the entire feature space is too high. Summary of the invention

[0005] Considering the application scenario of telecommunication network data storage, the present invention proposes a telecommunication program resource allocation method, system, device and medium based on online feature selection.

[0006] To achieve the above purpose, the technical solution provided by the present invention is:

[0007] A first aspect of the present application provides a telecommunication program resource allocation method based on online feature selection, comprising the following steps:

[0008] Obtain historical feature data associated with script program running resource consumption, and obtain different historical data sampling subsets;

[0009] Use the k-means clustering algorithm based on feature similarity to cluster different historical data sampling subsets to obtain multiple historical data clustering results;

[0010] Using the method based on the mutual correlation matrix, multiple feature groups are constructed from the clustering results of multiple historical data;

[0011] For the newly-flown flow features in the online flow features, the correlation between the newly-flown flow features and each of the feature groups is calculated, and the newly-flown flow features are added to the feature group with the greatest correlation;

[0012] Eliminate redundant features in the feature group to form the final feature subset;

[0013] According to the final feature subset, the KNN classifier is used to determine the resource interval of each script program, and the operation of the script program is determined according to the resource allocation.

[0014] To optimize the above technical solutions, the specific measures taken also include:

[0015] The method of obtaining historical feature data associated with resource consumption during script program execution includes: collecting script program historical data, filtering out dirty data therein, obtaining required script content, and extracting features associated with resource consumption from the script content; extracting features associated with resource consumption based on resource consumption data of historical execution of the script program; extracting features associated with resource consumption based on log data of historical execution of the script program; and also includes sampling the script program historical data to form multiple historical feature subsets with differences, thereby obtaining different sampling subsets.

[0016] Furthermore, for the newly-flowing flow features in the online flow features, the correlation between them and each of the feature groups is calculated, so that they are added to the feature group with the greatest correlation. This is achieved by calculating the correlation between the flow features and the representative features of the feature group. For discrete value data, the correlation is calculated using the mutual information method.

[0017] Furthermore, the data generated by the running of the script program is collected and stored as subsequent training data for continuous learning; a customized resource ratio is formed for each script program, and machine learning is continuously performed.

[0018] A second aspect of the present application provides a telecommunication program resource allocation system based on online feature selection, comprising:

[0019] The aggregation program management module is used to obtain historical feature data associated with the resource consumption of script program operation and obtain different historical data sampling subsets;

[0020] The feature selection module is used to cluster different historical data sampling subsets using the k-means clustering algorithm based on feature similarity to obtain multiple historical data clustering results; construct multiple feature groups based on the multiple historical data clustering results using a method based on the mutual correlation matrix; for the newly-inflow flow features in the online flow features, calculate their correlation with each of the feature groups and add them to the feature group with the greatest correlation; remove redundant features in the feature group to form a final feature subset;

[0021] The resource scheduling module is used to determine the resource interval of each script program based on the final feature subset and use the KNN classifier to determine the operation of the script program based on the resource allocation;

[0022] The telecommunication program resource allocation system based on online feature selection is used to execute the telecommunication program resource allocation method based on online feature selection as described above.

[0023] Furthermore, the aggregation program management module includes a code editing method for the aggregation script program and supports scripts in multiple languages, collects, parses, and writes network data from the data source to a unified data base, as well as the latest debugging records generated by the aggregation script program debugging and the training data collected in each debugging record.

[0024] Furthermore, the aggregation program management module includes a historical data collection module, which is used to collect all script program execution history records, including the occupied running resource time series data, the Prometheus time series library, the script history time consumption, the script content and the load peak and valley value data.

[0025] Furthermore, the feature selection module screens out data features that are closely related to the resource occupancy of the script program through group feature selection, performs group feature selection based on clustering integration on historical data, constructs feature groups, performs online feature selection on the newly created script program, updates the feature groups in real time, and obtains the final feature subset through group transformation.

[0026] A third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the telecommunication program resource allocation method based on online feature selection as described above is implemented.

[0027] A fourth aspect of the present application provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the telecommunication program resource allocation method based on online feature selection as described above.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] The present invention aims at the uncertainty of the feature space of the telecommunication network data aggregation program and the limitations of the batch learning method, takes into account the application scenarios and characteristics of telecommunication network data storage, takes user python scripts as training samples, and the uncertainty of their data content. Through a resource allocation scheme of the telecommunication network data aggregation program based on online feature selection, the training data of the algorithm includes both existing historical data, such as the historical time consumption of a single program run, resource consumption, script content, etc., and stream feature data, that is, the continuously newly created aggregation program, makes full use of historical data and stream feature data, and effectively solves the online feature selection problem in the telecommunication network data scenario.

[0030] The present invention also allocates different server resources, including CPU, memory, disk, etc., to different aggregation programs. On the one hand, a secondary review can be conducted based on programs with high resource requirements to avoid excessive resource usage leading to the collapse of the overall data aggregation base system and the spread of production impacts. On the other hand, personalized resource allocation based on the needs of each script can greatly avoid resource waste and improve server resource utilization, thereby reducing the operating cost of the data base. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 : A structural diagram of a telecommunication program resource allocation system based on online feature selection according to the present invention.

[0032] Figure 2 : A flow chart of a telecommunication program resource allocation method based on online feature selection according to the present invention.

[0033] Figure 3 : Schematic diagram of feature clustering integration process in online stream feature selection. DETAILED DESCRIPTION

[0034] The above contents of the present invention are further described in detail below in the form of embodiments, but this should not be understood as the scope of the above subject matter of the present invention being limited to the following embodiments, and all technologies realized based on the above contents of the present invention belong to the scope of the present invention.

[0035] Some of the terms in the present invention are explained as follows:

[0036] Bagging, the full name of which is Bootstrap Aggregating, is an integrated learning method used to improve the stability and accuracy of machine learning models. It reduces variance and improves the performance of the overall model by combining multiple independent prediction models, especially when dealing with models that are easily affected by noise in the data.

[0037] K-means is a classic clustering algorithm in machine learning, which is used to divide unlabeled data sets into K categories, where K is a user-specified number. The goal of K-means is to assign data points to these categories so that the sum of the distances between each data point and its assigned category centroid is minimized.

[0038] The present invention combines the characteristics of telecommunication data programs and provides a telecommunication program resource allocation method based on online feature selection, comprising the following steps:

[0039] Obtain historical feature data associated with script program running resource consumption, and obtain different historical data sampling subsets;

[0040] Use the k-means clustering algorithm based on feature similarity to cluster different historical data sampling subsets to obtain multiple historical data clustering results;

[0041] Using the method based on the mutual correlation matrix, multiple feature groups are constructed from the clustering results of multiple historical data;

[0042] For the newly-incoming flow features in the online flow features, calculate their correlation with each feature group and add them to the feature group with the greatest correlation;

[0043] Eliminate redundant features in the feature group to form the final feature subset;

[0044] According to the final feature subset, the KNN classifier is used to determine the resource interval of each script program, and the operation of the script program is determined according to the resource allocation.

[0045] Acquiring historical feature data associated with resource consumption during script program execution includes: collecting script program historical data, filtering out dirty data therein, obtaining required script content, and extracting features associated with resource consumption from the script content; extracting features associated with resource consumption based on resource consumption data of historical execution of script programs; extracting features associated with resource consumption based on log data of historical execution of script programs; and also including sampling script program historical data to form multiple different historical feature subsets, thereby obtaining different sampling subsets.

[0046] For the newly-flowing flow features in the online flow features, calculate their correlation with each feature group and add them to the feature group with the largest correlation. This is achieved by calculating the correlation between the flow features and the representative features of the feature group. For discrete value data, the correlation is calculated using the mutual information method.

[0047] The data generated by the running of the script program is collected and stored as subsequent training data for continuous learning; a customized resource ratio is formed for each script program, and machine learning is continuously performed.

[0048] The present invention also provides a telecommunication program resource allocation system based on online feature selection, such as Figure 1 As shown, including:

[0049] The aggregation program management module is used to obtain historical feature data associated with the resource consumption of script program operation and obtain different historical data sampling subsets;

[0050] The feature selection module is used to cluster different historical data sampling subsets using the k-means clustering algorithm based on feature similarity to obtain multiple historical data clustering results; construct multiple feature groups based on the multiple historical data clustering results using a method based on the mutual correlation matrix; for the newly-inflow flow features in the online flow features, calculate their correlation with each feature group and add them to the feature group with the largest correlation; remove redundant features in the feature group to form the final feature subset;

[0051] The resource scheduling module is used to determine the resource interval of each script program based on the final feature subset and use the KNN classifier to determine the operation of the script program based on the resource allocation;

[0052] The telecommunication program resource allocation system based on online feature selection is used for executing the telecommunication program resource allocation method based on online feature selection.

[0053] The aggregation program management module includes the code editing method of the aggregation script program and supports scripts in multiple languages. It collects, parses, and writes network data from the data source to a unified data base, as well as the latest debugging records generated by the aggregation script program debugging and the training data collected in each debugging record.

[0054] The aggregation program management module includes a historical data collection module, which is used to collect the execution history records of all script programs, including the time series data of the occupied running resources, the Prometheus time series library, the script history time consumption, the script content and the load peak and valley value data.

[0055] The feature selection module selects data features that are closely related to the resource usage of script programs through group feature selection, performs group feature selection based on clustering integration on historical data, constructs feature groups, performs online feature selection on newly created script programs, updates feature groups in real time, and obtains the final feature subset through group transformation.

[0056] In some implementations, the aggregation program management module collects and stores historical server resource consumption data, program content data, and program operation log data through the Prometheus time series library, PG relationship library, and ES library.

[0057] In some implementations, the feature selection module trains the base classifier using a bagging method, and a single clusterer uses a k-means method based on feature similarity to perform initial feature selection on the historical data collected by the aggregation program management module to form multiple feature groups.

[0058] In some embodiments, the feature selection module forms a final feature subset through an online process feature algorithm in machine learning, specifically through mutual information and Fisher's Z-test algorithm.

[0059] In some implementations, the resource scheduling module determines the interval to which the pre-allocated resources for each telecommunication data program belong through a KNN classifier, and issues an alarm according to a configured threshold.

[0060] In some implementations, an IPTV network element is used to run a log data collection program, and resources are allocated as follows:

[0061] Telecom O domain IPTV professional multiple data collection programs, such as IPTV EPG network metadata collection program has been running for a long time, and a large amount of historical data has been obtained and stored in the time series library, relationship library, and log library. Through the online feature selection method of machine learning, specifically through the feature clustering integration method and bagging method to train the base classifier, a single clusterer uses the k-means method based on feature similarity to realize the feature selection of historical data.

[0062] For the network data collection programs that are continuously newly connected, such as new network elements iptv iag, iaw, etc., the final feature subset is formed through the online process feature algorithm in machine learning, specifically the mutual information and Fisher's Z-test algorithm.

[0063] Finally, for the newly created IPTV professional data collection program of any network element, based on the final feature subset formed above, the KNN classifier algorithm is used to calculate the resource interval to be allocated in real time, so as to minimize resource allocation or early warning, improve resource utilization and system stability.

[0064] The technical solution of the present invention is further described in detail below in conjunction with specific embodiments:

[0065] A method for implementing telecommunication program resource allocation based on online feature selection, such as Figure 2 As shown, the following steps are included:

[0066] Step 1: Clean the collected script program historical data, including load timing data, script content, execution time, etc., and filter out dirty data, such as extremely short execution time due to abnormal script content and connection timeout due to network fluctuations.

[0067] Step 2.1: Extract features associated with resource consumption from the script content. The script content can be a program developed in Python, Groovy, or Java, and add it to the historical data set.

[0068] Step 2.2: Extract the features associated with resource consumption based on the historical execution data of the script program and add them to the candidate feature group; the Prometheus time series library contains the execution resource consumption, such as CPU, memory, bandwidth, disk, etc.

[0069] Step 2.3: Extract the features associated with resource consumption based on the log data of the historical execution of the script program and add them to the historical data set; the relationship database contains the task log data such as the execution start time, end time, execution result (normal / abnormal), and scheduling cycle of each script task;

[0070] Step 3: Sampling the historical data set multiple times to form multiple subsets of historical features with different characteristics;

[0071] Step 4: Use the same clustering algorithm to cluster on different sampling subsets formed in Step 3, so as to obtain multiple clustering results; train the base classifier through the bagging method to obtain different sample subsets, among which the single clusterer adopts the k-means method based on feature similarity;

[0072] Step 5: Use the method based on the mutual correlation matrix to construct multiple feature groups {G1, G2, …, Gs} for the multiple clustering results output by Step 4. So far, the historical data has been integrated into multiple feature groups through clustering. The following is the process of processing online stream features.

[0073] Step 6: For the newly-incoming flow feature f, calculate its correlation with each feature group {G1, G2, …, Gs} and add it to the feature group with the largest correlation; this is achieved by calculating the correlation between the flow feature f and the representative feature of the group. For discrete value data, the correlation calculation method uses mutual information;

[0074] Step 7: Redundant features are eliminated to form the final optimal feature subset; the group whose representative feature has a correlation with the category less than θ1 is deleted; the representative feature of each group is output as the optimal feature subset BF;

[0075] Step 8: Based on the BF obtained in Step 7, the KNN classifier is used to determine the resource range of each script program and compare it with the pre-assigned threshold θ2, such as 2 cores and 4G. If it exceeds the threshold, an alarm SMS is directly sent to the system administrator; if it is within the threshold, the user is allowed to start the script program;

[0076] Step 9: The user clicks Run to execute the script, or schedules the program through a timer; at the same time, the resource values ​​allocated in Step 8 are configured before execution;

[0077] Step 10: Collect and store the data generated by the program as subsequent training data for continuous learning;

[0078] At this point, each script program will have a customized resource allocation, and will continuously perform machine learning based on the new script program, effectively avoiding resource waste and platform instability.

[0079] The present invention also provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above telecommunication program resource allocation method based on online feature selection is implemented.

[0080] The present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the above method for allocating telecommunication program resources based on online feature selection.

[0081] At present, the resource allocation of the operating system of telecommunications data aggregation programs is either unlimited, resulting in resource waste and the risk of system crash due to excessive program resources. Or, there is no machine learning method, only fixed quotas are used, and there is no association with the program itself, resulting in execution failure when resources are insufficient and resource waste when resources are excessive.

[0082] The present invention can effectively solve the problem of not knowing how to allocate data programs, and meet the requirements of the telecommunications system for stability and resource utilization. Compared with the existing telecommunications data program resource allocation means, the solution of the present invention can predict the occupied resource value when the newly created telecommunications data aggregation program needs to run. If it exceeds the threshold, it can be discovered in advance to avoid excessive platform resource load, resulting in crashes, affecting the operation of all programs, and thus causing data interruption; there are different telecommunications data aggregation programs in different scenarios, even more than 1,000, and the present invention can allocate the minimum and stable server resources to each aggregation, thereby effectively reducing cloud resource costs;

[0083] The present invention can be applied to multiple professions in Jiangsu Telecom's O domain, such as IPTV, metropolitan area network, access network, and core network. After detailed testing and verification, the use effect is good. It can accurately allocate resources of multiple telecommunications data aggregation programs, ensure stable program operation, continuous and accurate telecommunications data, and low cloud resource consumption, avoiding IT system problems that lead to abnormal telecommunications operation data and customer complaints.

[0084] In the embodiments disclosed in the present application, the computer storage medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. The computer storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of computer storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0085] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any technician familiar with the profession, without departing from the scope of the technical solution of the present invention, according to the technical essence of the present invention, any simple modification, equivalent replacement and improvement made to the above embodiment still falls within the protection scope of the technical solution of the present invention.

Claims

1. A telecommunication program resource allocation method based on online feature selection, characterized in that: The following steps are involved: Obtain historical feature data associated with script program running resource consumption, and obtain different historical data sampling subsets; Use the k-means clustering algorithm based on feature similarity to cluster different historical data sampling subsets to obtain multiple historical data clustering results; Using the method based on the mutual correlation matrix, multiple feature groups are constructed from the clustering results of multiple historical data; For the newly-flown flow features in the online flow features, the correlation between the newly-flown flow features and each of the feature groups is calculated, and the newly-flown flow features are added to the feature group with the greatest correlation; Eliminate redundant features in the feature group to form the final feature subset; According to the final feature subset, the KNN classifier is used to determine the resource interval of each script program, and the operation of the script program is determined according to the resource allocation.

2. The method for allocating telecommunication program resources based on online feature selection according to claim 1, characterized in that: The method of obtaining historical feature data associated with resource consumption during script program execution includes: collecting script program historical data, filtering out dirty data therein, obtaining required script content, and extracting features associated with resource consumption from the script content; extracting features associated with resource consumption based on resource consumption data of historical execution of the script program; extracting features associated with resource consumption based on log data of historical execution of the script program; and also includes sampling the script program historical data to form multiple historical feature subsets with differences, thereby obtaining different sampling subsets.

3. The method for allocating telecommunication program resources based on online feature selection according to claim 1, characterized in that: For the newly-flowing flow features in the online flow features, the correlation between them and each of the feature groups is calculated, and they are added to the feature group with the largest correlation. This is achieved by calculating the correlation between the flow features and the representative features of the feature group. For discrete value data, the correlation is calculated using the mutual information method.

4. The method for allocating telecommunication program resources based on online feature selection according to claim 1, characterized in that: Collect and store the data generated by the script program as subsequent training data for continuous learning; Form a customized resource allocation for each script program and continuously perform machine learning.

5. A telecommunication program resource allocation system based on online feature selection, characterized in that: include: The aggregation program management module is used to obtain historical feature data associated with the resource consumption of script program operation and obtain different historical data sampling subsets; A feature selection module is used to cluster different historical data sampling subsets using a k-means clustering algorithm based on feature similarity to obtain multiple historical data clustering results; Using a method based on mutual correlation matrix, multiple feature groups are constructed from multiple historical data clustering results; for the newly-inflow flow features in the online flow features, the correlation between them and each of the feature groups is calculated, and they are added to the feature group with the greatest correlation; Culling Redundant features in the feature group form the final feature subset; The resource scheduling module is used to determine the resource interval of each script program based on the final feature subset and use the KNN classifier to determine the operation of the script program based on the resource allocation; The telecommunication program resource allocation system based on online feature selection is used to execute the telecommunication program resource allocation method based on online feature selection according to any one of claims 1 to 4.

6. The telecommunication program resource allocation system based on online feature selection according to claim 5, characterized in that: The aggregation program management module includes a code editing method for the aggregation script program and supports scripts in multiple languages. It collects, parses, and writes network data from data sources to a unified data base, as well as the latest debugging records generated by the aggregation script program debugging and the training data collected in each debugging record.

7. The telecommunication program resource allocation system based on online feature selection according to claim 5, characterized in that: The aggregation program management module includes a historical data collection module, which is used to collect all script program execution history records, including the occupied running resource time series data, the Prometheus time series library, the script history time consumption, the script content and the load peak and valley value data.

8. The telecommunication program resource allocation system based on online feature selection according to claim 5, characterized in that: The feature selection module screens out data features that are closely related to the resource occupancy of the script program through group feature selection, performs group feature selection based on clustering integration on historical data, constructs feature groups, performs online feature selection on the newly created script program, updates the feature groups in real time, and obtains the final feature subset through group transformation.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for allocating telecommunication program resources based on online feature selection as claimed in any one of claims 1 to 4 is implemented.

10. A computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the telecommunication program resource allocation method based on online feature selection according to any one of claims 1 to 4.

Images