Data processing method, electronic equipment and storage medium

Abstract

The invention provides a data processing method, electronic equipment and a storage medium, and relates to the technical field of data processing. The method comprises the following steps: in response to a data acquisition signal triggered by a sample task in a content distribution network, acquiring multiple pieces of sample index data corresponding to the sample task; performing feature analysis on each piece of sample index data according to a plurality of alternative processing algorithms to obtain a plurality of pieces of sample feature data corresponding to each piece of sample index data; according to the preset feature data and the multiple pieces of sample feature data corresponding to each piece of sample index data, screening out part of alternative processing algorithms from the multiple alternative processing algorithms as target processing algorithms, and obtaining a target processing algorithm set; and determining a target processing program according to the target processing algorithm in the target processing algorithm set. By using the target processing program, the processing efficiency of the to-be-processed index data can be improved, and the accuracy of feature extraction of the to-be-processed index data is improved.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular, to a data processing method, an electronic device, and a storage medium. Background Art

[0002] With the development of network technology, in order to enable users to obtain better communication services, it is usually necessary to optimize network performance. For example, based on different detection criteria, various different types of content data are detected to confirm whether the content data is secure data.

[0003] However, when analyzing content data with a large amount of data (for example, content data in a content distribution network with a wide coverage), operation and maintenance personnel need to manually classify and detect thousands of metric data to determine the characteristic information corresponding to each metric data to be detected, which greatly reduces the data processing efficiency and cannot guarantee the detection accuracy of the data. Summary of the Invention

[0004] This application provides a data processing method, an electronic device, and a storage medium.

[0005] An embodiment of this application provides a data processing method, which includes: in response to a data acquisition signal triggered by a sample task in a content distribution network, acquiring a plurality of sample metric data corresponding to the sample task; respectively performing feature analysis on each sample metric data according to a plurality of alternative processing algorithms to obtain a plurality of sample feature data corresponding to each sample metric data; screening out some of the alternative processing algorithms from the plurality of alternative processing algorithms as target processing algorithms according to preset feature data and the plurality of sample feature data corresponding to each sample metric data to obtain a target processing algorithm set; determining a target processing program according to the target processing algorithms in the target processing algorithm set, where the target processing program is used to extract features from the metric data to be processed corresponding to the task to be processed.

[0006] An embodiment of this application provides an electronic device, including: one or more processors; a memory storing one or more programs, when the one or more programs are executed by the one or more processors, enabling the one or more processors to implement any one of the data processing methods in the embodiments of this application.

[0007] An embodiment of this application provides a readable storage medium storing a computer program, where the computer program, when executed by a processor, implements any one of the data processing methods in the embodiments of this application.

[0008] According to the data processing method, electronic device, and storage medium of the embodiments of the present application, by using multiple alternative processing algorithms to respectively perform feature analysis on each sample metric data in the sample tasks in the content distribution network, multiple sample feature data corresponding to each sample metric data are obtained, and the feature information of the sample tasks in the content distribution network can be clarified; based on the preset feature data and the multiple sample feature data corresponding to each sample metric data, some alternative processing algorithms are selected from the multiple alternative processing algorithms as target processing algorithms to obtain a set of target processing algorithms, and a target processing program is determined according to the target processing algorithms in the set of target processing algorithms, so as to automatically use the target processing program to perform feature extraction on the to-be-processed metric data corresponding to the to-be-processed tasks. Compared with the existing method of manual feature extraction, the processing efficiency of the to-be-processed metric data can be improved, and the accuracy of feature extraction of the to-be-processed metric data can be enhanced.

[0009] More descriptions about the above embodiments and other aspects of the present application and their implementation manners are provided in the accompanying drawings, the specific implementation manners, and the claims. Description of the Drawings

[0010] Figure 1 The flowchart showing a data processing method provided by an embodiment of the present application.

[0011] Figure 2 The block diagram showing the composition of a data processing device provided by an embodiment of the present application.

[0012] Figure 3 The block diagram showing the composition of a data processing system provided by an embodiment of the present application.

[0013] Figure 4 The flowchart showing the working method of a data processing system provided by an embodiment of the present application.

[0014] Figure 5 The block diagram showing the composition of an electronic device provided by an embodiment of the present application. Detailed Description of the Embodiments

[0015] To enable those skilled in the art to better understand the technical solutions of the present application, the following makes descriptions of the exemplary embodiments of the present application with reference to the accompanying drawings. Various details of the embodiments of the present application are included to assist in understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the descriptions of well-known functions and structures are omitted below.

[0016] In the case of no conflict, the embodiments of the present application and the features in the embodiments may be combined with each other.

[0017] As used herein, the term "and / or" includes any and all combinations of one or more of the related listed items. The terms used herein are only for describing specific embodiments and are not intended to limit the present application. As used herein, the singular forms "a" and "the" are also intended to include the plural forms unless the context clearly indicates otherwise. It will also be understood that when the terms "comprises" and / or "consists of" are used in this specification, the specified features, wholes, steps, operations, elements and / or components are present, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their groups. "Connected" or "coupled" and other similar terms are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0018] Unless otherwise defined, the meanings of all terms (including technical and scientific terms) used herein are the same as those commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in common dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present application, and will not be interpreted as having an idealized or overly formal meaning unless clearly defined herein.

[0019] The number of metrics that need to be monitored for operation and maintenance is extremely large. At the same time, the content delivery network itself is a multi-level architecture. Taking the content delivery network as an example, it covers 31 provinces across the country and has a huge number of server clusters. In daily CDN network monitoring, there are thousands of common metrics. It is undoubtedly an impossible task to rely on manpower to perform feature engineering on them and then perform feature screening.

[0020] In the process of operating and maintaining the content delivery network, it is necessary to adjust for a variety of different metric data. However, since there are a huge number of server clusters in the content delivery network, and each server cluster needs to process thousands of metric data to determine whether the content data transmitted in the content delivery network is secure data.

[0021] Generally, when performing data analysis on content data with a large amount of data (for example, content data in a content delivery network with a wide coverage), operation and maintenance personnel need to manually classify and detect thousands of metric data to determine the feature information corresponding to each metric data to be detected, which greatly reduces the data processing efficiency and cannot guarantee the detection accuracy of the data.

[0022] The present application provides a data processing method, an electronic device and a storage medium to solve the above problems.

[0023] Figure 1 The flowchart shows a data processing method provided by an embodiment of the present application. This method can be applied to a data processing device. For example, the data processing device can be a centralized control device of a content delivery network. As Figure 1 shown, the data processing method in the embodiment of the present application includes but is not limited to the following steps.

[0024] Step S101: In response to a data acquisition signal triggered by a sample task in the content delivery network, obtain multiple sample metric data corresponding to the sample task.

[0025] Among them, the sample metric data is the metric data corresponding to the application data in the content delivery network. For example, the resolution of video data, etc.

[0026] Step S102: Perform feature analysis on each sample metric data according to multiple alternative processing algorithms respectively, and obtain multiple sample feature data corresponding to each sample metric data.

[0027] Step S103: According to the preset feature data and the multiple sample feature data corresponding to each sample metric data, screen out some alternative processing algorithms from the multiple alternative processing algorithms as target processing algorithms, and obtain a target processing algorithm set.

[0028] Step S104: Determine a target processing program according to the target processing algorithms in the target processing algorithm set.

[0029] Among them, the target processing program is used to extract features from the to-be-processed metric data corresponding to the to-be-processed task.

[0030] In this embodiment, by using multiple alternative processing algorithms to perform feature analysis on each sample metric data in the sample task in the content delivery network respectively, and obtaining multiple sample feature data corresponding to each sample metric data, the feature information of the sample task in the content delivery network can be clarified; according to the preset feature data and the multiple sample feature data corresponding to each sample metric data, screen out some alternative processing algorithms from the multiple alternative processing algorithms as target processing algorithms, obtain a target processing algorithm set, and determine a target processing program according to the target processing algorithms in the target processing algorithm set, so as to automatically use the target processing program to extract features from the to-be-processed metric data corresponding to the to-be-processed task. Compared with the existing method of manual feature extraction, the processing efficiency of the to-be-processed metric data can be improved, and the accuracy of feature extraction of the to-be-processed metric data can be enhanced.

[0031] In some exemplary embodiments, to screen out some alternative processing algorithms from multiple alternative processing algorithms as target processing algorithms based on preset feature data and multiple sample feature data corresponding to each sample index data, and obtain a set of target processing algorithms, the following method can be adopted: respectively match the multiple sample feature data corresponding to each sample index data with the preset feature data to determine the matching degree between the multiple sample feature data corresponding to each sample index data and the preset feature data; based on the matching degree, screen out some alternative processing algorithms from the multiple alternative processing algorithms as target processing algorithms to obtain a set of target processing algorithms.

[0032] Among them, the preset feature data is the feature data stored in the log file; for example, the fault information and / or alarm information that occurred during a certain period of device operation recorded in the log file, etc. According to the frequency of occurrence of the fault information and / or alarm information, match the multiple sample feature data to determine whether the sample feature data is the same as the fault information and / or alarm information recorded in the log file, so as to determine the matching degree between the multiple sample feature data corresponding to each sample index data and the preset feature data.

[0033] Furthermore, based on the matching degree between different sample feature data and the preset feature data, screen the multiple alternative processing algorithms, and use the alternative processing algorithms that meet the screening conditions (such as, the matching degree is greater than the preset degree threshold, etc.) as target processing algorithms, so that the sample index data can be processed more accurately using the target processing algorithm.

[0034] In some exemplary embodiments, the preset feature data includes: preset fault data and the first preset frequency data of the appearance of the preset fault data in the log file (for example, the first preset frequency data represents the number of times the preset fault data appears in the log file within a preset duration, etc.); and / or, preset alarm data and the second preset frequency data of the appearance of the preset alarm data in the log file (for example, the second preset frequency data represents the number of times the preset alarm data appears in the log file within a preset duration, etc.). The sample feature data includes: sample fault data and its corresponding first sample frequency data, and / or, sample alarm data and its corresponding second sample frequency data.

[0035] Among them, when the sample fault data is the same as the preset fault data, the closer the first sample frequency data is to the first preset frequency data, the greater the matching coefficient; and / or, when the sample alarm data is the same as the preset alarm data, the closer the second sample frequency data is to the second preset frequency data, the greater the matching coefficient.

[0036] It should be noted that the greater the matching coefficient, the more matching the sample feature data is with the preset feature data.

[0037] By comparing the sample fault data with the preset fault data in the log file, and comparing the first sample frequency data of the sample fault data in the sample index data with the first preset frequency data of the preset fault data in the log file, the matching degree between the sample fault data and the preset fault data can be determined; similarly, by comparing the sample alarm data with the preset alarm data in the log file, and comparing the second sample frequency data of the sample alarm data in the sample index data with the second preset frequency data of the preset alarm data in the log file, the matching degree between the sample alarm data and the preset alarm data can be determined. So as to screen out some alternative processing algorithms from multiple alternative processing algorithms as the target processing algorithms according to the determined matching degree above, and obtain a target processing algorithm set, so that the multiple target processing algorithms in the target processing algorithm set can process the sample index data more quickly and improve the processing efficiency of the sample index data.

[0038] In some exemplary embodiments, matching each of the multiple sample feature data corresponding to each sample index data with the preset feature data to determine the matching degree between the multiple sample feature data corresponding to each sample index data and the preset feature data includes: matching each sample feature data with the preset feature data respectively to obtain a plurality of matching coefficients; screening the plurality of matching coefficients to obtain a target matching coefficient set.

[0039] Wherein, the preset feature data is data representing the feature information of the sample task; the target matching coefficient set includes at least one target matching coefficient, and the target matching coefficient is used to represent the matching degree between the sample feature data and the preset feature data. The larger the target matching coefficient is, the higher the matching degree between the sample feature data and the preset feature data is.

[0040] It should be noted that different sample feature data correspond to different matching coefficients, and the following matching coefficient analysis table [sample feature data 1, matching coefficient 1; sample feature data 2, matching coefficient 2;...; sample feature data m, matching coefficient m] can be obtained, where m represents the number of sample feature data, and m is an integer greater than or equal to 1.

[0041] Then, sort the respective matching coefficients in the above matching coefficient analysis table (for example, in ascending order or descending order) so as to more quickly obtain multiple target matching coefficients with the highest matching coefficient (or matching coefficients greater than the preset coefficient threshold) and the sample feature data corresponding to each target matching coefficient.

[0042] The multiple target matching coefficients obtained through matching can characterize the matching degree between the sample feature data and the preset feature data, and can more effectively screen out some alternative processing algorithms from multiple alternative processing algorithms as the target processing algorithms according to the obtained matching degree, so that the target processing algorithms are more suitable for processing the sample index data and improve the processing efficiency of the sample index data.

[0043] In some exemplary embodiments, the sample task includes a load balancing task in a content delivery network, the sample index data includes the central processing unit (CPU) usage rate, the network bandwidth occupancy rate, and the network element load quantity, and the preset feature data includes a preset CPU usage rate threshold, a preset network bandwidth occupancy rate threshold, and a preset network element load quantity threshold.

[0044] Matching each sample feature data with the preset feature data respectively to obtain multiple matching coefficients, including: matching the usage rate feature data corresponding to the CPU usage rate with the preset CPU usage rate threshold to obtain a first matching coefficient; matching the bandwidth feature data corresponding to the network bandwidth occupancy rate with the preset network bandwidth occupancy rate threshold to obtain a second matching coefficient; matching the network element load feature data corresponding to the network element load quantity with the preset network element load quantity threshold to obtain a third matching coefficient.

[0045] Among them, the first matching coefficient can characterize the matching degree between the sample feature data and the preset feature data when the sample index data is the CPU usage rate; the second matching coefficient can characterize the matching degree between the sample feature data and the preset feature data when the sample index data is the network bandwidth occupancy rate; the third matching coefficient can characterize the matching degree between the sample feature data and the preset feature data when the sample index data is the network element load quantity.

[0046] By comparing and matching the sample feature data corresponding to different sample index data with the preset feature data during the process of processing the load balancing task, the matching degree of the sample feature data corresponding to each sample index data in the load balancing task can be clarified, so as to more quickly select appropriate sample feature data to measure the processing process of the load balancing task and improve the processing efficiency of the load balancing task.

[0047] In some exemplary embodiments, the sample task further includes at least one of the following: a data distribution task, a task of translating the first type of data, a task of deleting the second type of data, a real-time data pulling task, a network performance reporting task; the sample index data further includes at least one of the following: the memory occupancy rate, the disk occupancy rate, the disk input speed, and the disk output speed.

[0048] Among them, the first type of data is the data whose frequency of use by the user is higher than the preset frequency threshold, and the second type of data is the data whose frequency of use by the user is lower than the preset frequency threshold. For example, the task of shifting the first type of data is to shift the first type of data with a high frequency of use by the user to the target storage unit, so as to facilitate the user to extract the first type of data more quickly. Another example is that the task of deleting the second type of data is to delete the second type of data with a very low frequency of use by the user, thereby increasing the storage space of the system and facilitating the storage of data for other tasks.

[0049] By triggering the corresponding data acquisition signals through the above-mentioned various different sample tasks, multiple sample metric data corresponding to different sample tasks can be obtained, thereby making the sample metric data richer. For example, the running conditions of the system hardware during the execution of the data distribution task are characterized by sample metric data such as memory occupancy rate, disk occupancy rate, disk input speed, and disk output speed, etc., which is convenient for subsequent system optimization of the data distribution task.

[0050] In some exemplary embodiments, after determining the target processing program according to the target processing algorithm in the target processing algorithm set in step S104, the method further includes: obtaining the to-be-processed metric data corresponding to the to-be-processed task in response to the data acquisition signal triggered by the to-be-processed task; inputting the to-be-processed metric data into the target processing program for feature extraction to obtain the to-be-processed feature data corresponding to the to-be-processed metric data.

[0051] Among them, the to-be-processed feature data is used to optimize communication network parameters. For example, the to-be-processed feature data is used to optimize the signal quality parameters in the communication network, so as to improve the communication quality in the communication network and optimize the performance of the communication network.

[0052] By using the target processing program to perform feature extraction on the to-be-processed metric data corresponding to the to-be-processed task, compared with the existing method of manual feature extraction, the processing efficiency of the to-be-processed metric data can be improved, and the accuracy of feature extraction of the to-be-processed metric data can be enhanced.

[0053] Figure 2 The block diagram showing the composition of a data processing device provided by an embodiment of the present application is as Figure 2 shown. As

[0054] An acquisition module 201, configured to acquire multiple sample metric data corresponding to a sample task in response to a data acquisition signal triggered by a sample task in a content distribution network.

[0055] The analysis module 202 is configured to perform feature analysis on each sample metric data according to multiple alternative processing algorithms respectively, and obtain multiple sample feature data corresponding to each sample metric data.

[0056] The screening module 203 is configured to screen out some alternative processing algorithms from the multiple alternative processing algorithms as target processing algorithms according to the preset feature data and the multiple sample feature data corresponding to each sample metric data, and obtain a set of target processing algorithms.

[0057] The determination module 204 is configured to determine a target processing program according to the target processing algorithms in the set of target processing algorithms.

[0058] Wherein, the target processing program is used to perform feature extraction on the to-be-processed metric data corresponding to the to-be-processed task.

[0059] It should be noted that the data processing device 200 in this embodiment can implement any data processing method in the embodiments of the present application.

[0060] According to the data processing device of the embodiment of the present application, by using multiple alternative processing algorithms to perform feature analysis on each sample metric data in the sample tasks in the content delivery network respectively, and obtaining multiple sample feature data corresponding to each sample metric data, the feature information of the sample tasks in the content delivery network can be clarified; according to the preset feature data and the multiple sample feature data corresponding to each sample metric data, some alternative processing algorithms are screened out from the multiple alternative processing algorithms as target processing algorithms, and a set of target processing algorithms is obtained, and a target processing program is determined according to the target processing algorithms in the set of target processing algorithms, so as to automatically use the target processing program to perform feature extraction on the to-be-processed metric data corresponding to the to-be-processed task. Compared with the existing method of manual feature extraction, the processing efficiency of the to-be-processed metric data can be improved, and the accuracy of feature extraction of the to-be-processed metric data can be enhanced.

[0061] Figure 3 Show a block diagram of the composition of a data processing system provided by an embodiment of the present application. As Figure 3 shown, the data processing system includes but is not limited to the following levels: a hardware layer 310, a data layer 320, an application layer 330, and a main control layer 340.

[0062] Wherein, the hardware layer 310 includes existing network element devices 311 and data probes 312 in a content delivery network (Content Delivery Network, CDN). The data probes 312 are respectively connected to the network element devices 311, a data acquisition interface 321, and a data distribution interface 322.

[0063] The data layer 320 includes a data acquisition interface 321, a data delivery interface 322, a data task management module 323 and a log file system 324. The data acquisition interface 321 is connected to the data probe 312 and the data task management module 323 respectively. The data delivery interface 322 is connected to the data probe 312 and the data task management module 323 respectively. The data task management module 323 is connected to the data acquisition interface 321, the data delivery interface 322 and the feature generation module 331 respectively.

[0064] The application layer 330 includes a feature generation module 331, a machine learning scoring module 332, and an algorithm solidification module 333. The feature generation module 331 is connected to the main control module 341 and the data task module 323. The machine learning scoring module 332 is connected to the feature generation module 331, the algorithm solidification module 333, and the log file system 324, respectively.

[0065] The main control layer 340 includes a main control module 341. The main control module 341 is used to control multiple sample tasks and / or multiple tasks to be processed executed in the data processing system, and configure a task list and multiple candidate processing algorithms included in the task list.

[0066] The network element device 311 includes a server and / or a virtual grouping device, etc. The network element device 311 is used to provide sample indicator data or indicator data to be processed to the data probe 312, for example, video data and audio data transmitted in the CDN.

[0067] The data probe 312 is used to collect sample indicator data or unprocessed indicator data generated by the network element device 311. When the data probe 312 receives the data acquisition signal sent by the data sending interface 321, the data probe 312 will upload the sample indicator data or unprocessed indicator data generated by the network element device 311 to the data acquisition interface 321.

[0068] The data acquisition interface 321 is used to send the sample indicator data or the indicator data to be processed uploaded by the data probe 312 to the data task management module 323, so that the data task management module 323 can mark multiple sample indicator data or multiple indicator data to be processed. For example, based on the order of the acquired data, each indicator data is marked with timestamp information for preparation for the feature generation module 311 to call.

[0069] The data sending interface 322 is used to drive the data probe 312 to collect data in response to the data acquisition signal corresponding to the sample task (or the data acquisition signal triggered by the task to be processed) sent by the data task management module 323, so as to obtain multiple sample indicator data or multiple indicator data to be processed generated by the network element device 311.

[0070] The data task management module 323 is used to maintain each task (such as a data collection task) in the data task queue and send each task to the corresponding execution module for execution. Among them, for a newly added data collection task, the data task management module 323 registers the newly added data collection task into the data task queue; for a data collection task that needs to be deleted, the data task management module 323 unregisters and deletes the data collection task to be deleted from the data task queue; for a data collection task that needs to be modified, the data task management module 323 modifies the content of the data collection task to be modified; the data task management module 323 is also used to mark each data collection task with timestamp information based on the order of the obtained data collection tasks, and query the data collection tasks based on the timestamp information.

[0071] The log file system 324 is used to record the log files in the CDN system. For example, the log files include log information, alarm information, fault information, etc. generated by the CDN network element devices.

[0072] The feature generation module 331 is used to obtain multiple alternative processing algorithms generated by the main control module 341. For example, the main control module 341 generates a list of alternative processing algorithms [task1, task2,..., taskn], where taskn represents the nth alternative processing algorithm, and n is an integer greater than or equal to 1. The feature generation module 331 is also used to perform feature analysis on each sample metric data according to the multiple alternative processing algorithms respectively, and obtain multiple sample feature data corresponding to each sample metric data (for example, generate a data table [sample metric data, sample feature data 1, sample feature data 2,..., sample feature data n], etc.); according to the preset feature data and the multiple sample feature data corresponding to each sample metric data, select some alternative processing algorithms from the multiple alternative processing algorithms as the target processing algorithms, and obtain the target processing algorithm set.

[0073] Among them, the alternative processing algorithms can include at least one of the following: summation operation, mean operation, median operation, variance operation, standard deviation operation, range operation.

[0074] For example, when task1 is a summation operation, and the sample metric data includes param1, param2,..., paramk, the corresponding sample feature data is param1 + param2 +... + paramk. Among them, k represents the number of sample metric data, and k is an integer greater than or equal to 1.

[0075] For another example, when task2 is a mean operation, and the sample index data includes param1, param2, ..., paramk, the corresponding sample feature data is sum(param1, param2, ..., paramk) / k, where sum represents the sum of param1, param2, ..., paramk.

[0076] The machine learning scoring module 332 is used to match each sample feature data with preset feature data to obtain multiple matching coefficients, where the preset feature data is data representing feature information of the sample task; then, the multiple matching coefficients are screened to obtain a target matching coefficient set. For example, the preset feature data is feature data stored in a log file.

[0077] Among them, the target matching coefficient set includes at least one target matching coefficient, and the target matching coefficient is used to characterize the matching degree between the sample feature data and the preset feature data. The larger the target matching coefficient is, the higher the matching degree between the sample feature data and the preset feature data is.

[0078] In some embodiments, the target matching coefficient can also be represented as a score value for the sample feature data. For example, different sample feature data correspond to different score values, and the following score table can be obtained [sample feature data 1, score 1; sample feature data 2, score 2; ...; sample feature data m, score m], where m represents the number of sample feature data, and m is an integer greater than or equal to 1.

[0079] Furthermore, the score table is arranged in descending order to obtain the sample feature data with the highest score. For example, if the score corresponding to sample feature data 1 (data corresponding to the sum operation in the alternative processing algorithm) is 94, the score corresponding to sample feature data 2 (data corresponding to the variance operation in the alternative processing algorithm) is 92, the score corresponding to sample feature data 3 (data corresponding to the standard deviation operation in the alternative processing algorithm) is 91.5, the score corresponding to sample feature data 4 (data corresponding to the range operation in the alternative processing algorithm) is 82, ..., the score corresponding to sample feature data k (data corresponding to the mean operation in the alternative processing algorithm) is 23; then the sample feature data with the highest score is sample feature data 1 with a score of 94, that is, the feature data determined by the sum operation in the alternative processing algorithm, and the sum operation is determined as the target processing algorithm.

[0080] Then, the following information [sample feature data 1, score: 94, target processing algorithm (addition operation)] is sent to the algorithm solidification module 333.

[0081] The algorithm solidification module 333 is used to receive the information [sample feature data 1, score: 94, sum operation] sent by the machine learning scoring module 332, and extract the information, and convert the target processing algorithm (i.e., sum operation) therein into a target processing program. For example, the target processing program is expressed as: def sum(a,b):return a+b. Wherein, a and b both represent the input sample indicator data (or, the indicator data to be processed).

[0082] In some embodiments, there may be multiple target processing algorithms. For example, the first N candidate processing algorithms in the above-mentioned scoring table are all used as target processing algorithms to obtain a target processing algorithm set including N target processing algorithms, where N is an integer greater than 2); further, each sample feature data is matched with the preset feature data to obtain multiple matching coefficients, and the preset feature data is data characterizing the feature information of the sample task; the multiple matching coefficients are screened to obtain a target matching coefficient set; then, the algorithm solidification module 333 can solidify the corresponding multiple target processing algorithms according to the target matching coefficient set to form a target processing program.

[0083] By processing each module in the above-mentioned data processing system, a target processing program is obtained, so that when the indicator data to be processed is obtained subsequently, the target processing program can be used to directly process the indicator data to be processed, thereby quickly obtaining the characteristic data to be processed corresponding to the indicator data to be processed, and the characteristic data to be processed is used to optimize the communication network parameters.

[0084] For example, Figure 3 The data processing system shown is applied in a mobile communication network or a content distribution network, wherein the content distribution network includes multiple levels of network element devices, and each level of network element devices will generate different types of indicator data to be processed.

[0085] Figure 4 A schematic diagram showing a flow chart of a working method of a data processing system provided by an embodiment of the present application is shown. Figure 4 As shown, the working method of the data processing system includes but is not limited to the following steps.

[0086] Step S401 : the main control module 341 generates a list of candidate processing algorithms.

[0087] For example, Table 1 shows a list of candidate processing algorithms, where the candidate processing algorithms include a square algorithm and a mean algorithm.

[0088] Table 1 List of alternative processing algorithms

[0089] List of alternative processing algorithms Corresponding function representation Square x * x Mean mean(x)

[0090] Step S402: the main control module 341 sets monitoring indicators.

[0091] Among them, the monitoring metrics include the monitoring metrics between network elements and the monitoring metrics inside network elements, so as to monitor the network performance parameters in a mobile communication network or a content distribution network.

[0092] It should be noted that the monitoring metrics can be preset feature data (such as the feature data stored in a log file), which can represent the threshold data of the sample metric data between network elements (or inside network elements).

[0093] Step S403, the main control module 341 sends a list of alternative processing algorithms to the feature generation module 331.

[0094] Step S404, the feature generation module 331 sends a data acquisition signal triggered by a sample task to the data task management module 323.

[0095] Among them, the data acquisition signal is used to indicate the acquisition of the sample metric data corresponding to the sample task. For example, the data task management module 323 forwards the data acquisition signal to the data probe 312 through the data distribution interface 322, so that the data probe 312 can acquire the sample metric data corresponding to the sample task generated by the network element device 311 according to the indication of the data acquisition signal.

[0096] Step S405, when the data probe 312 acquires the sample metric data corresponding to the sample task generated by the network element device 311, the data probe 312 reports the sample metric data to the data task management module 323 through the data acquisition interface 321.

[0097] Step S406, the data task management module 323 marks the timestamp information for the sample metric data according to the chronological order of the received sample metric data, and forwards the sample metric data marked with the timestamp information to the feature generation module 331.

[0098] Step S407, the feature generation module 331 performs feature analysis on the sample metric data respectively according to multiple alternative processing algorithms, and obtains multiple sample feature data corresponding to the sample metric data.

[0099] For example, according to the "square" algorithm and the "mean" algorithm in the alternative processing algorithms in Table 1, feature analysis is performed on the sample metric data respectively, and multiple sample feature data corresponding to the sample metric data are obtained. Among them, the sample feature data corresponding to the "square" algorithm is the square value of the sample metric data; the sample feature data corresponding to the "mean" algorithm is the mean value of the sample metric data.

[0100] In step S408, the feature generation module 331 sends the generated multiple sample feature data to the machine learning scoring module 332, so that the machine learning scoring module 332 matches each sample feature data with the preset feature data respectively to obtain multiple matching coefficients; screens the multiple matching coefficients to obtain a target matching coefficient set; and based on the target matching coefficient set, screens some alternative processing algorithms from multiple alternative processing algorithms as target processing algorithms to obtain a target processing algorithm set.

[0101] The preset feature data is data that characterizes the feature information of the sample task. For example, the score in the scoring table is used to characterize the matching coefficient. The larger the matching coefficient, the higher the matching degree between the sample feature data and the preset feature data. The obtained scoring table is: [square value of sample index data, score 1; mean value of sample index data, score 2].

[0102] It should be noted that the above-mentioned scoring process can use a machine learning algorithm to evaluate multiple sample feature data to determine the degree of matching between the sample feature data and the preset feature data, thereby determining the corresponding matching coefficient.

[0103] In step S409, the machine learning scoring module 332 sends the obtained target matching coefficient set to the algorithm solidification module 333, so that the algorithm solidification module 333 converts the target processing algorithm in the target processing algorithm set into a target processing program according to the multiple target matching coefficients in the obtained target matching coefficient set.

[0104] Among them, the target processing program is used to extract features of the to-be-processed indicator data corresponding to the to-be-processed task.

[0105] In this embodiment, through the above-mentioned processing process, the machine learning algorithm can be used to find the sample feature data that best matches the sample task and determine the sample feature data, so as to clarify the feature information of the sample task in the content distribution network or the mobile communication network; further, the generated multiple sample feature data are matched with the preset feature data respectively to obtain multiple matching coefficients; the multiple matching coefficients are screened to obtain a target matching coefficient set; based on the target matching coefficient set, some alternative processing algorithms are screened from multiple alternative processing algorithms as target processing algorithms to obtain a target processing algorithm set, and finally the target processing algorithms in the target processing algorithm set are converted into a target processing program, so as to facilitate the subsequent use of the target processing program to quickly extract features from the to-be-processed indicator data corresponding to the to-be-processed task, improve the processing efficiency of the to-be-processed indicator data, and improve the accuracy of the feature extraction of the to-be-processed indicator data, so that the to-be-processed feature data corresponding to the obtained to-be-processed indicator data can be more quickly applied to the optimization of network performance, thereby providing users with better communication services.

[0106] It should be clear that the present invention is not limited to the specific configurations and processes described in the above embodiments and illustrated in the figures. For the convenience and brevity of description, the detailed description of known methods is omitted here, and for the specific working processes of the systems, modules, and units described above, reference may be made to the corresponding processes in the foregoing method embodiments, which will not be elaborated herein.

[0107] Figure 5 The block diagram showing the composition of an electronic device provided by an embodiment of the present application is shown.

[0108] As Figure 5 shown, the electronic device includes: at least one processor 501, at least one memory 502, and one or more I / O interfaces 503. Among them, the processor 501, the memory 502, and the I / O interface 503 are interconnected through a bus 504. The memory 502 stores one or more computer programs, and the one or more computer programs are executed by at least one processor 501 so that the at least one processor 501 can implement any one of the data processing methods described in the above embodiments.

[0109] Each module in the above electronic device can be implemented in whole or in part by software, hardware, and their combination. Each of the above modules can be embedded in the processor in the computer device in hardware form or independent of it, or stored in the memory in the computer device in software form, so that the processor can call and execute the operations corresponding to each of the above modules.

[0110] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, any one of the data processing methods described in the above embodiments is implemented. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0111] An embodiment of the present application also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in the processor of the electronic device, the processor in the electronic device executes the above data processing method.

[0112] Those of ordinary skill in the art will understand that all or some of the steps in the methods disclosed above, and the functional modules / units in systems and devices, can be implemented as software, firmware, hardware, and appropriate combinations thereof. In the hardware implementation, the division of functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be executed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable storage medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transitory medium).

[0113] As is well known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, as is well known to those of ordinary skill in the art, a communication medium typically includes computer-readable program instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and can include any information delivery medium.

[0114] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded to an external computer or external storage device through a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0115] The computer program instructions for performing the operations of the present application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine - related instructions, microcode, firmware instructions, state - setting data, or source code or object code written in any combination of one or more programming languages, including object - oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer - readable program instructions may be executed entirely on the user's computer, partially on the user's computer, executed as a stand - alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider). In some embodiments, by using the state information of the computer - readable program instructions to customize an electronic circuit, such as a programmable logic circuit, a field - programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer - readable program instructions to implement various aspects of the present application.

[0116] The computer program product described herein may be implemented specifically in the form of hardware, software, or a combination thereof. In an alternative embodiment, the computer program product is specifically embodied as a computer storage medium. In another alternative embodiment, the computer program product is specifically embodied as a software product, such as a Software Development Kit (SDK), etc.

[0117] Aspects of the present application are described herein with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present application. It should be understood that each block of the flowcharts and / or block diagrams, and the combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer - readable program instructions.

[0118] These computer-readable program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine such that the instructions, when executed by the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more boxes of the flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer-readable medium storing the instructions comprises a manufacture, including instructions that implement various aspects of the functions / acts specified in one or more boxes of the flowchart and / or block diagram.

[0119] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, such that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in one or more boxes of the flowchart and / or block diagram.

[0120] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, a segment of code, or a portion of an instruction, and the module, segment of code, or portion of an instruction may include one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two consecutive blocks may in fact be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system for performing the specified functions or acts, or by a combination of dedicated hardware and computer instructions.

[0121] Example embodiments have been disclosed herein, and although specific terms are employed, they are used in a general descriptive sense only and are not for purposes of limitation. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly stated, the features, characteristics, and / or elements described in connection with a particular embodiment may be used singly or in combination with those described in connection with other embodiments. Accordingly, those skilled in the art will appreciate that various forms and details may be changed without departing from the scope of the present application as set forth by the appended claims.

Claims

1. A data processing method, wherein, it includes: responding to a data acquisition signal triggered by a sample task in a content delivery network, and acquiring a plurality of sample metric data corresponding to the sample task; performing feature analysis on each of the sample metric data according to a plurality of alternative processing algorithms respectively, and obtaining a plurality of sample feature data corresponding to each of the sample metric data; screening out some of the alternative processing algorithms from the plurality of alternative processing algorithms as target processing algorithms according to preset feature data and the plurality of sample feature data corresponding to each of the sample metric data, and obtaining a target processing algorithm set; determining a target processing program according to the target processing algorithms in the target processing algorithm set, where the target processing program is used to perform feature extraction on the to-be-processed metric data corresponding to the to-be-processed task.

2. The method according to claim 1, wherein, the preset feature data is the feature data stored in a log file; the screening out some of the alternative processing algorithms from the plurality of alternative processing algorithms as target processing algorithms according to the preset feature data and the plurality of sample feature data corresponding to each of the sample metric data, and obtaining a target processing algorithm set includes: matching the plurality of sample feature data corresponding to each of the sample metric data with the preset feature data respectively, and determining the matching degree between the plurality of sample feature data corresponding to each of the sample metric data and the preset feature data; screening out some of the alternative processing algorithms from the plurality of alternative processing algorithms as the target processing algorithms according to the matching degree, and obtaining the target processing algorithm set.

3. The method according to claim 2, wherein, the matching the plurality of sample feature data corresponding to each of the sample metric data with the preset feature data respectively, and determining the matching degree between the plurality of sample feature data corresponding to each of the sample metric data and the preset feature data includes: matching each of the sample feature data with the preset feature data respectively, and obtaining a plurality of matching coefficients, where the preset feature data is the data characterizing the feature information of the sample task; screening the plurality of matching coefficients to obtain a target matching coefficient set; wherein, the target matching coefficient set includes at least one target matching coefficient, the target matching coefficient is used to characterize the matching degree between the sample feature data and the preset feature data, and the larger the target matching coefficient, the higher the matching degree between the sample feature data and the preset feature data.

4. The method according to claim 3, wherein, the sample task includes a load balancing task in the content delivery network, the sample metric data includes the central processing unit usage rate, the network bandwidth occupancy rate, and the network element load quantity, and the preset feature data includes a preset central processing unit usage rate threshold, a preset network bandwidth occupancy rate threshold, and a preset network element load quantity threshold; the matching each of the sample feature data with the preset feature data respectively, and obtaining a plurality of matching coefficients includes: Match the usage rate characteristic data corresponding to the central processor usage rate with the preset central processor usage rate threshold to obtain a first matching coefficient; Match the bandwidth characteristic data corresponding to the network bandwidth occupancy rate with the preset network bandwidth occupancy rate threshold to obtain a second matching coefficient; Match the network element load characteristic data corresponding to the number of network element loads with the preset network element load number threshold to obtain a third matching coefficient.

5. The method according to claim 4, wherein, The sample task further includes at least one of the following: a data distribution task, a task of translating first-class data, a task of deleting second-class data, a real-time data pulling task, a network performance reporting task; wherein, the first-class data is data with a user usage frequency higher than a preset frequency threshold, and the second-class data is data with a user usage frequency lower than the preset frequency threshold; The sample metric data further includes at least one of the following: memory occupancy rate, disk occupancy rate, disk input speed, and disk output speed.

6. The method according to claim 3, wherein, The preset feature data includes: preset fault data and first preset frequency data of the preset fault data appearing in the log file; and / or, preset alarm data and second preset frequency data of the preset alarm data appearing in the log file; The sample feature data includes: sample fault data and its corresponding first sample frequency data, and / or, sample alarm data and its corresponding second sample frequency data.

7. The method according to claim 6, wherein, When the sample fault data is the same as the preset fault data, the closer the first sample frequency data is to the first preset frequency data, the greater the matching coefficient; and / or, When the sample alarm data is the same as the preset alarm data, the closer the second sample frequency data is to the second preset frequency data, the greater the matching coefficient.

8. The method according to any one of claims 1 to 7, wherein, After determining the target processing program according to the target processing algorithm in the target processing algorithm set, the method further includes: In response to a data acquisition signal triggered by a task to be processed, obtain the to-be-processed metric data corresponding to the task to be processed; Input the to-be-processed metric data into the target processing program for feature extraction to obtain the to-be-processed feature data corresponding to the to-be-processed metric data, and the to-be-processed feature data is used to optimize communication network parameters.

9. An electronic device, wherein, includes: One or more processors; A memory having one or more programs stored thereon, and when the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method according to any one of claims 1 to 8.

10. A readable storage medium, wherein, The readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the data processing method according to any one of claims 1 to 8.

Images