Big data-oriented data link optimization method and system

Abstract

The invention discloses a big data-oriented data link optimization method and system, and relates to the technical field of data link optimization, and the method comprises the steps: obtaining monitoring data of all link nodes in a big data service operation range, and transmitting the obtained monitoring data to an adjustment and optimization center; deeply analyzing the link nodes with the comprehensive transmission quality below the standard by the tuning center, and identifying key factors influencing the transmission quality of the nodes; using an intelligent optimization algorithm of the link nodes to carry out targeted adjustment and optimization on the link nodes of which the comprehensive transmission quality does not reach the standard; and according to the actual change of each monitoring data of the link node before and after tuning, the influence coefficient of the key factor on the comprehensive transmission quality of the current link node is calibrated. Various user-defined configuration scenes are provided, so that the specific requirements of specific scenes can be met, and the universality and portability are not lost; by adopting the targeted tuning method, the rapid implementation of the tuning method can be ensured, and the accuracy of the tuning result can be ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of data link optimization, and in particular to a data link optimization method and system for big data. Background Art

[0002] Data link optimization methods for big data are an important research direction in the current information technology field, aiming to improve the efficiency and reliability of big data transmission, processing and analysis. At present, optimization methods based on data flow, task and resource scheduling, performance modeling and configuration have made significant progress, but there are still some defects and challenges: big data processing systems involve multiple components and complex interactions. Existing optimization methods are often targeted at specific scenarios and specific needs, lacking versatility and portability; big data processing needs to meet real-time requirements, but existing technologies often find it difficult to ensure data accuracy while ensuring processing speed; therefore, researchers in this field still need to strengthen, improve and optimize existing data links to solve the above problems. Summary of the invention

[0003] The present invention provides a data link optimization method for big data, comprising:

[0004] Step 1: Obtain monitoring data of all link nodes within the operation scope of the big data service, and transmit the acquired monitoring data to the tuning center;

[0005] Step 2: The tuning center calculates the comprehensive transmission quality of each link node according to the focus target of data link detection;

[0006] Step 3: Conduct in-depth analysis on link nodes with comprehensive transmission quality below the standard, identify the key factors affecting the transmission quality of the node, and place spare nodes in concentrated areas where the comprehensive transmission quality does not meet the standard. When a link node is out of service, enable the spare node to inherit the transmission service of the out-of-service link node.

[0007] Step 4: Use the intelligent optimization algorithm of the link node to calculate the influence coefficient of the key factors on the comprehensive transmission quality of the current link node, and perform targeted optimization on the link nodes whose comprehensive transmission quality does not meet the standard based on the calculation results;

[0008] Step 5: According to the actual changes of various monitoring data of the link node before and after the optimization, calibrate the influence coefficient of the key factors on the comprehensive transmission quality of the current link node.

[0009] As described above, a data link optimization method for big data, in which monitoring data of all link nodes within the operation scope of the big data service is obtained, and the obtained monitoring data is transmitted to the tuning center, specifically includes the following sub-steps:

[0010] Generate a topology diagram of the transmission link according to the actual deployment position and link relationship of each link node;

[0011] Configure monitoring data items of different link nodes in the topology diagram and select monitoring mode;

[0012] Based on the selected monitoring mode, the monitoring data on each link node is captured to the tuning center.

[0013] As described above, a data link optimization method for big data is provided, wherein link nodes with comprehensive transmission quality below the standard are analyzed in depth, specifically including the following sub-steps:

[0014] Add link nodes whose comprehensive transmission quality is lower than the standard value to the identification queue;

[0015] Traverse the identification queue. If the current link node is in an outage state, transfer the transmission service of the current node to the nearest backup node. After the staff releases the outage state, transfer the transmission service back to the current link node.

[0016] If the current link node is in operation, the value sequence of the comprehensive transmission quality of the current link node in the most recent identification cycle is extracted and input into the key factor identification model, and the key factors affecting the transmission quality of the node are output.

[0017] As described above, a data link optimization method for big data is described, in which an intelligent optimization algorithm for link nodes is used to calculate the influence coefficient of key factors on the comprehensive transmission quality of the current link nodes, and link nodes with substandard comprehensive transmission quality are optimized in a targeted manner according to the calculation results, specifically including the following sub-steps:

[0018] Query the node monitoring data from the historical monitoring records, except for the key factors, whose other factors are consistent with the link node to be optimized;

[0019] According to the queried monitoring data, calculate the influence coefficient of key factors on the comprehensive transmission quality of the current link node;

[0020] The optimal solution of the key factor on the current link node is estimated based on the calculation result of the influence coefficient, and the value of the key factor is updated as the optimal solution.

[0021] The present invention also provides a data link optimization system for big data, including: a link node monitoring module and a tuning center;

[0022] The link node monitoring module is used to obtain the monitoring data of all link nodes within the operation scope of the big data service and transmit the obtained monitoring data to the tuning center;

[0023] The tuning center is used to optimize link nodes whose comprehensive transmission quality does not meet the standards based on the monitoring data obtained.

[0024] A data link optimization system for big data as described above, wherein the tuning center includes:

[0025] A comprehensive transmission quality calculation module is used to calculate the comprehensive transmission quality of each link node according to the focus target of data link detection;

[0026] The link node analysis and switching module is used to conduct in-depth analysis of link nodes with comprehensive transmission quality below the standard, identify the key factors affecting the transmission quality of the node, and enable the standby node to inherit the transmission service of the out-of-service link node when the link node is out of service;

[0027] The link node optimization module is used to use the intelligent optimization algorithm of the link node to calculate the influence coefficient of the key factors on the comprehensive transmission quality of the current link node, and optimize the link node whose comprehensive transmission quality does not meet the standard according to the calculation results;

[0028] The optimization parameter calibration module is used to calibrate the influence coefficient of key factors on the comprehensive transmission quality of the current link node according to the actual changes of various monitoring data of the link node before and after the optimization.

[0029] The beneficial effects achieved by the present invention are as follows: providing a variety of custom configuration scenarios, which can meet the specific needs of specific scenarios while maintaining versatility and portability; adopting targeted tuning methods can ensure both the rapid implementation of the tuning methods and the accuracy of the tuning results. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0031] Figure 1 This is a flow chart of a data link optimization method for big data provided in Example 1 of the present application;

[0032] Figure 2 This is a schematic diagram of a data link optimization system for big data provided in Example 2 of the present application. DETAILED DESCRIPTION

[0033] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0034] Embodiment 1

[0035] like Figure 1 As shown, the first embodiment of the present application provides a data link optimization method for big data, including:

[0036] Step S10: Acquire monitoring data of all link nodes within the operation scope of the big data service, and transmit the acquired monitoring data to the tuning center;

[0037] Big data services have high-intensity data interaction and complex interactive relationships, so the monitored link nodes must cover all data sources, data processing centers, data transmission channels, data storage facilities and final data application services within the scope of operation. The specific monitoring process includes the following sub-steps:

[0038] Step S11: generating a topology diagram of a transmission link according to the actual deployment position and link relationship of each link node;

[0039] Link nodes are stored in the form of a topology graph to trace data sources, making data transmission relationships transparent and thereby improving link optimization efficiency.

[0040] Step S12: configuring monitoring data items of different link nodes in the topology diagram and selecting a monitoring mode;

[0041] The monitoring data items are user-defined and can be added and modified according to the special monitoring requirements of different link nodes. The system preset monitoring data items include: the type, size, quantity, timestamp, source node and status code of the data packet passing through the node.

[0042] Different monitoring modes are used to adapt to different monitoring scenarios, including: real-time monitoring mode, periodic monitoring mode and event-triggered monitoring mode; real-time monitoring mode is suitable for scenarios that require immediate feedback and rapid response, such as financial trading systems, online game servers, real-time communication systems, etc.; periodic monitoring mode is suitable for systems with relatively stable data changes that do not require real-time monitoring, such as enterprise resource planning (ERP) systems, data warehouses, etc.; event-triggered monitoring mode is suitable for scenarios that require special attention when certain specific events occur, such as large database backups, before and after system upgrades, etc. Choose the appropriate monitoring mode based on the specific circumstances of the actual monitoring scenario.

[0043] Step S13: based on the selected monitoring mode, capture the monitoring data on each link node to the tuning center;

[0044] The tuning center can be deployed locally or on a cloud server to perform data link tuning. The captured monitoring data must first be pre-processed by denoising, cleaning, and normalization before it can be uploaded to the tuning center.

[0045] Step S20: the tuning center calculates the comprehensive transmission quality of each link node according to the focus target of the data link detection;

[0046] Before optimizing the data link, the user needs to configure the focus of the data link detection in the tuning center. These configurable items are to improve the versatility and portability of the optimization method. Specifically:

[0047] Step S21: configuring a focus target of data link detection for a link node;

[0048] Users can set unique focus targets for each data link node to meet the special application scenarios of different nodes. The system preset data link detection focus targets include: delay, that is, the time required for a data packet to reach the receiver from the sender; packet loss rate, that is, the proportion of data packets lost during transmission to the total data packets; throughput, that is, the amount of data successfully transmitted per unit time; jitter, that is, the degree of delay variation; error rate, that is, the proportion of data packets with errors during transmission to the total number. In addition to the above system preset configuration items, users can also customize and add or modify other configuration items according to actual conditions.

[0049] Step S22: encapsulating the value acquisition function for each emphasis target;

[0050] The value acquisition function is also added and modified by the user to extract the value of a certain focus target from the monitoring data. For example, the value acquisition function of the delay focus target can be expressed as: G(R (i) )=|r rtime -r stime |, where R (i) is the i-th monitoring record obtained by the current link node, r rtime Indicates monitoring record X (i) The packet receiving time (the time it arrives at this node), r stime Indicates monitoring record R (i) The sending time of the data packet (the time it leaves the previous node). It should be noted that the parameters entering the value acquisition function must be normalized.

[0051] Step S23: Calculate the comprehensive transmission quality of each link node based on the encapsulated value function;

[0052] The calculation formula of comprehensive transmission quality is expressed as:

[0053] Where Q k represents the comprehensive transmission quality of the kth link node, G j (R (ki) ) represents the j-th target-focused value acquisition function, R (ki) represents the i-th monitoring record obtained on the k-th link node, L (j) Indicates that in the entire transmission link topology, the number of link nodes focusing on target j is set, L (total) represents the total number of link nodes contained in the entire transmission link topology graph, λ j Indicates the calculation flag of the jth focus target. When the focus target j is a positive target (i.e., the higher the value, the better), λ j The value is 1. When the focus target j is a negative target (i.e., the lower the value, the better), λ j The value is -1, j is 1 to m, m is the number of emphasis targets set for the kth link node, i is 1 to n, n is the total number of monitoring records obtained on the kth link node.

[0054] Step S30: Conduct in-depth analysis on link nodes whose comprehensive transmission quality is below the standard, identify key factors affecting the transmission quality of the node, place spare nodes in concentrated areas where the comprehensive transmission quality does not meet the standard, and when a link node is out of service, enable the spare node to inherit the transmission service of the out-of-service link node;

[0055] The concentrated area where the comprehensive transmission quality does not meet the standard refers to the area where the number of link nodes that do not meet the comprehensive transmission quality is significantly higher within the unit range. This can be determined by observing the transmission link topology diagram. Identifying the key factors that affect the transmission quality of link nodes specifically includes the following sub-steps:

[0056] Step S31: adding link nodes whose comprehensive transmission quality is lower than the standard value to the identification queue;

[0057] The standard value of the comprehensive transmission quality is calculated based on the lowest critical values ​​of various monitoring data of link nodes where it is known that the transmission operation is in good performance.

[0058] Step S32: traverse the identification queue, if the current link node is in an outage state, transfer the transmission service of the current node to the nearest standby node, and transfer the transmission service back to the current link node after the staff releases the outage state;

[0059] The distance between the current link node and the standby node refers to the physical transmission distance, which is calculated based on the physical IP addresses of the two nodes.

[0060] Step S33: if the current link node is in the running state, extract the value sequence of the comprehensive transmission quality of the current link node in the most recent identification cycle, input it into the key factor identification model, and output the key factors affecting the transmission quality of the node;

[0061] The recognition cycle refers to the minimum requirement of the key factor recognition model for the length of the input set, which is also the minimum requirement to ensure the recognition accuracy of the model. The training process of the key factor recognition model includes the following sub-steps:

[0062] Step S331: arranging the historical diagnosis records of the link nodes into a training data set;

[0063] The historical diagnosis record records the historical monitoring data of different link nodes when the transmission service is abnormal, as well as the diagnosis results at that time. This diagnosis result includes the key factors that affect the comprehensive transmission quality of the link node. The comprehensive transmission quality calculation formula is used to calculate the value sequence of the comprehensive transmission quality of each link node during the abnormal record period as the input sample, and the key factors mentioned in the diagnosis result are extracted as the output label. The input and output pairs are sorted and stored as the training data set P, which is expressed as: P = {[X u ,Y u ]}, where X u is the input value sequence, Y u is the key factor label of the output, u takes values ​​from 1 to A, and A is the total number of training samples.

[0064] Step S332: using the training data set to train the key factor identification model and determine the model parameters;

[0065] The key factor identification model φ(X) is expressed as:

[0066]

[0067] Where X is the input comprehensive transmission quality value sequence, x v 、x v+1 、x v+2 、x B-1 、x B They represent the values ​​of the value sequence X when the subscripts are v, v+1, v+2, B-1, and B, respectively. v takes values ​​from 1 to B-1, and B is the length of the value sequence X. σ1, σ2, …, σ z are the error terms in 1 to z hidden layers respectively, are the weights in 1 to z hidden layers respectively.

[0068] The key factor identification model is trained using the training data set, and the parameters in the model are continuously adjusted during the training process until the accuracy of the model is no longer significantly improved.

[0069] Step S333: verify the influence of input samples of different lengths on the model accuracy and determine the recognition period of the model;

[0070] After multiple batches of input samples of varying lengths have passed through the model, the difference between the model output and the actual labels is compared. It is found that input samples that are too short will cause the model to have an inaccurate understanding of the data, while input samples that are too long will cause the model to pay less attention to valid recognition items, which will also affect the model accuracy. Therefore, after the model parameters are confirmed, the most appropriate sample length must be selected as the model's recognition period to enable the model to have better predictive and adaptable capabilities.

[0071] Step S40: using the intelligent optimization algorithm of the link node, calculating the influence coefficient of the key factors on the comprehensive transmission quality of the current link node, and performing targeted optimization on the link node whose comprehensive transmission quality does not meet the standard according to the calculation result;

[0072] Tuning the identified key factors can improve the tuning rate and enable link nodes to quickly adapt to the current service scenario. The link node intelligent optimization algorithm specifically includes the following sub-steps:

[0073] Step S41: Query the node monitoring data whose factors, except for the key factors, are consistent with the link node to be optimized from the historical monitoring records;

[0074] First, all link nodes contained in the historical monitoring records are counted, and the configuration files of these link nodes are extracted. The link nodes whose configuration items are consistent with the link nodes to be optimized except for the configuration items corresponding to the key factors are screened out, and stored as the node set C1; then, the monitoring data of the node set C1 is queried from the historical monitoring records and stored as the monitoring data set C2; then, the monitoring data in the monitoring data set C2 with a similarity of more than 90% with the current link node throughput and transmission data packet type are screened and stored as C3, and the C3 data set is returned.

[0075] Step S42: Calculate the influence coefficient of the key factors on the comprehensive transmission quality of the current link node according to the queried monitoring data;

[0076] First, the queried monitoring data, namely the C3 data set, is grouped and sorted according to the values ​​of the key factors. Then, the comprehensive transmission quality calculation formula is used to calculate the comprehensive transmission quality of each set of monitoring data, and the data pairs are formed with the values ​​of the key factors. These data pairs are processed as input parameter sets, recorded as C4, and C4 is passed into the influence coefficient calculation function to return the influence coefficient of the key factor on the comprehensive transmission quality of the current link node. The influence coefficient calculation function S(C4) is expressed as:

[0077] Among them, C4 is the input parameter set, E his the value of the key factor in the hth data pair in the input parameter set, E h+1 is the value of the key factor in the h+1th data pair in the input parameter set, is the mean of all key factors in the input set, F h is the comprehensive transmission quality of the h-th data pair in the input parameter set, F h+1 is the comprehensive transmission quality of the h+1th data pair in the input parameter set, is the mean of all comprehensive transmission qualities in the input parameter set, h ranges from 1 to H-1, and H is the total number of data pairs in the input parameter set.

[0078] Step S43: estimating the optimal solution of the key factor on the current link node based on the calculation result of the influence coefficient, and updating the value of the key factor to be the optimal solution;

[0079] Use the formula: E best =E current +S(C4)×|F current -F target | Estimate the optimal solution of key factors at the current link node E best , where E current Indicates the current value of the key factor. S(C4) returns the calculation result of the influence coefficient of the key factor on the current link node. F current is the comprehensive transmission quality of the current link node, F target It is the target transmission quality that the current link node wants to achieve.

[0080] Step S50: calibrating the influence coefficient of key factors on the comprehensive transmission quality of the current link node according to the actual changes of various monitoring data of the link node before and after the optimization;

[0081] In step S40, the influence coefficient of the key factor on the comprehensive transmission quality of the current link node is calculated based on the historical monitoring data and has a certain lag. In order to ensure the accuracy and applicability of the tuning work, it is necessary to calibrate the influence coefficient in combination with the actual monitoring data to provide scientific data support for subsequent tuning work. The calibration of the influence coefficient specifically includes the following sub-steps:

[0082] Step S51: Acquire monitoring data of the optimized link node;

[0083] Step S52: Calculate the comprehensive transmission quality of the optimized link node according to the acquired monitoring data;

[0084] Step S53: Processing the monitored comprehensive transmission quality of the link nodes before and after the tuning and the values ​​of the key factors as the input parameter set of the influence coefficient calculation function, and returning the calibration result of the influence coefficient.

[0085] Embodiment 2

[0086] like Figure 2 As shown, Embodiment 2 of the present application provides a data link optimization system for big data, including: a link node monitoring module 1 and a tuning center 2, the link node monitoring module 1 is used to obtain the monitoring data of all link nodes within the operation scope of the big data service, and transmit the acquired monitoring data to the tuning center; the tuning center 2 is used to optimize the link nodes whose comprehensive transmission quality does not meet the standards according to the acquired monitoring data; the tuning center 2 is provided with a comprehensive transmission quality calculation module 21, a link node analysis and switching module 22, a link node optimization module 23, and an optimization parameter calibration module 24.

[0087] The link node monitoring module 1 includes: a transmission link topology map creation submodule, a monitoring configuration submodule, and a monitoring data acquisition and transmission submodule;

[0088] The transmission link topology map creation submodule is used to generate a transmission link topology map according to the actual deployment location and link relationship of each link node;

[0089] Link nodes are stored in the form of a topology graph to trace data sources, making data transmission relationships transparent and thereby improving link optimization efficiency.

[0090] The monitoring configuration submodule is used to configure the monitoring data items of different link nodes in the topology diagram and select the monitoring mode;

[0091] The monitoring data items are user-defined and can be added and modified according to the special monitoring requirements of different link nodes. The system preset monitoring data items include: the type, size, quantity, timestamp, source node and status code of the data packet passing through the node.

[0092] Different monitoring modes are used to adapt to different monitoring scenarios, including: real-time monitoring mode, periodic monitoring mode and event-triggered monitoring mode; real-time monitoring mode is suitable for scenarios that require immediate feedback and rapid response, such as financial trading systems, online game servers, real-time communication systems, etc.; periodic monitoring mode is suitable for systems with relatively stable data changes that do not require real-time monitoring, such as enterprise resource planning (ERP) systems, data warehouses, etc.; event-triggered monitoring mode is suitable for scenarios that require special attention when certain specific events occur, such as large database backups, before and after system upgrades, etc. Choose the appropriate monitoring mode based on the specific circumstances of the actual monitoring scenario.

[0093] The monitoring data acquisition and transmission submodule is used to capture the monitoring data on each link node to the tuning center based on the selected monitoring mode;

[0094] The tuning center can be deployed locally or on a cloud server to perform data link tuning. The captured monitoring data must first be pre-processed by denoising, cleaning, and normalization before it can be uploaded to the tuning center.

[0095] The comprehensive transmission quality calculation module 21 is used to calculate the comprehensive transmission quality of each link node according to the focus target of the data link detection; it includes: a focus target configuration submodule, a value function encapsulation submodule, and a comprehensive transmission quality calculation submodule;

[0096] A focus target configuration submodule is used to configure a focus target of data link detection for a link node;

[0097] Users can set unique focus targets for each data link node to meet the special application scenarios of different nodes. The system preset data link detection focus targets include: delay, that is, the time required for a data packet to reach the receiver from the sender; packet loss rate, that is, the proportion of data packets lost during transmission to the total data packets; throughput, that is, the amount of data successfully transmitted per unit time; jitter, that is, the degree of delay variation; error rate, that is, the proportion of data packets with errors during transmission to the total number. In addition to the above system preset configuration items, users can also customize and add or modify other configuration items according to actual conditions.

[0098] The value function encapsulation submodule is used to encapsulate the value acquisition function for each focus target;

[0099] The value acquisition function is also added and modified by the user to extract the value of a certain focus target from the monitoring data. For example, the value acquisition function of the delay focus target can be expressed as: G(R (i) )=|r rtime -r stime |, where R (i) is the i-th monitoring record obtained by the current link node, r rtime Indicates monitoring record X (i) The packet receiving time (the time it arrives at this node), r stime Indicates monitoring record R (i) The sending time of the data packet (the time it leaves the previous node). It should be noted that the parameters entering the value acquisition function must be normalized.

[0100] The comprehensive transmission quality calculation submodule is used to calculate the comprehensive transmission quality of each link node based on the encapsulated value function;

[0101] The calculation formula of comprehensive transmission quality is expressed as:

[0102] Where Q krepresents the comprehensive transmission quality of the kth link node, G j (R (ki) ) represents the j-th target-focused value acquisition function, R (ki) represents the i-th monitoring record obtained on the k-th link node, L (j) Indicates that in the entire transmission link topology, the number of link nodes focusing on target j is set, L (total) represents the total number of link nodes contained in the entire transmission link topology graph, λ j Indicates the calculation flag of the jth focus target. When the focus target j is a positive target (i.e., the higher the value, the better), λ j The value is 1. When the focus target j is a negative target (i.e., the lower the value, the better), λ j The value is -1, j is 1 to m, m is the number of emphasis targets set for the kth link node, i is 1 to n, n is the total number of monitoring records obtained on the kth link node.

[0103] The link node analysis and switching module 22 is used to conduct in-depth analysis on the link nodes whose comprehensive transmission quality is below the standard, identify the key factors affecting the transmission quality of the node, and enable the standby node to inherit the transmission service of the out-of-service link node when the link node is out of service; it includes: a standby node placement submodule, a transmission service switching submodule, and a key factor identification submodule;

[0104] The standard value of the comprehensive transmission quality is calculated based on the lowest critical values ​​of various monitoring data of link nodes with known good transmission operation execution status. Link nodes with comprehensive transmission quality lower than the standard value will be added to the identification queue, and the link node analysis and switching module 22 will analyze the link nodes in the identification queue in turn and return the analysis results.

[0105] A standby node placement submodule is used to place standby nodes in a centralized area where the comprehensive transmission quality does not meet the standard;

[0106] A concentrated area where the comprehensive transmission quality does not meet the standards refers to an area where the number of link nodes where the comprehensive transmission quality does not meet the standards is significantly greater within a unit range, which can be determined by observing the transmission link topology diagram.

[0107] The transmission service transfer submodule is used to determine whether the current link node is in an outage state. If so, the transmission service of the current node will be transferred to the nearest standby node. After the staff releases the outage state, the transmission service will be transferred back to the current link node. Otherwise, it will be handled by the key factor identification submodule;

[0108] The distance between the current link node and the standby node refers to the physical transmission distance, which is calculated based on the physical IP addresses of the two nodes.

[0109] The key factor identification submodule is used to extract the value sequence of the comprehensive transmission quality of the current link node in the most recent identification cycle, input it into the key factor identification model, and output the key factors that affect the transmission quality of the node;

[0110] The recognition cycle refers to the minimum requirement of the key factor recognition model for the length of the input set, which is also the minimum requirement to ensure the recognition accuracy of the model. The training process of the key factor recognition model includes:

[0111] I. Organize the historical diagnostic records of link nodes into training data sets;

[0112] The historical diagnosis record records the historical monitoring data of different link nodes when the transmission service is abnormal, as well as the diagnosis results at that time. This diagnosis result includes the key factors that affect the comprehensive transmission quality of the link node. The comprehensive transmission quality calculation formula is used to calculate the value sequence of the comprehensive transmission quality of each link node during the abnormal record period as the input sample, and the key factors mentioned in the diagnosis result are extracted as the output label. The input and output pairs are sorted and stored as the training data set P, which is expressed as: P = {[X u ,Y u ]}, where X u is the input value sequence, Y u is the key factor label of the output, u takes values ​​from 1 to A, and A is the total number of training samples.

[0113] II. Use the training data set to train the key factor identification model and determine the model parameters;

[0114] The key factor identification model φ(X) is expressed as:

[0115]

[0116] Where X is the input comprehensive transmission quality value sequence, x v 、x v+1 、x v+2 、x B-1 、x B They represent the values ​​of the value sequence X when the subscripts are v, v+1, v+2, B-1, and B, respectively. v takes values ​​from 1 to B-1, and B is the length of the value sequence X. σ1, σ2, …, σ z are the error terms in 1 to z hidden layers respectively, are the weights in 1 to z hidden layers respectively.

[0117] The key factor identification model is trained using the training data set, and the parameters in the model are continuously adjusted during the training process until the accuracy of the model is no longer significantly improved.

[0118] III. Verify the impact of input samples of different lengths on model accuracy and determine the recognition cycle of the model;

[0119] After multiple batches of input samples of varying lengths have passed through the model, the difference between the model output and the actual labels is compared. It is found that input samples that are too short will cause the model to have an inaccurate understanding of the data, while input samples that are too long will cause the model to pay less attention to valid recognition items, which will also affect the model accuracy. Therefore, after the model parameters are confirmed, the most appropriate sample length must be selected as the model's recognition period to enable the model to have better predictive and adaptable capabilities.

[0120] The link node optimization module 23 is used to use the intelligent optimization algorithm of the link node to calculate the influence coefficient of the key factors on the comprehensive transmission quality of the current link node, and optimize the link node whose comprehensive transmission quality does not meet the standard according to the calculation result; including: monitoring data extraction submodule, influence coefficient calculation submodule, key factor optimization submodule;

[0121] The monitoring data extraction submodule is used to query the node monitoring data whose other factors are consistent with the link node to be optimized from the historical monitoring records except the key factors;

[0122] First, all link nodes contained in the historical monitoring records are counted, and the configuration files of these link nodes are extracted. The link nodes whose configuration items are consistent with the link nodes to be optimized except for the configuration items corresponding to the key factors are screened out, and stored as the node set C1; then, the monitoring data of the node set C1 is queried from the historical monitoring records and stored as the monitoring data set C2; then, the monitoring data in the monitoring data set C2 with a similarity of more than 90% with the current link node throughput and transmission data packet type are screened and stored as C3, and the C3 data set is returned.

[0123] The influence coefficient calculation submodule is used to calculate the influence coefficient of the key factors on the comprehensive transmission quality of the current link node based on the queried monitoring data;

[0124] First, the queried monitoring data, namely the C3 data set, is grouped and sorted according to the values ​​of the key factors. Then, the comprehensive transmission quality calculation formula is used to calculate the comprehensive transmission quality of each set of monitoring data, and the data pairs are formed with the values ​​of the key factors. These data pairs are processed as input parameter sets, recorded as C4, and C4 is passed into the influence coefficient calculation function to return the influence coefficient of the key factor on the comprehensive transmission quality of the current link node. The influence coefficient calculation function S(C4) is expressed as:

[0125] Among them, C4 is the input parameter set, E h is the value of the key factor in the hth data pair in the input parameter set, E h+1is the value of the key factor in the h+1th data pair in the input parameter set, is the mean of all key factors in the input set, F h is the comprehensive transmission quality of the h-th data pair in the input parameter set, F h+1 is the comprehensive transmission quality of the h+1th data pair in the input parameter set, is the mean of all comprehensive transmission qualities in the input parameter set, h ranges from 1 to H-1, and H is the total number of data pairs in the input parameter set.

[0126] The key factor tuning submodule is used to estimate the optimal solution of the key factor on the current link node based on the influence coefficient calculation result, and update the value of the key factor to the optimal solution;

[0127] Use the formula: E best =E current +S(C4)×|F current -F target | Estimate the optimal solution of key factors at the current link node E best , where E current Indicates the current value of the key factor. S(C4) returns the calculation result of the influence coefficient of the key factor on the current link node. F current is the comprehensive transmission quality of the current link node, F target It is the target transmission quality that the current link node wants to achieve.

[0128] The optimization parameter calibration module 24 is used to calibrate the influence coefficient of the key factors on the comprehensive transmission quality of the current link node according to the actual changes of the monitoring data of the link node before and after the optimization;

[0129] In the link node optimization module 23, the influence coefficient of the key factors on the comprehensive transmission quality of the current link node is calculated based on the historical monitoring data and has a certain lag. In order to ensure the accuracy and applicability of the tuning work, it is necessary to calibrate the influence coefficient in combination with the actual monitoring data to provide scientific data support for subsequent tuning work. The calibration of the influence coefficient specifically includes:

[0130] I. Obtain monitoring data of the link nodes after tuning;

[0131] II. Calculate the comprehensive transmission quality of the link node after tuning based on the acquired monitoring data;

[0132] III. The monitored comprehensive transmission quality of the link nodes before and after the optimization and the values ​​of the key factors are processed as the input parameter set of the influence coefficient calculation function, and the calibration result of the influence coefficient is returned.

[0133] Corresponding to the above embodiment, an embodiment of the present invention provides a computer storage medium, including: at least one memory and at least one processor;

[0134] The memory is used to store one or more program instructions;

[0135] A processor is used to run one or more program instructions to execute a data link optimization method for big data.

[0136] Corresponding to the above-mentioned embodiment, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer storage medium contains one or more program instructions, and the one or more program instructions are used by a processor to execute a data link optimization method for big data.

[0137] The embodiments disclosed in the present invention provide a computer-readable storage medium, in which computer program instructions are stored. When the computer program instructions are executed on a computer, the computer executes the above-mentioned data link optimization method for big data.

[0138] In the embodiment of the present invention, the processor may be an integrated circuit chip having the ability to process signals. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0139] The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The processor reads the information in the storage medium and completes the steps of the above method in combination with its hardware.

[0140] The storage medium may be a memory, which may be, for example, a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memory.

[0141] Among them, the non-volatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory.

[0142] The volatile memory may be a random access memory (RAM) which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDRSDRAM), enhanced synchronous DRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus DRAM (DRRAM).

[0143] The storage media described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0144] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the present invention can be implemented using a combination of hardware and software. When software is used, the corresponding functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. Storage media can be any available media that can be accessed by general or special-purpose computers.

[0145] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data link optimization method for big data, characterized in that: include: Step 1: Obtain monitoring data of all link nodes within the operation scope of the big data service, and transmit the acquired monitoring data to the tuning center; Step 2: The tuning center calculates the comprehensive transmission quality of each link node according to the focus target of data link detection; Step 3: Conduct in-depth analysis on link nodes with comprehensive transmission quality below the standard, identify the key factors affecting the transmission quality of the node, and place spare nodes in concentrated areas where the comprehensive transmission quality does not meet the standard. When a link node is out of service, enable the spare node to inherit the transmission service of the out-of-service link node. Step 4: Use the intelligent optimization algorithm of the link node to calculate the influence coefficient of the key factors on the comprehensive transmission quality of the current link node, and perform targeted optimization on the link nodes whose comprehensive transmission quality does not meet the standard based on the calculation results; Step 5: According to the actual changes of various monitoring data of the link node before and after the optimization, calibrate the influence coefficient of the key factors on the comprehensive transmission quality of the current link node.

2. A data link optimization method for big data according to claim 1, characterized in that: Obtain monitoring data of all link nodes within the scope of big data service operation and transmit the acquired monitoring data to the tuning center. The specific steps include the following: Generate a topology diagram of the transmission link according to the actual deployment position and link relationship of each link node; Configure monitoring data items of different link nodes in the topology diagram and select monitoring mode; Based on the selected monitoring mode, the monitoring data on each link node is captured to the tuning center.

3. The data link optimization method for big data according to claim 1, characterized in that: The tuning center calculates the comprehensive transmission quality of each link node according to the focus target of data link detection, which includes the following sub-steps: Configure the focus target of data link detection for link nodes; Encapsulate value acquisition functions for each focus target; The comprehensive transmission quality of each link node is calculated based on the encapsulated value function.

4. The data link optimization method for big data according to claim 1, characterized in that: Conduct in-depth analysis of link nodes whose comprehensive transmission quality is below the standard, including the following sub-steps: Add link nodes whose comprehensive transmission quality is lower than the standard value to the identification queue; Traverse the identification queue. If the current link node is in an outage state, transfer the transmission service of the current node to the nearest backup node. After the staff releases the outage state, transfer the transmission service back to the current link node. If the current link node is in operation, the value sequence of the comprehensive transmission quality of the current link node in the most recent identification cycle is extracted and input into the key factor identification model, and the key factors affecting the transmission quality of the node are output.

5. The data link optimization method for big data according to claim 4 is characterized in that: The training process of the key factor identification model includes the following sub-steps: Organize the historical diagnostic records of link nodes into a training data set; Use the training data set to train the key factor identification model and determine the model parameters; Verify the impact of input samples of different lengths on model accuracy and determine the recognition cycle of the model.

6. The data link optimization method for big data according to claim 1, characterized in that: The intelligent optimization algorithm of the link node is used to calculate the influence coefficient of the key factors on the comprehensive transmission quality of the current link node. According to the calculation results, the link nodes with substandard comprehensive transmission quality are optimized in a targeted manner. The specific steps include the following: Query the node monitoring data from the historical monitoring records, except for the key factors, whose other factors are consistent with the link node to be optimized; According to the queried monitoring data, calculate the influence coefficient of key factors on the comprehensive transmission quality of the current link node; The optimal solution of the key factor on the current link node is estimated based on the calculation result of the influence coefficient, and the value of the key factor is updated as the optimal solution.

7. A data link optimization system for big data, characterized in that: include: Link node monitoring module and tuning center; The link node monitoring module is used to obtain the monitoring data of all link nodes within the operation scope of the big data service and transmit the obtained monitoring data to the tuning center; The tuning center is used to optimize link nodes whose comprehensive transmission quality does not meet the standards based on the monitoring data obtained.

8. The data link optimization system for big data according to claim 7, characterized in that: The link node monitoring module includes: The transmission link topology map creation submodule is used to generate a transmission link topology map according to the actual deployment location and link relationship of each link node; The monitoring configuration submodule is used to configure the monitoring data items of different link nodes in the topology diagram and select the monitoring mode; The monitoring data acquisition and transmission submodule is used to capture the monitoring data on each link node to the tuning center based on the selected monitoring mode.

9. The data link optimization system for big data according to claim 7, characterized in that: The Tuning Center includes: A comprehensive transmission quality calculation module is used to calculate the comprehensive transmission quality of each link node according to the focus target of data link detection; The link node analysis and switching module is used to conduct in-depth analysis of link nodes with comprehensive transmission quality below the standard, identify the key factors affecting the transmission quality of the node, and enable the standby node to inherit the transmission service of the out-of-service link node when the link node is out of service; The link node optimization module is used to use the intelligent optimization algorithm of the link node to calculate the influence coefficient of the key factors on the comprehensive transmission quality of the current link node, and optimize the link node whose comprehensive transmission quality does not meet the standard according to the calculation results; The optimization parameter calibration module is used to calibrate the influence coefficient of key factors on the comprehensive transmission quality of the current link node according to the actual changes of various monitoring data of the link node before and after the optimization.

10. A computer storage medium, characterized in that: include: at least one memory and at least one processor; A memory for storing one or more program instructions; A processor, used to run one or more program instructions to execute a data link optimization method for big data as described in any one of claims 1-9.

Images