Method for detecting data applicability of deep reinforcement learning method

The time stationarity and Markov nature of the network data set are detected and enhanced through the DAADRL and EDAADRL methods, which solves the shortcomings of data applicability judgment in network applications, and ensures the effectiveness and scope of optimization decisions.

CN119026665APending Publication Date: 2024-11-26NANJING UNIV OF AERONAUTICS & ASTRONAUTICS
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411080227.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-07
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

When optimizing network applications using deep reinforcement learning (DRL) methods, the existing technology fails to effectively judge and ensure whether the network data set meets mathematical properties such as time stationarity and Markovness, resulting in inappropriate research conclusions and potential adverse consequences .

Method used

DRL data applicability detection method DAADRL and data applicability enhancement method EDAADRL are proposed. By detecting the continuity, stationarity and Markov nature of traffic samples, the data sets are clustered and tested using hierarchical clustering algorithm and DBSCAN algorithm to form A segment that satisfies the applicability of DRL data.

Benefits of technology

Effectively detecting whether the data set is applicable, expanding the scope of application of DRL methods in network applications, ensuring the effectiveness and accuracy of optimization decisions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119026665A_ABST
    Figure CN119026665A_ABST
Patent Text Reader

Abstract

When the DRL method is used for optimizing the network application, a good result can be obtained in prediction and decision-making of the network application only when the used network data is ensured to meet the requirements of mathematical properties such as Markov property, stability, conductivity, continuity and the like. The invention provides a method for defining DRL data applicability and detecting the DRL data applicability so as to judge whether a certain data set can use a DRL method or not; the invention provides a DRL (Data Reporting Language) data applicability enhancement method EDDADRL to find out a section meeting DRL data adaptability in a data set, so that the DRL method can be used in a larger range. The method provided by the invention has the characteristics of simplicity and practicability, and has important practical value and wide application prospect for optimizing network application by using the DRL method.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of network big data, and specifically is a method for detecting data applicability (Detecting Data Applicability for DRL, DAADRL) of a deep reinforcement learning (DRL) method, so as to determine whether a network data set can be used for decision optimization using the DRL method and an algorithm for enhancing the DRL data applicability of the data set (Enhancement Detecting Data Applicability for DRL, EDAADRL). Background Art

[0002] The DRL method combines the characteristics of deep learning and reinforcement learning, and has the advantages of being able to process high-dimensional state and action space, automatically extract features, have potential generalization capabilities, and have high training efficiency. It is highly versatile and adaptable. It has been widely used in the optimization of network applications such as resource allocation and scheduling, network slicing, edge computing, network security, intelligent routing, and the Internet of Things.

[0003] As a machine learning method, DRL has specific requirements for the mathematical properties of the data used, such as Markov property, stationarity, differentiability and continuity. The theoretical framework of reinforcement learning is usually the Markov decision process (MDP). The goal of MDP is to obtain the optimal strategy by maximizing the long-term cumulative reward. When using the DRL method to optimize network applications, only by ensuring that the network data used can meet the requirements of the above mathematical properties, can good results be achieved in the prediction and decision-making of network applications.

[0004] In the process of optimizing network applications using DRL, if the mathematical properties that the network data should possess are ignored, inappropriate research conclusions and even adverse consequences will be caused. The present invention proposes a definition of DRL data applicability and a method for detecting DRL data applicability to determine whether a certain data set can use the DRL method; a DRL data applicability enhancement method EDDADRL is proposed to find out the segments in the data set that meet the DRL data adaptability, so that the DRL method can be used in a larger range. The present invention has important practical value and broad application prospects for optimizing network applications using the DRL method. Summary of the invention

[0005] [Purpose of the invention]:

[0006] When using the DRL method to optimize network applications, the network data used is required to have characteristics such as time stability and Markov properties. The present invention proposes a definition of DRL data applicability and a method for detecting DRL data applicability to determine whether the network data set is applicable to the DRL method; a DRL data applicability enhancement method EDDADRL is proposed to find out the segments in the data set that meet the DRL data adaptability, so that the DRL method can be used in a larger range.

[0007] [Technical solution]:

[0008] The technical solution proposed by the present invention is:

[0009] 1. The concept of DRL data suitability of a dataset and a method for detecting DRL data suitability are proposed, which are characterized by:

[0010] A. Data applicability for DRL refers to the fact that when using the DRL method to analyze and optimize network applications, the network data flow can satisfy the characteristics of Markov and stationarity in a continuous period of time.

[0011] B. To test the continuity of f(X), we can test whether the timestamps of the traffic samples are continuous. To test the stationarity of f(X), we can use the statsmodel package in Python to perform an ADF test. If p-value < 0.05, f(X) is stationary, otherwise it is not stationary. To test the Markov property of f(X), we first use a hierarchical clustering algorithm to divide the sequence f(X) into m states and construct an m×m transition frequency matrix M, M ab It represents the frequency of state a transferring to state b after one step, a∈N, b∈N, a≤m, b≤m. The marginal probability calculation formula is

[0012]

[0013] For the confidence level α, the quantile can be obtained by looking up the table The statistic is subject to degrees of freedom (m-1) 2 The χ2 distribution is calculated as

[0014]

[0015] like Then f(X) satisfies the Markov property.

[0016] C. A Detecting Data Applicability for DRL (DAADRL) algorithm. The pseudo code of the DAADRL algorithm is shown below.

[0017]

[0018] 2. A DRL data applicability enhancement algorithm EDAADRL is proposed, which is characterized by:

[0019] A. The working process of EDAADRL: First, for the input data set, the network flow samples are clustered based on the DBSCAN algorithm with the sample timestamp and numerical size as the distance; secondly, the DAADRL algorithm is used to perform continuity, Markov and stationarity tests on each cluster obtained from the previous clustering. If the test result meets the conditions, the segment corresponding to the cluster is a segment that meets the DRL data suitability; thirdly, a new data set that meets the DRL data suitability is formed by each segment that meets the DRL data suitability.

[0020] B. The pseudo code of the EDAADRL algorithm is as follows:

[0021]

[0022] [Beneficial Effects]: The significance of the present invention is that, in order to obtain useful results by using the DRL method to optimize network applications, the DAADRL algorithm can be used to detect whether a data set has the required properties, so that DRL can be used to optimize decision-making network applications; the enhanced DRL data applicability algorithm EDAADRL is used to search for segments in the data set that meet the DRL data adaptability, thereby expanding the scope of application of DRL to optimize network applications. These methods are simple and practical. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 EDDADRL processing TEST1 clustering results DETAILED DESCRIPTION

[0024] The application of the present invention is described in detail below with reference to the accompanying drawings and specific examples.

[0025] Dataset All satisfy the Markov property and stationarity. Take out 50 consecutive groups of data from each to form a data set

[0026] Use DAADRL algorithm to test the dataset and TEST1, after data verification, It satisfies continuity, Markov and stationarity, while TEST1 does not satisfy the above properties. It can be seen that the DAADRL algorithm can detect the properties of the data set, and connecting the data sets that meet the Markov and stationarity characteristics of different time periods will cause the network data set to no longer have stationary characteristics and Markov properties.

[0027] The EDAADRL algorithm is used to process the data set TEST1. Figure 1 The clustering results of TEST1 processed by the EDAADRL algorithm are given. It can be seen that the entire data set is divided into 5 clusters, and the timestamps of the samples in these 5 clusters correspond to the corresponding time segments of the data set TEST1. The samples of each cluster meet the characteristics of continuity, stationarity and Markov. The DRL method can be used for clusters that meet the applicability of DRL data. It can be seen that the EDAADRL algorithm can effectively identify data segments suitable for DRL, thereby expanding the data applicability of DRL optimization network applications.

Claims

1. The concept of DRL data suitability of a dataset and a method for detecting DRL data suitability are proposed, which are characterized by: A. Data applicability for DRL refers to the fact that when using the DRL method to analyze and optimize network applications, the network data flow can satisfy the characteristics of Markov and stationarity in a continuous period of time. B. To test the continuity of f(X), we can test whether the timestamps of the traffic samples are continuous. To test the stationarity of f(X), we can use the statsmodel package in Python to perform an ADF test. If p-value < 0.05, f(X) is stationary, otherwise it is not stationary. To test the Markov property of f(X), we first use a hierarchical clustering algorithm to divide the sequence f(X) into m states and construct an m×m transition frequency matrix M, M ab It represents the frequency of state a transferring to state b after one step, a∈N, b∈Na≤m, b≤m. The marginal probability calculation formula is For the confidence level α, the quantile can be obtained by looking up the table The statistic is subject to degrees of freedom (m-1) 2 χ 2 Distribution, calculated as like Then f(X) satisfies the Markov property. C. A Detecting Data Applicability for DRL (DAADRL) algorithm. The pseudo code of the DAADRL algorithm is shown below.

2. A DRL data applicability enhancement algorithm EDAADRL is proposed, which is characterized by: A. Working process of EDAADRL: First, for the input data set, the network flow samples are clustered based on the DBSCAN algorithm with the sample timestamp and numerical size as the distance; secondly, the DAADRL algorithm is used to perform continuity, Markov and stationarity tests on each cluster obtained by the previous clustering. If the test result meets the conditions, the segment corresponding to the cluster is the segment that meets the suitability of DRL data; Third, a new data set satisfying DRL data applicability is formed from each segment satisfying DRL data applicability. B. The pseudo code of the EDAADRL algorithm is as follows: