Terminal identification active and passive scheduling method and system based on reinforcement learning algorithm

Abstract

The invention relates to the field of electric digital data processing, in particular to a terminal identification active and passive scheduling method and system based on a reinforcement learning algorithm, and the method comprises the following steps: passively monitoring network flow, obtaining basic terminal information, and generating a terminal list; actively scanning the terminals in the terminal list, and outputting related data; based on the related data, optimizing the scanning execution strategy through a reinforcement learning algorithm, and outputting the optimized scanning execution strategy; and adjusting the scanning frequency and the scanning intensity of active scanning, and performing active scanning on the terminals in the terminal list. According to the method, the advantages of active scanning and passive monitoring are utilized, and through continuous optimization of reinforcement learning, self-adaptive adjustment of the load of the industrial control terminal and the load of the network where the terminal is located is achieved. The scanning mode combining active scanning and passive monitoring not only can enhance the real-time performance of information acquisition, but also can improve the accuracy and comprehensiveness of terminal identification through mutual evidence and supplement of the acquired information.

Description

Technical Field

[0001] The present invention relates to the field of electronic digital data processing, and in particular to a method and system for active and passive scheduling of terminal identification based on a reinforcement learning algorithm. Background Art

[0002] As industrial information security continues to receive high attention and close concern from all sectors of society, enterprises urgently need to sort out network assets within the enterprise when responding to network security inspections and carrying out their own network security construction. In many cases, the configuration of network terminals is constantly changing, and enterprises cannot record the status of industrial control terminals well. At this time, it is necessary to conduct a comprehensive scan of the industrial control terminals in the network.

[0003] However, in the process of network scanning and identification of industrial control terminals, due to different network environments and performance differences of industrial control terminals, frequent scanning may cause unnecessary burden and impact on them. And too rare scanning cannot meet the demand for real-time information acquisition. In order to meet this challenge, it has become particularly urgent to find an effective solution.

[0004] The current mainstream identification methods usually adopt active identification or passive identification, which is based on network scanning tools or network traffic monitoring. However, the impact of these methods on industrial control terminals is different. Active identification methods usually require frequent active requests or scanning of industrial control terminals, which will occupy the computing resources, network bandwidth and storage space of industrial control terminals. Especially for some old industrial control terminals, frequent active scanning may increase the load of industrial control terminals, resulting in performance degradation or delay. In contrast, the real-time performance of passive identification methods may be subject to certain limitations. Since passive identification is based on network traffic analysis, the information related to the attributes of the industrial control terminal to be obtained is rarely carried in normal working communications. If there is no active request, it is generally only available when the industrial control terminal is just connected to the network or when establishing communication with other industrial control terminals. If there is no relevant traffic generated, it may be impossible to obtain relevant information. Summary of the invention

[0005] In order to alleviate the contradiction that active identification increases the load of the industrial control terminal and passive identification obtains terminal information incompletely and timely during the network scanning and identification process of the industrial control terminal, the present invention provides an active and passive scheduling method for terminal identification based on a reinforcement learning algorithm.

[0006] The present invention provides a terminal identification active and passive scheduling method based on a reinforcement learning algorithm, which adopts the following technical solutions:

[0007] A terminal identification active and passive scheduling method based on a reinforcement learning algorithm comprises the following steps:

[0008] Passively monitor network traffic, obtain basic terminal information, divide the basic terminal information into detailed terminal information and incomplete terminal information, and generate a terminal list based on the incomplete terminal information;

[0009] Call the scan execution strategy to actively scan the terminals in the terminal list, and output relevant data including the effect of the scan strategy;

[0010] Based on relevant data, the scanning execution strategy is optimized through the reinforcement learning algorithm, and the optimized scanning execution strategy is output;

[0011] According to the current network status and the optimized scanning execution strategy, the scanning frequency and scanning intensity of the active scan are adjusted to actively scan the terminals in the terminal list.

[0012] In a specific feasible implementation scheme, after actively scanning the terminals in the terminal list, the incomplete terminal information is completed to generate detailed terminal information, and the detailed terminal information obtained by active scanning and the detailed terminal information obtained by passive monitoring are integrated and output.

[0013] In a specific implementation scheme, the method for optimizing the scanning execution strategy using a reinforcement learning algorithm includes the following steps:

[0014] Define the state;

[0015] Establish a mapping between state space and terminal;

[0016] Define actions;

[0017] Define rewards and construct reward functions;

[0018] According to the calculation result of the reward function, prediction is made through the machine learning algorithm, and the actions in the scanning execution strategy are optimized to obtain the optimized scanning execution strategy.

[0019] In a specific implementation scheme, the method for establishing a mapping between a state space and a terminal comprises the following steps:

[0020] The IP address and / or MAC address is used as the unique identifier of the terminal, and the IP address and / or MAC address is not added to the state space;

[0021] Each IP address and / or MAC address corresponds to a machine learning model;

[0022] The factors in the state space are mapped to the scan execution strategy through machine learning algorithms.

[0023] In a specific implementation scheme, the reward function R(S, A) is defined as: R(S, A) = ω1·E(S, A) + ω2·N(S, A) + ω3·U(S, A);

[0024] Where E(S,A) represents the scanning efficiency after taking action A in state S;

[0025] N(S,A) represents the impact of taking action A on network stability in state S;

[0026] U(S,A) represents the impact of taking action A on the terminal resource usage in state S;

[0027] ω1, ω2 and ω3 are all weight coefficients.

[0028] In a specific feasible implementation, the machine learning algorithm is a Q-Learning algorithm, which calculates the Q value of each action in each state according to the calculation result of the reward function and generates a Q value table.

[0029] In a specific feasible implementation, when calculating the Q value, an ε-greedy strategy is adopted, and when the ε value of the ε-greedy strategy decreases, an exponential decay method is adopted.

[0030] In a specific possible implementation scheme, when defining the state, the load parameters are divided into intervals and represented by different parameters to discretize the state.

[0031] The present invention also provides a terminal identification active and passive scheduling system based on a reinforcement learning algorithm, which adopts the following technical solution:

[0032] A terminal identification active and passive scheduling system based on a reinforcement learning algorithm, used to execute the above-mentioned terminal identification active and passive scheduling method based on a reinforcement learning algorithm, comprising:

[0033] The passive monitoring module analyzes the network traffic, divides the detailed terminal information and the incomplete terminal information, generates and outputs the terminal list based on the incomplete terminal information, and outputs the detailed terminal information;

[0034] Active scanning module, obtains the terminal list, actively scans the terminals in the terminal list according to the scanning parameters, and outputs detailed terminal information and related data;

[0035] The reinforcement learning module obtains relevant data, optimizes the scanning execution strategy through the reinforcement learning algorithm, and outputs the optimized scanning execution strategy;

[0036] The scanning scheduling module obtains the optimized scanning execution strategy, adjusts the scanning parameters and outputs them to the active scanning module for execution;

[0037] The result integration and feedback module obtains and integrates the detailed terminal information output by the active scanning module and the passive scanning module, and outputs the integrated data.

[0038] In summary, the present invention includes at least one of the following beneficial technical effects:

[0039] 1. It can dynamically adjust the intensity and frequency of active scanning according to the real-time status of the network environment, thereby achieving the best balance between scanning efficiency and network stability. Through the introduction of multi-objective optimization and adaptive learning rate, this method can not only optimize short-term scanning tasks in real time, but also maintain the stability of the network and terminals in the long term.

[0040] 2. First, through passive monitoring, deep terminal information is mined to reduce the impact on the network, and then, when necessary, a layered active scanning strategy is used to further detect specific terminals. In particular, through the dynamic feedback mechanism of the reinforcement learning algorithm, the accuracy and completeness of terminal information can be ensured while minimizing network load.

[0041] 3. The model is established based on the IP and MAC addresses of the terminal, and the historical behavior patterns of the terminal are also introduced for modeling. At the same time, an adaptive model update mechanism is adopted so that the model can respond to changes in the terminal status in real time and maintain high-precision prediction and scheduling capabilities.

[0042] 4. Q-Learning can automatically optimize the recognition strategy according to the terminal status and network conditions to improve scanning efficiency. When the demand for information acquisition is high, the scanning frequency can be appropriately increased, and when the terminal load is high, the frequency is automatically reduced to avoid affecting the normal operation of the terminal. This intelligent scheduling mechanism makes the recognition process more efficient without affecting the overall efficiency of the system due to overly frequent scanning or overly sparse scanning.

[0043] 5. By fusing the data from active scanning and passive monitoring, Q-Learning can take advantage of the advantages of both methods, reduce their respective limitations, and improve the accuracy and reliability of recognition. A single recognition method may not provide complete information in some cases or may be susceptible to false positives, while data fusion can provide more comprehensive terminal status information. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flow chart of the active and passive scheduling method of terminal identification based on the reinforcement learning algorithm.

[0045] Figure 2 It is the learning flow chart of the enhanced learning module. DETAILED DESCRIPTION

[0046] The following is combined with Figure 1-2 The present invention is described in further detail.

[0047] To facilitate understanding, some terms that appear in the following text are explained first.

[0048] Active scanning: refers to actively detecting or scanning terminals through the network management system, sending different requests according to the different services enabled by the terminals, obtaining responses and parsing the responses to obtain the desired data, including terminal information, terminal load, network load, etc.

[0049] Passive monitoring: Capturing and analyzing real-time network traffic, or traffic statistics, at a specific location within the network, such as where the network connects to an application server. As the name implies, passive monitoring does not require the inclusion or even knowledge of another point within the network. In simple terms, passive monitoring is the periodic collection of port statistics, such as the number of bytes and packets transmitted and received. More commonly, passive monitoring involves capturing some or all of the traffic passing through a port for detailed, non-real-time analysis, such as signaling protocols, application usage, or the applications that are consuming the most bandwidth.

[0050] Reference Figure 1 The active and passive scheduling method for terminal identification based on the reinforcement learning algorithm includes the following steps:

[0051] S100, passively monitor network traffic, obtain basic terminal information, and divide the basic terminal information into detailed terminal information and incomplete terminal information according to the completeness of the basic terminal information. Active scanning is used to complete the missing item data in the incomplete terminal information.

[0052] Passive monitoring of network traffic. Since the terminals in the industrial control network will open dedicated communication ports and use specific industrial control protocols for communication during communication, during the process of passive monitoring of network traffic, the traffic packets in the terminal communication process can be obtained, and the traffic packets will contain basic terminal information; when the traffic packets do not contain basic terminal information, basic terminal information can also be inferred through communication behavior. Basic terminal information includes IP address, MAC address, communication port and the protocol used, CPU usage, memory usage, bandwidth usage, etc.

[0053] After a period of passive monitoring, some items in the basic terminal information of some terminals are missing data, or the basic terminal information of some terminals is not obtained. These basic terminal information are divided into incomplete terminal information, the terminal names are recorded, and a terminal list is generated. The basic terminal information is completely obtained and divided into detailed terminal information for subsequent integration.

[0054] According to the terminal name in the terminal list, the scanning execution strategy is called, and the communication port used by the terminal obtained through passive monitoring is selected to perform low-frequency and low-intensity active scanning: a data packet of a specific protocol is sent to the terminal, the required data is actively requested, and the data packet returned by the terminal is parsed; or the required data is obtained from the terminal by crawlers and other methods, the incomplete terminal information is completed, and it is re-divided into detailed terminal information for subsequent integration.

[0055] S200, detailed terminal information integration and overall network status assessment.

[0056] Obtain detailed terminal information obtained through passive monitoring and detailed terminal information obtained through active scanning, and merge the two sets of detailed terminal information to obtain more comprehensive and accurate terminal information. Evaluate the terminal load and the overall network status based on the terminal information to obtain the evaluation results.

[0057] The terminal information and evaluation results are output to the database, or visual charts are generated for visual display, so as to facilitate network management and terminal optimization based on the terminal information and evaluation results.

[0058] S300, optimizes the active scanning execution strategy through a reinforcement learning algorithm.

[0059] S310, define status.

[0060] The reinforcement learning algorithm specifically uses the Q-Learning algorithm learning model. It is easy to understand that in addition to the Q-Learning algorithm, it can also be a decision tree, random forest and other machine learning algorithm models.

[0061] The state S is defined by the industrial control terminal CPU usage C, the industrial control terminal memory usage M and the network bandwidth usage N, S = (C, M, N). Among them, the load parameters of the industrial control terminal CPU usage C, the industrial control terminal memory usage M and the network bandwidth usage N are divided into limited intervals and represented by 0, 1, 2, 3, specifically:

[0062] 0-20%: low load, indicated by 0;

[0063] 21%-50%: medium load, represented by 1;

[0064] 51%-80%: high load, indicated by 2;

[0065] 81%-100%: Very high load, indicated by 3;

[0066] For example, S=(1,2,2) means that in this state, the CPU usage C of the industrial control terminal is a medium load, and the memory usage M and network bandwidth usage N of the industrial control terminal are both high loads.

[0067] By defining the state S and dividing the load parameters into intervals, the state can be discretized, the state space can be reduced, and the feasibility and efficiency of subsequent data processing can be improved. The state space is a collection of states S. If the load parameters are directly used to define the state, due to the high numerical accuracy of the percentage, noise and small fluctuations will make it difficult for the subsequent model to converge. By dividing the load parameters into intervals, the number of states is greatly reduced, and the convergence efficiency of the subsequent model is greatly improved.

[0068] S320, establishing a mapping.

[0069] Establish a mapping from the unique terminal identifier to the learning model: In the basic terminal information obtained in step S100, the IP address and MAC address will not be affected by the scanning intensity and scanning frequency. Therefore, the IP address and MAC address are used as the unique identifier of the terminal. Each IP address and MAC address corresponds to a learning model, that is, a separate learning model is used for training each terminal, which avoids the problem of improper scanning strategy scheduling caused by multiple terminals with different sensitivities to active scanning intensity and scanning frequency in the network sharing a learning model, making it more accurate and effective when optimizing the scanning execution strategy.

[0070] It is also possible to exclude IP addresses and MAC addresses from the state space, so that the variables in the state space focus on dynamic factors that affect the scan execution strategy, thereby further reducing the dimension of the state space, reducing the complexity of training, and improving the efficiency of model training.

[0071] Establish a mapping from state space to scan execution strategy: Through the Q-Learning algorithm, the dynamic factors in the state space are mapped to the optimal scan execution strategy to maximize active scanning efficiency and environmental stability.

[0072] By treating each terminal as an independent learning entity and each terminal corresponding to a learning model, the learning model is trained through the state space corresponding to the IP address and MAC address, and the optimal scanning execution strategy is output, thus realizing the construction of the mapping between the state space and the terminal.

[0073] S330, define an action.

[0074] The action identifier adjusts the action in the scan execution strategy, and in this embodiment, specifically changes the scan intensity and frequency. The combination of different scan intensities G and scan frequencies F constitutes action A, A = {(F1, G1), (G2, G2), (F3, G3), ...}.

[0075] The scanning intensity G is divided into three levels: low, medium, and high according to the scanning interval between different terminals and the number of concurrent scans. The shorter the scanning interval between different terminals and the more scans there are, the higher the scanning intensity G. The scanning frequency G is divided into three levels: low, medium, and high according to the interval between multiple scans of the same terminal. The shorter the interval, the higher the scanning frequency F.

[0076] S340, define reward.

[0077] The reward indicates the quality of the action. The reward function directly affects the learning process of the Q-Learning algorithm and the effect of the output scanning execution strategy. The reward function reflects the goal of balancing scanning efficiency and network environment stability. The reward function is designed with scanning efficiency, network stability and terminal resource usage as constraints.

[0078] Scanning efficiency refers to the amount of effective information obtained by active scanning; network stability refers to network bandwidth usage; and terminal resource usage refers to the CPU usage and memory usage of the terminal. The goal is to obtain as much effective information as possible while maintaining network stability and avoiding excessive scanning that consumes terminal resources.

[0079] The reward function R(S,A) is defined as: R(S,A)=ω1·E(S,A)+ω2·N(S,A)+ω3·U(S,A); where E(S,A) represents the scanning efficiency after taking action A in state S; N(S,A) represents the impact of taking action A on network stability in state S; U(S,A) represents the impact of taking action A on terminal resource usage in state S; ω1, ω2 and ω3 are all weight coefficients.

[0080] Different on-site environments place different emphasis on scanning efficiency, network load and terminal load. Therefore, when calculating the reward function R(S,A) in the face of different requirements, the weight coefficients ω1, ω2 and ω3 are preset and adjusted according to changes in constraints.

[0081] Specifically, the scanning efficiency E of each terminal is single It is expressed as: In the formula, n represents the amount of effective information successfully obtained, and T represents the time spent on active scanning. threshold Construct a reward function for scanning efficiency:

[0082]

[0083] In the formula, α1 and β1 are both positive proportional coefficients used to adjust the intensity of rewards and penalties. They can be both 1 or adjusted according to actual needs. single Greater than or equal to the preset efficiency threshold E threshold , that is, Esingle ≥E threshold When , it indicates that the active scanning efficiency is high, and a positive reward is given; when the terminal's scanning efficiency E single Less than the preset efficiency threshold E threshold , that is, E single <E threshold When , it means that the active scanning efficiency is low and a penalty is given.

[0084] The reward function for network stability is: N(S,A)=R ΔB (ΔB)+R H (σ B ), where ΔB is the change in bandwidth occupancy, which represents the change in the percentage of network bandwidth occupancy within the time window; R ΔB (ΔB) represents the reward or penalty for the change in bandwidth occupancy; H is the load stability, which represents the load volatility within the time window, and the standard deviation of bandwidth occupancy σ is used. B To express, H = σ B ; R H (σ B ) represents the reward or penalty for load stability.

[0085]

[0086] In the above formula, ΔB threshold is the preset change threshold, and β2 is a positive proportional coefficient used to adjust the intensity of rewards and penalties. threshold When ΔB<ΔB, it means that the active scanning causes a high network load and a penalty is given; threshold , it indicates that the network load caused by active scanning is normal, and no reward or penalty is given.

[0087]

[0088] In the above formula, σ thresholf is the preset standard deviation threshold, α2 and γ are positive proportional coefficients used to adjust the intensity of rewards and penalties. B ≤σ threshold When σ , it means that the network load does not change much, and a positive reward is given. The size of the reward is inversely proportional to the degree of fluctuation of the network load. B >σ threshold When , it means that the network load changes too much, and a penalty is given. The size of the penalty is proportional to the standard deviation threshold σ threshold Portion size is proportional.

[0089] The reward function for terminal resource usage is: U(S,A)=R cpu (ΔU cpu )+R mem (ΔU mem )

[0090] In the above formula, ΔU cpu is the change in CPU usage, R cpu (ΔU cpu ) represents the reward function of CPU usage; ΔU mem is the change in memory usage, R mem (ΔU mem ) represents the reward function of memory usage.

[0091]

[0092] In the above formula, α3 and β3 are both positive proportional coefficients used to adjust the intensity of rewards and penalties. The larger the value, the heavier the penalty.

[0093] It should be noted that there is no sequence between steps S310 - 340 , and the step numbers are only used to distinguish the steps.

[0094] S400, recording the Q value of each action a in each state s in a Q value table.

[0095] The Q value is calculated by the reward function R(S,A) and is used to evaluate the effect of each action a in each state s. The Q value table records each action a in each state s and the corresponding Q value, so that the best action can be selected in different states, and then the value in the Q value table can be updated, and the optimal scanning execution strategy can be gradually learned. The Q value table is shown in Table 1 below.

[0096] Table 1

[0097] Status / Action <![CDATA[a1]]> <![CDATA[a2]]> … <![CDATA[a y <!-- 6 -->]]> <![CDATA[s1]]> <![CDATA[Q(s1,a1)]]> <![CDATA[Q(s1,a2)]]> … <![CDATA[Q(s1,a y )]]> <![CDATA[s2]]> <![CDATA[Q(s2,a1)]]> <![CDATA[Q(s2,a2)]]> … <![CDATA[Q(s2,a y )]]> … … … … … <![CDATA[s x ]]> <![CDATA[Q(s x ,a1)]]> <h2 style=";text-align:left;direction:ltr"><![CDATA[Q(s <h2 style=";text-align:left;direction:ltr"> x <h2 style=";text-align:left;direction:ltr"> ,a2)]]><h2 style=";text-align:left;direction:ltr"> … <![CDATA[Q(s x ,a y )]]>

[0098] Initialize all Q values ​​in the Q value table so that the Q values ​​are all random numbers between 0 and 1. Use the ε-greedy strategy to balance exploration and exploitation, that is, randomly select actions, explore new actions and rewards, avoid local optimality, and select the action with the largest Q value based on the data in the Q value table.

[0099] In the initial stage, a preset larger exploration probability, i.e., the ε value, is obtained, and the ε value is gradually reduced by exponential decay until it decays to close to or equal to 0. For ease of understanding, ε = 0.9 is used as an example for further explanation. When ε = 0.9, it indicates that in the current state, there is a 90% probability of randomly selecting an action and a 10% probability of selecting the action with the largest Q value in the Q value table. By presetting a larger ε value at the beginning of the algorithm, more exploration can be performed during the ε value decay process, so that the algorithm can make more use of the learned content and avoid falling into the local optimum.

[0100] The calculation formula for the exponential decay of the ε value is: ε(t) = ε end +(εstart -ε end )·e -λt ,

[0101] In the above formula, ε(t) represents the ε value at time t;

[0102] ε start represents the initial ε value;

[0103] ε end represents the final ε value;

[0104] λ is the decay rate, which is used to control the decay speed of the ε value. The larger λ is, the faster the ε value decreases.

[0105] e is the base of natural logarithms.

[0106] Calculate the Q value corresponding to the current action in the current state and update the Q value in the Q value table. The calculation formula of Q value is:

[0107] Q(S,A)←Q(S,A)+θ[R(s,a)+μmax a′ Q(s′,a′)-Q(s,a)]

[0108] In the above formula, Q(S,A) represents the Q value corresponding to the current state s and action a;

[0109] θ is the learning rate, 0<θ≤1, which controls the step size of each update;

[0110] R(s,a) represents the immediate reward obtained by executing action a in state s;

[0111] μ is the discount factor, 0<μ≤1, used to balance current rewards and future rewards;

[0112] s′ is the new state obtained after executing action a;

[0113] a′ is a new action that may be performed in the new state s′;

[0114] max a′ Q(s′,a′) is the maximum Q value of all possible actions a′ in the new state s′.

[0115] S500, adjusting the task scheduling strategy according to the Q value table, outputting the optimal scan execution strategy, and scheduling active scanning.

[0116] According to the obtained detailed terminal information, the Q value table is queried to obtain the Q value of all actions in the current state, and the action with the highest Q value is selected. The action with the highest Q value is the optimal scheduling strategy in the current state, that is, the optimal scanning execution strategy. After each scheduling, the scanning intensity and scanning frequency of the next active scan are dynamically adjusted according to the returned system load and network environment changes, so as to realize adaptive adjustment of the load of the industrial control terminal and the load of the network where the terminal is located, so as to achieve optimal resource utilization, coordinate the contradiction between active scanning and passive monitoring, and reduce the impact of active scanning on the network and the scanned terminal. Furthermore, the scanning method combining active scanning and passive identification monitoring can not only enhance the real-time nature of information acquisition, but also improve the accuracy and comprehensiveness of terminal identification through mutual corroboration and supplementation of the acquired information.

[0117] The present invention also discloses a terminal identification active and passive scheduling system based on a reinforcement learning algorithm, comprising:

[0118] The passive monitoring module obtains network traffic, parses the network traffic, obtains basic terminal information and divides it into detailed terminal information and incomplete terminal information, and generates a terminal list based on the incomplete terminal information.

[0119] The active scanning module obtains the terminal list and actively scans the terminals in the terminal list. If the first active scan is performed, the scanning parameters use the default values; otherwise, the scanning parameters output by the scanning scheduling module are obtained. The output completes the data and includes detailed terminal information, active scanning effects, and other data related to scanning efficiency and strategy, as well as bandwidth usage.

[0120] The result integration and feedback module obtains the detailed terminal information of the passive monitoring module and the detailed terminal information output by the active scanning module, merges the two sets of detailed terminal information, and generates charts for visual output. It outputs detailed terminal information and network bandwidth usage data to the reinforcement learning module.

[0121] The reinforcement learning module obtains the relevant data and bandwidth usage output by the active scanning module, as well as the detailed terminal information in the result integration and feedback module, optimizes the scanning execution strategy based on the Q-Learning algorithm, and outputs the optimized and optimal scanning execution strategy.

[0122] The scanning scheduling module obtains the scanning execution strategy output by the reinforcement learning module, adjusts the scanning parameters including the scanning frequency and scanning intensity according to the current network environment and terminal status, and the scanning execution strategy, and outputs the scanning parameters to the active scanning module.

[0123] The above are all preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A terminal identification active and passive scheduling method based on a reinforcement learning algorithm, characterized in that: The steps include: Passively monitor network traffic, obtain basic terminal information, divide the basic terminal information into detailed terminal information and incomplete terminal information, and generate a terminal list based on the incomplete terminal information; Call the scan execution strategy to actively scan the terminals in the terminal list, and output relevant data including the effect of the scan strategy; Based on relevant data, the scanning execution strategy is optimized through the reinforcement learning algorithm, and the optimized scanning execution strategy is output; According to the current network status and the optimized scanning execution strategy, the scanning frequency and scanning intensity of the active scan are adjusted to actively scan the terminals in the terminal list.

2. The method for active and passive scheduling of terminal identification based on reinforcement learning algorithm according to claim 1 is characterized in that: After actively scanning the terminals in the terminal list, the incomplete terminal information is completed to generate detailed terminal information. The detailed terminal information obtained by active scanning and the detailed terminal information obtained by passive monitoring are integrated and output.

3. The method for active and passive scheduling of terminal identification based on reinforcement learning algorithm according to claim 1, characterized in that: The method for optimizing the scanning execution strategy by the reinforcement learning algorithm includes the following steps: Define the state; Establish a mapping between state space and terminal; Define actions; Define rewards and construct reward functions; According to the calculation result of the reward function, prediction is made through the machine learning algorithm, and the actions in the scanning execution strategy are optimized to obtain the optimized scanning execution strategy.

4. The method for active and passive scheduling of terminal identification based on reinforcement learning algorithm according to claim 3 is characterized in that: The method for establishing the mapping between the state space and the terminal comprises the following steps: The IP address and / or MAC address is used as the unique identifier of the terminal, and the IP address and / or MAC address is not added to the state space; Each IP address and / or MAC address corresponds to a machine learning model; The factors in the state space are mapped to the scan execution strategy through machine learning algorithms.

5. The method for active and passive scheduling of terminal identification based on reinforcement learning algorithm according to claim 3 is characterized in that: The reward function R(S,A) is defined as: R(S,A) = ω1·E(S,A) + ω2·N(S,A) + ω3·U(S,A); Where E(S,A) represents the scanning efficiency after taking action A in state S; N(S,A) represents the impact of taking action A on network stability in state S; U(S,A) represents the impact of taking action A on the terminal resource usage in state S; ω1, ω2 and ω3 are all weight coefficients.

6. The method for active and passive scheduling of terminal identification based on reinforcement learning algorithm according to claim 3 is characterized in that: The machine learning algorithm is the Q-Learning algorithm, which calculates the Q value of each action in each state based on the calculation result of the reward function and generates a Q value table.

7. The method for active and passive scheduling of terminal identification based on reinforcement learning algorithm according to claim 6 is characterized in that: When calculating the Q value, the ε-greedy strategy is used, and an exponential decay method is used when the ε value of the ε-greedy strategy decreases.

8. The method for active and passive scheduling of terminal identification based on reinforcement learning algorithm according to claim 3 is characterized in that: When defining the state, the load parameters are divided into intervals and represented by different parameters to discretize the state.

9. A terminal identification active and passive scheduling system based on reinforcement learning algorithm, characterized in that: A method for performing active and passive scheduling of terminal identification based on a reinforcement learning algorithm as described in any one of claims 1 to 8, comprising: The passive monitoring module analyzes the network traffic, divides the detailed terminal information and the incomplete terminal information, generates and outputs the terminal list based on the incomplete terminal information, and outputs the detailed terminal information; Active scanning module, obtains the terminal list, actively scans the terminals in the terminal list according to the scanning parameters, and outputs detailed terminal information and related data; The reinforcement learning module obtains relevant data, optimizes the scanning execution strategy through the reinforcement learning algorithm, and outputs the optimized scanning execution strategy; The scanning scheduling module obtains the optimized scanning execution strategy, adjusts the scanning parameters and outputs them to the active scanning module for execution; The result integration and feedback module obtains and integrates the detailed terminal information output by the active scanning module and the passive scanning module, and outputs the integrated data.

Images