Intelligent vulnerability scanning strategy generation method and device based on reinforcement learning

Abstract

The invention provides an intelligent vulnerability scanning strategy generation method and device based on reinforcement learning. According to the embodiment of the invention, the current scanning plug-in selection strategy is dynamically generated based on the real-time perceived equipment feature vector of the target equipment and the historical scanning result of the target equipment through reinforcement learning, so that the dynamically generated scanning plug-in selection strategy is ensured to have relatively high dynamic adaptive capacity, and false alarms and missing alarms are reduced. Furthermore, the embodiment optimizes and adjusts an Actor model in a reinforcement learning model in real time by means of a reward mechanism of reinforcement learning according to the historical performance of different scanning plug-ins and the current scanning result (even user feedback information) of the target equipment, so as to realize continuous optimization of a subsequent scanning plug-in selection strategy, and improve the user experience. And the detection efficiency and accuracy are improved.

Description

Technical Field

[0001] The present application relates to network security, and in particular to a method and device for generating an intelligent vulnerability scanning strategy based on reinforcement learning. Background Art

[0002] In the network security defense system, scanning devices is often used as a core means to actively discover system vulnerabilities. However, scanning is highly dependent on the rationality of the scanning strategy. However, current scanning strategies are often configured manually. And manually configured scanning strategies are difficult to cope with the increasingly complex network attack and defense confrontation environment. Summary of the invention

[0003] The present application provides a method and device for generating an intelligent vulnerability scanning strategy based on reinforcement learning, so as to realize intelligent generation of scanning strategies based on reinforcement learning.

[0004] The embodiment of the present application provides a method for generating an intelligent vulnerability scanning strategy based on reinforcement learning, which is applied to a scanning device and includes: Determine an encoding method for matching each feature data in a device portrait of a target device; the device portrait is generated based on device information of the target device obtained by actively scanning the target device; the device portrait includes feature data corresponding to multiple different feature types, and feature data of at least two different feature types are matched with different encoding methods; According to the encoding method matched with each feature data in the device portrait, each feature data in the device portrait is encoded, and according to the encoding result of each feature data, a device feature vector of the target device is generated; Based on the historical scanning behavior executed on the target device, a reference scanning result is obtained; the reference scanning result at least includes: the scanning situation of each scanning plug-in when scanning the target device, and the scanning situation at least includes: missed abnormal information, false positive, unrecognized abnormal information, and correctly recognized abnormal information; Using the strategy model Actor model in the current reinforcement learning model, based on the device feature vector and the reference scanning result, the potential benefits of each scanning plug-in are evaluated to obtain a scanning plug-in selection strategy; and for each scanning plug-in indicated by the scanning plug-in selection strategy to be used when scanning the target device, according to the resources occupied by the scanning plug-in during execution, the risk of abnormal information that can be scanned by the scanning plug-in, and the most recent time when the scanning plug-in was used, the scanning priority of the scanning plug-in when scanning the target device is currently determined; According to the scanning priority of each scanning plug-in and in accordance with the priority scheduling algorithm, each scanning plug-in is scheduled to scan the target device to obtain the current scanning result of each scanning plug-in; based on the current scanning result of each scanning plug-in and the reward function of each scanning plug-in, the reward value of each scanning plug-in is determined, and the value Critic model in the current reinforcement learning model evaluates the behavioral performance of the scanning plug-in selection strategy based on the reward value of each scanning plug-in, so as to optimize the Actor model according to the evaluation result of the Critic model; the optimized Actor model outputs a better scanning plug-in selection strategy than before optimization.

[0005] The present application also provides a scanning device, which includes: A data perception module, used to determine the encoding method matching each feature data in the device portrait of the target device; the device portrait is generated based on the device information of the target device obtained by actively scanning the target device; the device portrait contains feature data corresponding to multiple different feature types, and the encoding methods matching feature data of at least two different feature types are different; and, used to encode each feature data in the device portrait according to the encoding method matching each feature data in the device portrait, and generate a device feature vector of the target device according to the encoding results of each feature data; A strategy generation module is used to obtain a reference scanning result based on the historical scanning behavior executed on the target device; the reference scanning result at least includes: the scanning situation of each scanning plug-in when scanning the target device, and the scanning situation at least includes: missed abnormal information, false positive, unrecognized abnormal information, and correctly recognized abnormal information; and, using the strategy model Actor model in the current reinforcement learning model, based on the device feature vector and the reference scanning result, evaluate the potential benefits of each scanning plug-in to obtain a scanning plug-in selection strategy; a policy execution module, configured to determine, for each scanning plug-in that is required to be used when scanning the target device currently as indicated by the scanning plug-in selection policy, a scanning priority of the scanning plug-in when scanning the target device currently according to resources occupied by the scanning plug-in when executing, risk of abnormal information that can be scanned by the scanning plug-in, and a recent time when the scanning plug-in was used; and According to the scanning priority of each scanning plug-in and the priority scheduling algorithm, each scanning plug-in is scheduled to scan the target device to obtain the current scanning result of each scanning plug-in; A strategy feedback module, used to determine the reward value of each scanning plug-in based on the current scanning result of each scanning plug-in and the reward function of each scanning plug-in, and feed back the reward value of each scanning plug-in and the current scanning result of each scanning plug-in to the strategy generation module; The strategy generation module is further used to evaluate the behavioral performance of the scanning plug-in selection strategy based on the reward value of each scanning plug-in according to the feedback of the strategy feedback module and using the value Critic model in the current reinforcement learning model, so as to optimize the Actor model according to the evaluation result of the Critic model; the optimized Actor model outputs a better scanning plug-in selection strategy than before optimization.

[0006] It can be seen from the above technical solution that in this application, this embodiment dynamically generates the current scanning plug-in selection strategy through reinforcement learning based on the device feature vector of the target device perceived in real time and the historical scanning results of the target device, so as to ensure that the dynamically generated scanning plug-in selection strategy has strong dynamic adaptability, thereby reducing false alarms and missed alarms.

[0007] Furthermore, this embodiment uses the reward mechanism of reinforcement learning to optimize and adjust the Actor model in the reinforcement learning model in real time according to the historical performance of different scanning plug-ins and the current scanning results of the target device (even with user feedback information), so as to continuously optimize the subsequent scanning plug-in selection strategy and improve detection efficiency and accuracy.

[0008] Furthermore, this embodiment describes the target device with as comprehensive information as possible by profiling the target device, and encodes each feature data in the device portrait according to the encoding method that matches each feature data in the device portrait to generate a device feature vector of the target device. This ensures data planning and management, avoids the conventional convolutional neural network that forces each feature data to be encoded as a vector of equal length according to a unified encoding format, thereby increasing noise such as useless data, and can further ensure that when a scanning strategy is subsequently generated for the target device based on the device feature vector, the accuracy of the strategy is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0010] Figure 1 A flow chart of the method provided in the embodiment of the present application; Figure 2 A schematic diagram of a device feature vector provided in an embodiment of the present application; Figure 3 Provided for the embodiments of this application Figure 1 A schematic diagram of the process shown; Figure 4 A flowchart for implementing step 104 provided in an embodiment of the present application; Figure 5 A structural diagram of a device provided in an embodiment of the present application; Figure 6A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0011] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and understandable, the technical solutions in the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings.

[0012] The method provided in the embodiment of the present application can break through the reliance on manual experience and realize adaptive dynamic generation of scanning plug-in strategies. In addition, the embodiment of the present application will also build a closed-loop feedback system based on the dynamically generated scanning plug-in strategy to optimize the subsequently generated scanning plug-in strategy (indicating which scanning plug-ins can be used to scan the device to be scanned), reduce frequent false alarms, and multiple failures to identify abnormal information, and improve detection efficiency.

[0013] The method provided in the embodiment of the present application is described below: See also Figure 1 , Figure 1 A flowchart of a method provided in an embodiment of the present application. The process can be applied to a scanning device. This embodiment does not specifically limit the deployment location of the scanning device in the network. As an embodiment, the scanning device can be deployed in existing network devices in the network, such as switches, gateways and other network devices, or can be independent of existing network devices in the network, which is not specifically limited in this embodiment.

[0014] When you need to scan the target device, you can enable the scanning device to perform the scanning task on the target device. For details, see Figure 1 The process shown.

[0015] like Figure 1 As shown, the process may include: Step 101, determine the encoding method for matching each feature data in the device portrait of the target device; the device portrait is generated based on the device information of the target device obtained by actively scanning the target device.

[0016] In this embodiment, the device information of the target device is collected by active scanning to generate a device profile of the target device. The device profile of the target device is used to provide a description of the target device as perfect as possible.

[0017] Specifically, the device portrait may include feature data corresponding to multiple different feature types, such as port type, service version type, device description type, operating system type, etc.

[0018] For the port type, it is used to indicate the service openness of the port on the target device. That is, the characteristic data under the port type at least includes: the service openness of at least one port. For example, the service openness of any port may include: whether the port is open, whether the port is a high-risk port that meets the high-risk definition, such as port 80, port 3389, etc., and the services that the port can support, such as TCP service, UDP service, etc.

[0019] The service version type is used to indicate the service version information of the target device. That is, the characteristic data under the service version type at least includes: the service version information of the target device.

[0020] The device description type is used to indicate the description information of the target device. That is, the feature data under the device description type at least includes: description information of the target device such as device manufacturer, device type, device model, device firmware version information, etc.

[0021] As for the operating system type, it is used to indicate the operating system of the target device. That is, the following characteristic data at least includes: the operating system of the target device such as Windows, Linux, etc.

[0022] Those skilled in the art know that for a convolutional neural network, the input data is required to be a regular vector matrix, while in this embodiment, each feature data is mostly a discrete classification variable and has a variable length. According to the requirements of the convolutional neural network, it is necessary to force each feature data to be encoded as a vector of equal length according to a unified encoding format, which will lead to the introduction of noise such as useless data. In this embodiment, the different characteristics of each feature data are retained to match the encoding method for each feature data according to the characteristics of each feature data, which makes the encoding method matched by feature data of at least two different feature types different in this embodiment. For example, the encoding method for matching feature data under the service version type, the encoding method for matching feature data under the device description type, and the encoding method for matching feature data under the operating system type are different from the encoding method for matching feature data under the port type.

[0023] Specifically, for each feature type, the encoding method of the feature data under the feature type can be determined based on the longest length of the feature data under the feature type. For example, for the feature data under the port type, which is generally long, the service openness of each port under the port type can be encoded using a high-dimensional sparse matrix encoding method. When encoding, if there is a high-risk port that meets the high-risk definition, the corresponding weight coefficient is added to the service openness of the high-risk port; then, the high-dimensional sparse matrix encoding is reduced in dimension.

[0024] For the encoding method of feature data matching under the service version type, it can be encoded using the bert encoding method; for the encoding method of feature data matching under the device description type, it can use the transE algorithm to generate a word embedding vector; for the encoding method of feature data matching under the operating system type, it can use a one-dimensional vector to encode the feature data under the operating system type.

[0025] Step 102, encode each feature data in the device portrait according to the encoding method matched with each feature data in the device portrait, and generate a device feature vector of the target device according to the encoding result of each feature data.

[0026] After encoding each feature data in the device portrait according to the encoding method matching each feature data in the device portrait, the encoded data can be spliced ​​to finally obtain the device feature vector of the target device. Figure 2 The specific process from step 101 to step 102 is shown by way of example.

[0027] Step 103, obtaining a reference scanning result based on the historical scanning behavior executed on the target device; the reference scanning result at least includes: the scanning status of each scanning plug-in when scanning the target device, and the scanning status at least includes: missed reporting of abnormal information, false alarm, unidentified abnormal information, and correctly identified abnormal information.

[0028] It should be noted that this embodiment does not blindly use the historical scan results of the target device as the reference scan results. Instead, it uses the historical scan results to obtain a message: focus on the latest scan results of the target device, but also refer to other non-latest scan results of the target device. This information is recorded as the reference scan result.

[0029] As for how to obtain the reference scanning result, this embodiment uses the LSTM model, a prediction mechanism that combines long-term preference and short-term preference. Specifically, the historical scanning results of the target device are input into the LSTM model, and the LSTM model analyzes the historical scanning results of the target device to generate a reference scanning result; the reference scanning result focuses on the most recent scanning result of the target device, and also refers to other non-recent scanning results of the target device.

[0030] The reference scanning result here at least includes: the scanning situation of each scanning plug-in when scanning the target device, and the scanning situation at least includes: missed reporting of abnormal information, false reporting, unrecognized abnormal information, and correctly recognized abnormal information.

[0031] Step 104, using the strategy model Actor model in the current reinforcement learning model, based on the device feature vector and the reference scanning result, evaluate the potential benefits of each scanning plug-in to obtain a scanning plug-in selection strategy; and for each scanning plug-in indicated by the scanning plug-in selection strategy to be used when scanning the target device, determine the scanning priority of the scanning plug-in when scanning the target device according to the resources occupied by the scanning plug-in during execution, the risk of abnormal information that can be scanned by the scanning plug-in, and the most recent time when the scanning plug-in was used.

[0032] In this embodiment, a state space can be constructed based on the device feature vector and the reference scanning result, where the state space is defined as: {device feature vector, reference scanning result}. Afterwards, the state space is input into the current reinforcement learning model, so that the policy model (referred to as the Actor model) in the current reinforcement learning model evaluates the potential benefits of each scanning plug-in based on the input state space to obtain a scanning plug-in selection strategy. It can be seen that when the target device is currently scanned, the current features of the dynamically perceived target device, i.e., the device feature vector mentioned above, and the reference scanning results obtained based on the historical scanning behavior of the target device (which may include user feedback such as missed reports, false reports, unrecognized abnormal information, correctly recognized abnormal information, etc.), dynamically generate the scanning plug-in selection strategy used when the target device is currently scanned (which indicates which scanning plug-ins are decided to be used for scanning when the target device is currently scanned).

[0033] As an embodiment, the scanning plug-in selection strategy can be represented by an action space. The action space can be defined as: {a set of whether the scanning plug-in is called}. For example, if there are three scanning plug-ins A, B, and C, the action space can be {1, 0, 1}, indicating that the A and C scanning plug-ins can be used to scan the target device at present, and the B scanning plug-in is not used to scan the target device at present.

[0034] Optionally, in this embodiment, for each scanning plug-in indicated by the scanning plug-in selection policy to be used when scanning the target device, the scanning plug-in's scanning priority when scanning the target device is currently determined according to the resources occupied by the scanning plug-in during execution, the risk of abnormal information that can be scanned by the scanning plug-in, and the most recent time when the scanning plug-in was used. The scanning priority of each scanning plug-in when scanning the target device is currently determined is to be able to preferentially call a scanning plug-in with a higher scanning priority for scanning in the subsequent priority scheduling algorithm.

[0035] The following article will give an example to describe one of the methods for determining the scanning priority, which will not be described in detail here.

[0036] Step 105, according to the scanning priority of each scanning plug-in and in accordance with the priority scheduling algorithm, schedule each scanning plug-in to scan the target device to obtain the current scanning result of each scanning plug-in; determine the reward value of each scanning plug-in based on the current scanning result of each scanning plug-in and the reward function of each scanning plug-in, and use the value critic model in the current reinforcement learning model to evaluate the behavioral performance of the scanning plug-in selection strategy based on the reward value of each scanning plug-in, so as to optimize the Actor model according to the evaluation result of the Critic model; the optimized Actor model outputs a better scanning plug-in selection strategy than before optimization.

[0037] It can be seen that even if the scanning plug-in selection strategy is obtained in this embodiment, it does not blindly use the scanning plug-ins indicated by the scanning plug-in selection strategy to scan the target device. Instead, it uses the scanning priority of each scanning plug-in determined dynamically. According to the priority scheduling algorithm, the scanning plug-in with the highest scanning priority is preferentially scheduled for scanning, which ensures that the scanning plug-in with the highest benefit is preferentially scheduled.

[0038] It can also be seen that after scheduling each scanning plug-in to scan the target device, this embodiment will also determine the reward value of each scanning plug-in based on the current scanning result of each scanning plug-in and the reward function of each scanning plug-in, and adjust and optimize the Actor model in real time, that is, it realizes the real-time update of the Actor model by using the current scanning result feedback of each scanning plug-in during the application process, instead of the existing convolutional neural network once the training is completed, and then using the fixed model parameters of the convolutional neural network in multiple subsequent uses, and the model parameters cannot be dynamically fed back to adjust, which easily leads to repeated calls to the same scanning plug-in in multiple rounds of scanning, resulting in waste of resources, or multiple rounds of scanning without calling an actual existing scanning plug-in and generating false alarms.

[0039] As an example, the reward function of any scanning plug-in can be expressed as follows: ; in, Represents the reward value, , , is the weight coefficient.

[0040] As an example, Represents a scanning reward item, which is determined based on the risk of abnormal information that can be scanned by the scanning plug-in and the success rate of the scanning plug-in being called to successfully scan abnormal information. Specifically, It is expressed by the following formula: ;in, Indicates the risk score of the i-th scanning plug-in, which is determined according to the risk level of the abnormal information that can be scanned by the i-th scanning plug-in. For example, the higher the risk level of the abnormal information that can be scanned by the i-th scanning plug-in .... The higher the value, the lower the value and vice versa. Indicates the success rate of the i-th scanning plug-in being called to successfully scan abnormal information within a period of time.

[0041] As an example, Indicates the penalty item for scanning plug-in false alarms and scanning plug-in being called but not detecting abnormal information, which is used to instruct the scanning plug-in to reduce the call of the scanning plug-in when it performs poorly in the current state. Specifically, It is expressed by the following formula: ; Among them, when the scanning plug-in does not have a false alarm and the scanning plug-in is called without detecting abnormal information, is a first value, such as 0, otherwise it is a second value, such as 1.

[0042] As an example, Indicates exploration reward items; The number of times the scanning plug-in is called is determined based on the number of times the scanning plug-in is called and the frequency suppression factor to prevent the scanning plug-in from being called excessively; the number of times the new scanning plug-in is called is recorded as an initial value such as 0. Specifically, It is expressed by the following formula: ;in, , is an expression. When the number of times the scanning plug-in is called is less than the set threshold, the expression is the default value, such as 1. In other cases, it can be the set value, such as 0. The default value is 1, which means efficient; represents the frequency suppression factor, Indicates the number of times the scanning plug-in is called within a period of time. Indicates the total number of scans during this period of time.

[0043] After determining the reward value of each scanning plug-in based on the current scanning result of each scanning plug-in and the reward function of each scanning plug-in, the reward value of each scanning plug-in and the above-mentioned device feature vector are input into the current reinforcement learning model, so that the value Critic model in the current reinforcement learning model can evaluate the behavior performance of the scanning plug-in selection strategy, so as to optimize the above-mentioned Actor model according to the evaluation results of the Critic model; the optimized Actor model outputs a better scanning plug-in selection strategy than before optimization. That is, with the help of the evaluation results of the Critic model, the Actor model is continuously optimized to realize the understanding of the logical relationship between different state spaces when the reinforcement learning model is used later, so that when the target device is scanned again in the same state space later, a better scanning plug-in selection strategy will be output, achieving the purpose of continuously optimizing the scanning plug-in selection strategy.

[0044] So far, completed Figure 1 To make Figure 1 The process shown is more intuitive and clear. Figure 3 The specific implementation is illustrated with the keywords involved in each step.

[0045] pass Figure 1 It can be seen from the process shown that this embodiment dynamically generates the current scanning plug-in selection strategy through reinforcement learning based on the device feature vector of the target device perceived in real time and the historical scanning results of the target device, so as to ensure that the dynamically generated scanning plug-in selection strategy has strong dynamic adaptability, thereby reducing false positives and missed negatives.

[0046] Furthermore, this embodiment uses the reward mechanism of reinforcement learning to optimize and adjust the Actor model in the reinforcement learning model in real time according to the historical performance of different scanning plug-ins and the current scanning results of the target device (even with user feedback information), so as to continuously optimize the subsequent scanning plug-in selection strategy and improve detection efficiency and accuracy.

[0047] Furthermore, this embodiment describes the target device with as comprehensive information as possible by profiling the target device, and encodes each feature data in the device portrait according to the encoding method that matches each feature data in the device portrait to generate a device feature vector of the target device. This ensures data planning and management, avoids the conventional convolutional neural network that forces each feature data to be encoded as a vector of equal length according to a unified encoding format, thereby increasing noise such as useless data, and can further ensure that when a scanning strategy is subsequently generated for the target device based on the device feature vector, the accuracy of the strategy is improved.

[0048] The following describes how to determine the scanning priority of the scanning plug-in in the above step 104: See also Figure 4 , Figure 4The step 104 implementation flow chart provided in the embodiment of the present application. Figure 4 As shown, the process may include the following steps: Step 401: for each scanning plug-in, determine the risk score of the scanning plug-in according to the risk of abnormal information that the scanning plug-in can scan. .

[0049] For example, the higher the risk level of abnormal information that the i-th scanning plug-in can scan, the higher the risk score of the i-th scanning plug-in. The higher the value, the lower the value and vice versa.

[0050] Step 402: Determine the time decay factor corresponding to the scan plug-in based on the current time and the most recent time the scan plug-in was used. .

[0051] As an example, the time decay factor This can be achieved by an exponential function, for example, the time decay factor Determine as follows: ; in, Indicates the current time. Indicates the most recent time the scan plugin was used.

[0052] Step 403: Obtain the similarity Sim between the feature vector corresponding to the scanning plug-in and the device feature vector.

[0053] In this embodiment, the feature vector corresponding to the scanning plug-in is also encoded according to the encoding method matching each feature data in the plug-in portrait of the scanning plug-in. The feature vector corresponding to the scanning plug-in is used to describe the scanning plug-in. The plug-in portrait contains feature data corresponding to multiple different feature types, and each feature type in the plug-in portrait is similar to each feature type in the above-mentioned device portrait.

[0054] Step 404, based on risk score Assigned weights, time decay factors The weight assigned to the scan plugin and the weight assigned to the similarity Sim determine the overall score of the scan plugin. .

[0055] for example, It is expressed by the following formula: = ;in, , , Risk score Assigned weights, time decay factors The weight assigned to the similarity Sim.

[0056] Step 405: The score corresponding to the resources occupied by the scanning plug-in during execution is calculated. and the composite score , determining the scanning priority of the scanning plug-in when currently scanning the target device.

[0057] For example, the more resources the scanning plug-in occupies during execution, the higher the corresponding score. The smaller the scan plug-in is, the less resources it occupies when executing. The bigger it is.

[0058] As an example, this embodiment can and The quotient of is used as the scanning priority of the scanning plug-in. Optionally, the smaller the quotient, the higher the scanning priority.

[0059] So far, completed Figure 4 The process shown.

[0060] pass Figure 4 The process shown in the figure illustrates how to determine the scanning priority of the scanning plug-in.

[0061] As an embodiment, this embodiment adopts - Greedy strategy, when the Actor model decision scanning plug-in selects the strategy, by setting a small positive number (<1) probability of selecting an unknown scanning plugin, leaving 1- The probability of selecting the most valuable scanning plug-in among the existing scanning plug-ins is . With the use of the reinforcement learning model, the number of unknown scanning plug-ins decided by the Actor model is getting smaller and smaller, and the above Under this premise, the reinforcement learning model turns from exploration to utilization. That is, the current reinforcement learning model at this time is the optimal reinforcement learning model. Under this premise, before determining the reward value of each scanning plug-in based on the current scanning result of each scanning plug-in and the reward function of each scanning plug-in, this embodiment further identifies whether the current reinforcement learning model is the optimal reinforcement learning model. If so, the current process is terminated. If not, the reward value of each scanning plug-in is determined based on the current scanning result of each scanning plug-in and the reward function of each scanning plug-in.

[0062] The method provided in the embodiment of the present application is described above, and the device provided in the embodiment of the present application is described below.

[0063] See also Figure 5 , Figure 5 This is a diagram of the device structure provided in the embodiment of the present application. Figure 5 As shown, the process mainly includes: data perception module, strategy generation module, strategy execution module, and strategy feedback module.

[0064] As an embodiment, a data perception module is used to collect a device portrait of a target device and determine a coding method that matches each feature data in the device portrait of the target device; the device portrait contains feature data corresponding to multiple different feature types, and the feature data of at least two different feature types are matched with different coding methods; according to the coding method that matches each feature data in the device portrait, each feature data in the device portrait is encoded, and a device feature vector of the target device is generated according to the encoding result of each feature data; and, historical scanning results of the target device are collected.

[0065] The strategy generation module is used to obtain reference scanning results based on the historical scanning results of the target device, and use the strategy model Actor model in the current reinforcement learning model to evaluate the potential benefits of each scanning plug-in based on the device feature vector and the reference scanning results to obtain a scanning plug-in selection strategy.

[0066] A policy execution module is used to select each scanning plug-in that is required to be used when scanning the target device currently according to the policy indication for the scanning plug-in generated by the policy generation module, and determine the scanning priority of the scanning plug-in when scanning the target device currently according to the resources occupied by the scanning plug-in during execution, the risk of abnormal information that can be scanned by the scanning plug-in, the similarity between the feature vector corresponding to the scanning plug-in and the device feature vector, and the most recent time when the scanning plug-in was used; according to the scanning priority of each scanning plug-in and in accordance with the priority scheduling algorithm, schedule each scanning plug-in to scan the target device to obtain the current scanning result of each scanning plug-in.

[0067] The strategy feedback module is used to feed back the current scanning results, such as whether the called scanning plug-in recognizes abnormal information, whether it generates false positives or missed negatives, to the strategy generation module, and to determine the reward value of each scanning plug-in based on the current scanning results of each scanning plug-in and the reward function of each scanning plug-in and feed it back to the strategy generation module.

[0068] The strategy generation module uses the value Critic model in the current reinforcement learning model according to the feedback from the strategy feedback module to evaluate the behavioral performance of the scanning plug-in selection strategy based on the reward value of each scanning plug-in, so as to optimize the Actor model according to the evaluation result of the Critic model; the optimized Actor model outputs a better scanning plug-in selection strategy than before optimization.

[0069] As an embodiment, the device portrait involves at least two of a port type, a service version type, a device description type, and an operating system type; The characteristic data under the port type at least includes: the service opening status of at least one port; the encoding method of matching the characteristic data under the port type is: firstly use a high-dimensional sparse matrix encoding method to encode the service opening status of each port under the port type, wherein, when encoding, if there is a high-risk port that meets the high-risk definition, the corresponding weight coefficient is added to the service opening status of the high-risk port; then, the high-dimensional sparse matrix encoding is reduced in dimension; The characteristic data under the service version type at least includes: the service version information of the target device; the characteristic data under the device description type at least includes: the description information of the target device; the characteristic data under the operating system type at least includes: the operating system of the target device; Among them, the encoding method of feature data matching under the service version type, the encoding method of feature data matching under the device description type, and the encoding method of feature data matching under the operating system type are different from the encoding method of feature data matching under the port type.

[0070] As an embodiment, the encoding method of the feature data matching under the service version type is: encoding using the BERT encoding method; The description information at least includes the device manufacturer, device type, device model, and device firmware version information; the encoding method of the feature data matching under the device description type is: using the transE algorithm to generate a word embedding vector; The encoding method for matching the characteristic data under the operating system type is: using a one-dimensional vector to encode the characteristic data under the operating system type.

[0071] As an embodiment, obtaining a reference scanning result based on a historical scanning behavior performed on the target device includes: The historical scan results of the target device are input into the LSTM model, so that the LSTM model analyzes the historical scan results of the target device to generate a reference scan result; the reference scan result focuses on the most recent scan result of the target device and also refers to other non-most recent scan results of the target device.

[0072] As an embodiment, determining the scanning priority of the scanning plug-in when currently scanning the target device according to the resources occupied by the scanning plug-in when executing, the risk of abnormal information that can be scanned by the scanning plug-in, and the most recent time when the scanning plug-in was used includes: Determine the risk score of the scanning plug-in based on the risk of abnormal information that the scanning plug-in can scan ; Determine the time decay factor corresponding to the scan plugin based on the current time and the most recent time the scan plugin was used ; Obtaining the similarity Sim between the feature vector corresponding to the scanning plug-in and the device feature vector; Based on risk score Assigned weights, time decay factors The weight assigned to the scan plugin and the weight assigned to the similarity Sim determine the overall score of the scan plugin. ; The score corresponding to the resources occupied by the scanning plug-in during execution and the composite score , determining the scanning priority of the scanning plug-in when currently scanning the target device.

[0073] As an embodiment, the time decay factor Determine as follows: ; in, Indicates the current time. Indicates the most recent time the scan plugin was used.

[0074] As an example, the reward function of any scanning plug-in is expressed as follows: ; in, Indicates the scanning reward item, which is determined based on the risk of abnormal information that can be scanned by the scanning plug-in and the success rate of the scanning plug-in being called to successfully scan abnormal information; Indicates the penalty item for scanning plug-in false alarms and scanning plug-in being called but no abnormal information is detected. It is used to instruct the scanning plug-in to reduce the call of the scanning plug-in when it performs poorly in the current state; Indicates exploration reward items; The number of times the scanning plug-in is called is determined based on the number of times the scanning plug-in is called and the frequency suppression factor to prevent the scanning plug-in from being called excessively; the number of times the new scanning plug-in is called is recorded as the initial value; , , is the weight coefficient.

[0075] As an example, It is expressed by the following formula: ; in, represents the risk score of the i-th scanning plug-in, which is determined according to the risk of abnormal information that can be scanned by the i-th scanning plug-in; Indicates the success rate of the i-th scanning plug-in being called to successfully scan abnormal information within a period of time; It is expressed by the following formula: ; Among them, when the scanning plug-in does not have a false alarm and the scanning plug-in is called without detecting abnormal information, is the first value, otherwise it is the second value; It is expressed by the following formula: ; in, , which is the default value when the number of times the scanning plug-in is called is less than the set threshold. represents the frequency suppression factor, Indicates the number of times the scanning plug-in is called within a period of time. Indicates the total number of scans during this period of time.

[0076] As an embodiment, before determining the reward value of each scanning plug-in based on the current scanning result of each scanning plug-in and the reward function of each scanning plug-in, the method further includes: Identify whether the current reinforcement learning model meets The optimal reinforcement learning model required by the greedy strategy is Greedy strategy is used to describe unknown probabilities Decreases over time, and the probability of an unknown scanning plug-in explored by the reinforcement learning model is less than or equal to When the current reinforcement learning model is determined to satisfy The optimal reinforcement learning model required by the greedy strategy; if so, end the current process, if not, continue to determine the reward value of each scanning plug-in based on the current scanning results of each scanning plug-in and the reward function of each scanning plug-in.

[0077] So far, completed Figure 5 Structural description of the device shown.

[0078] The present application also provides Figure 5 The hardware structure of the device shown. Figure 6 , Figure 6 This is a structural diagram of an electronic device provided in an embodiment of the present application. Figure 6 As shown, the hardware structure may include: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the method disclosed in the above example of this application.

[0079] Based on the same application concept as the above method, an embodiment of the present application also provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the method disclosed in the above example of the present application can be implemented.

[0080] Exemplarily, the above-mentioned machine-readable storage medium can be any electronic, magnetic, optical or other physical storage device, which can contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium can be: RAM (Radom Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as optical disk, DVD, etc.), or similar storage medium, or a combination thereof.

[0081] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for generating intelligent vulnerability scanning strategies based on reinforcement learning, characterized in that: The method is applied to a scanning device, and the method comprises: Determine an encoding method for matching each feature data in a device portrait of a target device; the device portrait is generated based on device information of the target device obtained by actively scanning the target device; the device portrait includes feature data corresponding to a plurality of different feature types, and feature data of at least two different feature types are matched with different encoding methods; According to the encoding method matched with each feature data in the device portrait, each feature data in the device portrait is encoded, and according to the encoding result of each feature data, a device feature vector of the target device is generated; Based on the historical scanning behavior executed on the target device, a reference scanning result is obtained; the reference scanning result at least includes: the scanning situation of each scanning plug-in when scanning the target device, and the scanning situation at least includes: missed abnormal information, false positive, unrecognized abnormal information, and correctly recognized abnormal information; Using the strategy model Actor model in the current reinforcement learning model, based on the device feature vector and the reference scanning result, the potential benefits of each scanning plug-in are evaluated to obtain a scanning plug-in selection strategy; and for each scanning plug-in indicated by the scanning plug-in selection strategy to be used when scanning the target device, according to the resources occupied by the scanning plug-in during execution, the risk of abnormal information that can be scanned by the scanning plug-in, and the most recent time when the scanning plug-in was used, the scanning priority of the scanning plug-in when scanning the target device is currently determined; According to the scanning priority of each scanning plug-in and in accordance with the priority scheduling algorithm, each scanning plug-in is scheduled to scan the target device to obtain the current scanning result of each scanning plug-in; based on the current scanning result of each scanning plug-in and the reward function of each scanning plug-in, the reward value of each scanning plug-in is determined, and the value Critic model in the current reinforcement learning model evaluates the behavioral performance of the scanning plug-in selection strategy based on the reward value of each scanning plug-in, so as to optimize the Actor model according to the evaluation result of the Critic model; the optimized Actor model outputs a better scanning plug-in selection strategy than before optimization.

2. The method according to claim 1, characterized in that The device portrait involves at least two of a port type, a service version type, a device description type, and an operating system type; The characteristic data under the port type at least includes: the service opening status of at least one port; the encoding method of matching the characteristic data under the port type is: firstly use a high-dimensional sparse matrix encoding method to encode the service opening status of each port under the port type, wherein, when encoding, if there is a high-risk port that meets the high-risk definition, the corresponding weight coefficient is added to the service opening status of the high-risk port; then, the high-dimensional sparse matrix encoding is reduced in dimension; The characteristic data under the service version type at least includes: the service version information of the target device; the characteristic data under the device description type at least includes: the description information of the target device; the characteristic data under the operating system type at least includes: the operating system of the target device; Among them, the encoding method of feature data matching under the service version type, the encoding method of feature data matching under the device description type, and the encoding method of feature data matching under the operating system type are different from the encoding method of feature data matching under the port type.

3. The method according to claim 2, characterized in that The encoding method of the feature data matching under the service version type is: encoding using the BERT encoding method; The description information at least includes the device manufacturer, device type, device model, and device firmware version information; the encoding method of the feature data matching under the device description type is: using the transE algorithm to generate a word embedding vector; The encoding method for matching the characteristic data under the operating system type is: using a one-dimensional vector to encode the characteristic data under the operating system type.

4. The method according to claim 1, characterized in that The obtaining of a reference scanning result based on a historical scanning behavior performed on the target device includes: The historical scan results of the target device are input into the LSTM model, so that the LSTM model analyzes the historical scan results of the target device to generate a reference scan result; the reference scan result focuses on the most recent scan result of the target device and also refers to other non-most recent scan results of the target device.

5. The method according to claim 1, characterized in that: The determining, according to the resources occupied by the scanning plug-in when executing, the risk of abnormal information that can be scanned by the scanning plug-in, and the most recent time when the scanning plug-in was used, the scanning priority of the scanning plug-in when currently scanning the target device comprises: Determine the risk score of the scanning plug-in based on the risk of abnormal information that the scanning plug-in can scan ; Determine the time decay factor corresponding to the scan plugin based on the current time and the most recent time the scan plugin was used ; Obtaining the similarity Sim between the feature vector corresponding to the scanning plug-in and the device feature vector; Based on risk score Assigned weights, time decay factors The weight assigned to the scan plugin and the weight assigned to the similarity Sim determine the overall score of the scan plugin. ; The score corresponding to the resources occupied by the scanning plug-in during execution and the composite score , determine the scanning priority of the scanning plug-in when currently scanning the target device.

6. The method according to claim 5, characterized in that The time decay factor Determine as follows: ； in, Indicates the current time. Indicates the most recent time the scan plugin was used.

7. The method according to claim 1, characterized in that The reward function of any scanning plugin is expressed as follows: ; in, Indicates the scanning reward item, which is determined based on the risk of abnormal information that can be scanned by the scanning plug-in and the success rate of the scanning plug-in being called to successfully scan abnormal information; Indicates the penalty item for scanning plug-in false alarms and scanning plug-in being called but no abnormal information is detected. It is used to instruct the scanning plug-in to reduce the call of the scanning plug-in when it performs poorly in the current state; Indicates exploration reward items; The number of times the scanning plug-in is called is determined based on the number of times the scanning plug-in is called and the frequency suppression factor to prevent the scanning plug-in from being called excessively; the number of times the new scanning plug-in is called is recorded as the initial value; , , is the weight coefficient.

8. The method according to claim 7, characterized in that It is expressed by the following formula: ； in, represents the risk score of the i-th scanning plug-in, which is determined according to the risk of abnormal information that can be scanned by the i-th scanning plug-in; Indicates the success rate of the i-th scanning plug-in being called to successfully scan abnormal information within a period of time; It is expressed by the following formula: ; Among them, when the scanning plug-in does not have a false alarm and the scanning plug-in is called without detecting abnormal information, is the first value, otherwise it is the second value; It is expressed by the following formula: ; in, , when the number of times the scanning plug-in is called is less than the set threshold efficient, is the default value; represents the frequency suppression factor, Indicates the number of times the scanning plug-in is called within a period of time. Indicates the total number of scans during this period of time.

9. The method according to claim 1, characterized in that: Before determining the reward value of each scanning plug-in based on the current scanning result of each scanning plug-in and the reward function of each scanning plug-in, the method further includes: Identify whether the current reinforcement learning model meets -The optimal reinforcement learning model required by the greedy strategy, the ϵ-greedy strategy is used to describe the unknown probability Decreases over time, and the probability of an unknown scanning plug-in explored by the reinforcement learning model is less than or equal to When the current reinforcement learning model is determined to satisfy -The optimal reinforcement learning model required by the greedy strategy; If yes, then the current process ends; if no, then the reward value of each scanning plug-in is determined based on the current scanning result of each scanning plug-in and the reward function of each scanning plug-in.

10. A scanning device, characterized in that: The device includes: A data perception module, used to determine the encoding method matching each feature data in the device portrait of the target device; the device portrait is generated based on the device information of the target device obtained by actively scanning the target device; the device portrait contains feature data corresponding to multiple different feature types, and the encoding methods matching feature data of at least two different feature types are different; and, used to encode each feature data in the device portrait according to the encoding method matching each feature data in the device portrait, and generate a device feature vector of the target device according to the encoding results of each feature data; A strategy generation module is used to obtain a reference scanning result based on the historical scanning behavior executed on the target device; the reference scanning result at least includes: the scanning situation of each scanning plug-in when scanning the target device, and the scanning situation at least includes: missed abnormal information, false positive, unrecognized abnormal information, and correctly recognized abnormal information; and, using the strategy model Actor model in the current reinforcement learning model, based on the device feature vector and the reference scanning result, evaluate the potential benefits of each scanning plug-in to obtain a scanning plug-in selection strategy; a policy execution module, configured to determine, for each scanning plug-in that is required to be used when scanning the target device currently as indicated by the scanning plug-in selection policy, a scanning priority of the scanning plug-in when scanning the target device currently according to resources occupied by the scanning plug-in when executing, risk of abnormal information that can be scanned by the scanning plug-in, and a recent time when the scanning plug-in was used; and According to the scanning priority of each scanning plug-in and the priority scheduling algorithm, each scanning plug-in is scheduled to scan the target device to obtain the current scanning result of each scanning plug-in; A strategy feedback module, used to determine the reward value of each scanning plug-in based on the current scanning result of each scanning plug-in and the reward function of each scanning plug-in, and feed back the reward value of each scanning plug-in and the current scanning result of each scanning plug-in to the strategy generation module; The strategy generation module is further used to evaluate the behavioral performance of the scanning plug-in selection strategy based on the reward value of each scanning plug-in according to the feedback of the strategy feedback module and using the value Critic model in the current reinforcement learning model, so as to optimize the Actor model according to the evaluation result of the Critic model; the optimized Actor model outputs a better scanning plug-in selection strategy than before optimization.

Images