Network target range task evaluation method and device, equipment and storage medium

Abstract

The invention discloses a network target range task evaluation method and device, equipment and a storage medium. The method comprises the steps of updating a pre-constructed evaluation model according to a network target range task; wherein the evaluation model is of a tree structure formed by a plurality of evaluation indexes, the score of each upper-layer evaluation index is obtained by gathering the scores of at least one lower-layer evaluation index, the bottom-layer evaluation indexes serve as leaf nodes to be bound with at least one calculation rule, and each calculation rule corresponds to a set rule score; according to each calculation rule bound with the bottom layer evaluation index in the evaluation model, constructing a plurality of alarm rules; executing a network target range task, matching the real-time equipment acquisition data of each network target range equipment with each alarm rule, and generating alarm information after successful matching; based on the alarm information and the score convergence mode of the evaluation indexes of each level in the evaluation model, the evaluation score of the network target range task is obtained through step-by-step calculation, and the evaluation accuracy of the network target range task is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of shooting range effectiveness evaluation, and in particular to an evaluation method, device, equipment and storage medium for network shooting range tasks. Background Art

[0002] As a network system simulation platform for collaborative use by all parties, the cyber range is mainly used to simulate and reproduce the operating status and operating environment of network architecture, system equipment, and business processes in real cyberspace, so as to more effectively realize network security-related learning, research, testing, competition, and exercises, thereby improving the network security confrontation level of personnel and institutions. In the actual application of the cyber range, scenario instances are usually built on the cloud platform through the range platform to perform cyber range tasks.

[0003] In the prior art, in order to evaluate the completion of a cyber range task, data is usually collected on a device corresponding to a scenario instance, and the collected data is analyzed and evaluated through a large language model.

[0004] However, since the accuracy and credibility of the evaluation method of the effectiveness of the cyber range task through the large language model depend on the training depth of the model, when the training data is insufficient, the effectiveness of the cyber range task cannot be accurately determined by the large language model. Summary of the invention

[0005] The present invention provides a method, device, equipment and storage medium for evaluating a network range task, which can accurately and reliably evaluate the execution effect of the network range task and reduce the evaluation cost of the network range task.

[0006] In a first aspect, an embodiment of the present invention provides a method for evaluating a network range task, the method comprising:

[0007] Acquire a network range task, wherein the network range task includes at least one network range device, and each network range device is used to collect at least one type of data;

[0008] The pre-built evaluation model is updated according to the network range task; wherein the evaluation model is a tree structure composed of multiple evaluation indicators, the score of each upper evaluation indicator is obtained by aggregating the score of at least one lower evaluation indicator, the leaf node is the bottom evaluation indicator, the bottom evaluation indicator is bound to at least one calculation rule, and each calculation rule corresponds to a set rule score;

[0009] Construct multiple alarm rules based on the calculation rules bound to the underlying evaluation indicators in the evaluation model;

[0010] Execute network range tasks, match the real-time device collection data of each network range device with each alarm rule, and generate alarm information after the match is successful;

[0011] Based on the alarm information and the score aggregation method of each level of evaluation indicators in the evaluation model, the evaluation score of the network target range task is calculated step by step.

[0012] Optionally, an evaluation model is constructed based on a network target range task, including: selecting task measurement indicators from at least one type of data collected by each network target range device based on the task characteristics of the network target range task; constructing calculation rules corresponding to each task measurement indicator, and binding the determined calculation rules to the underlying evaluation indicators; and updating the pre-constructed evaluation model according to the selected underlying evaluation indicators.

[0013] Optionally, after obtaining the evaluation score of the network target range task by step-by-step calculation, it also includes: calculating a target difference obtained by subtracting the evaluation score from a preset standard score; if the target difference is greater than a preset failure threshold, returning to execute the operation of selecting a task measurement indicator from at least one type of data collected by each network target range device according to the task characteristics of the network target range task, until the target difference is less than or equal to the preset failure threshold.

[0014] Optionally, after the target difference is less than or equal to the preset failure threshold, it also includes: determining whether the target difference is greater than a preset exit threshold; if so, returning to execute the operation of constructing calculation rules corresponding to each task metric; if not, adding the calculation rules to the preset evaluation rule library, so as to directly determine the evaluation score of the network target range task according to the calculation rules in the preset evaluation rule library.

[0015] Optionally, based on each alarm information and the aggregation method of evaluation indicators at each level in the evaluation model, the evaluation score of the network target range task is calculated step by step, including: obtaining each target calculation rule corresponding to each alarm information, the target rule score corresponding to each target calculation rule, and the target underlying evaluation indicator bound to each target calculation rule, and determining the target indicator score of the target underlying evaluation indicator according to each target rule score and the benchmark score corresponding to the target underlying evaluation indicator; determining the evaluation score of the network target range task according to the target indicator score of the target underlying evaluation indicator and a predefined upper-level evaluation indicator aggregation method; wherein the predefined upper-level evaluation indicator aggregation method includes at least one of the following: weighted calculation and selective use.

[0016] Optionally, the target indicator score of the target underlying evaluation indicator is determined according to each target rule score and a benchmark score corresponding to the target underlying evaluation indicator, including: if the benchmark score corresponding to the target underlying evaluation indicator is a first preset score, then after determining the first initial indicator score of the target underlying evaluation indicator according to the sum of each target rule score and the first preset score, the target indicator score of the target underlying evaluation indicator is determined according to the comparison result of the first initial indicator score and the first score threshold; if the benchmark score corresponding to the target underlying evaluation indicator is a second preset score, then determining the target indicator score of the target underlying evaluation indicator according to the sum of each target rule score and the second preset score; if the benchmark score corresponding to the target underlying evaluation indicator is a third preset score, then after determining the second initial indicator score of the target underlying evaluation indicator according to the sum of each target rule score and the third preset score, the target indicator score of the target underlying evaluation indicator is determined according to the comparison result of the second initial indicator score and the second score threshold; wherein the second preset score is greater than the first preset score and less than the third preset score.

[0017] Optionally, before determining the target indicator score of the target underlying evaluation indicator based on the scores of each target rule and the benchmark score corresponding to the target underlying evaluation indicator, it also includes: judging whether the rule scores of each calculation rule corresponding to the target underlying evaluation indicator are all positive numbers; if so, setting the benchmark score corresponding to the target underlying evaluation indicator to the first preset score; if not, judging whether the rule scores of each calculation rule corresponding to the target underlying evaluation indicator are all negative numbers; if the rule scores of each calculation rule corresponding to the target underlying evaluation indicator are all negative numbers, setting the benchmark score corresponding to the target underlying evaluation indicator to the third preset score; if the rule scores of each calculation rule corresponding to the target underlying evaluation indicator are not all negative numbers, setting the benchmark score corresponding to the target underlying evaluation indicator to the second preset score.

[0018] In a second aspect, an embodiment of the present invention further provides an evaluation device for a network range task, the device comprising:

[0019] A range task acquisition module is used to acquire a network range task, wherein the network range task includes at least one network range device, and each network range device is used to collect at least one type of data;

[0020] An evaluation model update module is used to update a pre-built evaluation model according to a network range task; wherein the evaluation model is a tree structure composed of multiple evaluation indicators, the score of each upper evaluation indicator is obtained by aggregating the score of at least one lower evaluation indicator, the leaf node is the bottom evaluation indicator, the bottom evaluation indicator is bound to at least one calculation rule, and each calculation rule corresponds to a set rule score;

[0021] An alarm rule construction module is used to construct multiple alarm rules according to various calculation rules bound to the underlying evaluation indicators in the evaluation model;

[0022] The alarm information generation module is used to execute the network range task and match the real-time device collection data of each network range device with each alarm rule, and generate alarm information after the match is successful;

[0023] The evaluation score calculation module is used to calculate the evaluation score of the network target range task step by step based on the score aggregation method of each alarm information and the evaluation indicators of each level in the evaluation model.

[0024] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the network target range task evaluation method provided in any embodiment of the present invention.

[0025] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a processor to implement the network target range task evaluation method provided by any embodiment of the present invention when executed.

[0026] The technical solution provided by the embodiment of the present invention updates the evaluation model composed of evaluation indicators of the network range task, and calculates the evaluation score of the network range task step by step according to the score aggregation method of the evaluation indicators of each level in the evaluation model, which solves the problem that the execution effect of the network range task cannot be accurately determined by the large language model when the training data is insufficient, and can accurately and reliably evaluate the execution effect of the network range task. Secondly, since the training cost of the large language model is relatively high, the technical solution of this embodiment evaluates the execution effect of the network range task through a mathematical model (i.e., an evaluation model) containing multiple evaluation indicators, thereby reducing the evaluation cost of the network range task.

[0027] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 is a flow chart of a method for evaluating a network range task provided according to Embodiment 1 of the present invention;

[0030] Figure 2 is a flow chart of another network range task evaluation method provided according to Embodiment 2 of the present invention;

[0031] Figure 3 It is a logical architecture diagram of an evaluation system for a network range task provided according to an embodiment of the present invention;

[0032] Figure 4 is a flowchart of a preferred method for evaluating a network range task provided according to an embodiment of the present invention;

[0033] Figure 5 is a schematic diagram of the structure of an evaluation device for a network range task provided according to Embodiment 3 of the present invention;

[0034] Figure 6 It is a schematic diagram of the structure of an electronic device provided by Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] Embodiment 1

[0038] Figure 1 It is a flowchart of a method for evaluating a network range task provided according to the first embodiment of the present invention. This embodiment is applicable to the situation where the execution effect of a network range task is evaluated. The method can be executed by an evaluation device for a network range task. The evaluation device for the network range task can be implemented in the form of hardware and / or software. The evaluation device for the network range task can be configured in an electronic device such as a computer.

[0039] like Figure 1 As shown, the present embodiment discloses a method for evaluating a network range task, including:

[0040] S110. Obtain a network range task, wherein the network range task includes at least one network range device, and each network range device is used to collect at least one type of data.

[0041] In this embodiment, the network range task may be a task for conducting attack and defense training on a pre-built network range. Specifically, at least one type of data corresponding to the network range task may be collected by the network range device. Among them, the various types of data collected by the network range device may be used to reflect the execution effect of the network range task.

[0042] S120. Update the pre-constructed evaluation model according to the network target range task; wherein the evaluation model is a tree structure composed of multiple evaluation indicators, the score of each upper-level evaluation indicator is obtained by aggregating the score of at least one lower-level evaluation indicator, the leaf node is the underlying evaluation indicator, the underlying evaluation indicator is bound to at least one calculation rule, and each calculation rule corresponds to a set rule score.

[0043] In this step, specifically, multiple calculation rules can be determined based on at least one type of data collected by each network range device. Then, multiple underlying evaluation indicators can be determined based on the multiple calculation rules and the rule scores corresponding to each calculation rule. Finally, the upper-level evaluation indicators can be obtained by aggregating the underlying evaluation indicators, and the pre-built evaluation model can be updated based on the underlying evaluation indicators and the upper-level evaluation indicators.

[0044] S130: Construct multiple alarm rules according to the calculation rules bound to the underlying evaluation indicators in the evaluation model.

[0045] In this embodiment, the alarm rule can be used to generate alarm information when the data collected by the network range device meets the calculation rule. Specifically, each calculation rule bound to the underlying evaluation index in the evaluation model can be converted into an alarm rule.

[0046] For example, assuming that a calculation rule bound to the underlying evaluation indicator is that the real-time collected central processing unit (CPU) usage rate increases by more than 80% within 1 minute, the alarm rule corresponding to the calculation rule is to generate an alarm message when the CPU usage rate increases by more than 80% within 1 minute.

[0047] S140, executing the network target range task, and matching the real-time device collection data of each network target range device with each alarm rule, and generating alarm information after the match is successful.

[0048] Among them, the real-time device collection data can be data collected by network target range equipment.

[0049] Continuing with the previous example, if the real-time collected CPU usage is 90%, an alarm message is generated.

[0050] S150, based on the alarm information and the scoring aggregation method of the evaluation indicators at each level in the evaluation model, the evaluation score of the network target range task is calculated step by step.

[0051] In this step, specifically, the calculation rules corresponding to each alarm information can be obtained, and the indicator score of each underlying evaluation indicator corresponding to the alarm information can be determined according to the rule score corresponding to each calculation rule. Then, the evaluation score of the network range task can be calculated step by step according to the indicator score of each underlying evaluation indicator and the updated evaluation model.

[0052] The technical solution of this embodiment is to obtain a network range task, wherein the network range task includes at least one network range device, and each network range device is used to collect at least one type of data; update a pre-constructed evaluation model according to the network range task; wherein the evaluation model is a tree structure composed of multiple evaluation indicators, and the score of each upper evaluation indicator is obtained by aggregating the score of at least one lower evaluation indicator, and the leaf node is the bottom evaluation indicator, and the bottom evaluation indicator is bound to at least one calculation rule, and each calculation rule corresponds to a set rule score; construct multiple alarm rules according to each calculation rule bound to the bottom evaluation indicator in the evaluation model; execute the network range task, and match the real-time device collection data of each network range device with each alarm rule, and generate alarm information after the match is successful; based on each alarm information and the score aggregation method of each level of evaluation indicators in the evaluation model, the technical means of calculating the evaluation score of the network range task step by step solves the problem that when the training data is insufficient, the execution effect of the network range task cannot be accurately determined by the large language model, and the execution effect of the network range task can be accurately and reliably evaluated. Secondly, since the training cost of a large language model is relatively high, the technical solution of this embodiment evaluates the execution effect of the network target range task through a mathematical model (i.e., an evaluation model) that includes multiple evaluation indicators, thereby reducing the evaluation cost of the network target range task.

[0053] Embodiment 2

[0054] Figure 2 It is a flowchart of a method for evaluating a network range task provided according to Embodiment 2 of the present invention. This embodiment is a further optimization and expansion based on the above embodiments, and can be combined with various optional technical solutions in the above implementation modes.

[0055] like Figure 2 As shown, the present embodiment discloses a method for evaluating a network range task, including:

[0056] S210. Obtain a network range task, wherein the network range task includes at least one network range device, and each network range device is used to collect at least one type of data.

[0057] S220. According to the task characteristics of the network range task, select a task measurement indicator from at least one type of data collected by each network range device.

[0058] In this embodiment, the task metric can be used to evaluate the execution effect of the network range task.

[0059] For example, assuming that the task characteristics of the network target range task are that the task operation will increase the CPU utilization, memory utilization, network traffic and the number of processes, the CPU utilization, memory utilization, network traffic and the number of processes can be used as task measurement indicators.

[0060] S230, constructing calculation rules corresponding to each task measurement indicator, and binding the determined calculation rules to the underlying evaluation indicators.

[0061] In this step, specifically, the calculation rules corresponding to each task metric and the rule scores when the calculation rules are met can be determined according to the fluctuation of the historical data collected by the network range equipment and / or user needs. Then, the determined calculation rules and the rule scores corresponding to the calculation rules can be bound to the underlying evaluation indicators.

[0062] Continuing with the previous example, assuming that the task metric is CPU usage, the calculation rule corresponding to the CPU usage can be determined to be whether the CPU usage increases by more than 80% within 1 minute, and when the CPU usage increases by more than 80% within 1 minute, the rule score corresponding to the CPU usage is determined to be -80.

[0063] S240: Update the pre-built evaluation model according to the selected underlying evaluation indicators.

[0064] In this step, specifically, the upper-level evaluation indicators corresponding to the selected underlying evaluation indicators may be updated according to the selected underlying evaluation indicators, and the pre-built evaluation model may be updated according to the selected underlying evaluation indicators and the updated upper-level evaluation indicators.

[0065] S250: Construct multiple alarm rules according to the calculation rules bound to the underlying evaluation indicators in the evaluation model.

[0066] S260, executing the network target range task, and matching the real-time device collection data of each network target range device with each alarm rule, and generating alarm information after the match is successful.

[0067] S270. Obtain each target calculation rule corresponding to each alarm information, the target rule score corresponding to each target calculation rule, and the target underlying evaluation indicator bound to each target calculation rule, and determine the target indicator score of the target underlying evaluation indicator based on each target rule score and the benchmark score corresponding to the target underlying evaluation indicator.

[0068] In this step, specifically, first, after obtaining each target calculation rule corresponding to each alarm information and the target rule score corresponding to each target calculation rule, the target underlying evaluation index bound to each target calculation rule can be obtained. Then, the benchmark score corresponding to the target underlying evaluation index can be obtained, and the target index score of the target underlying evaluation index can be determined according to each target rule score and the benchmark score. Among them, the benchmark score can be determined according to the rule score of each calculation rule bound to the target underlying evaluation index.

[0069] Exemplarily, it is assumed that the target calculation rules corresponding to each alarm information are the first calculation rule, the second calculation rule, the third calculation rule, the fourth calculation rule, the fifth calculation rule and the sixth calculation rule, and the target underlying evaluation index corresponding to the first calculation rule and the second calculation rule is the first underlying evaluation index, the target underlying evaluation index corresponding to the third calculation rule and the fourth calculation rule is the second underlying evaluation index, and the target underlying evaluation index corresponding to the fifth calculation rule and the sixth calculation rule is the third underlying evaluation index. At this time, the target index score of the first underlying evaluation index can be determined according to the rule scores corresponding to the first calculation rule and the second calculation rule, and the benchmark score corresponding to the first underlying evaluation index. The target index score of the second underlying evaluation index is determined according to the rule scores corresponding to the third calculation rule and the fourth calculation rule, and the benchmark score corresponding to the second underlying evaluation index. The target index score of the third underlying evaluation index is determined according to the rule scores corresponding to the fifth calculation rule and the sixth calculation rule, and the benchmark score corresponding to the third underlying evaluation index.

[0070] Optionally, the target indicator score of the target underlying evaluation indicator is determined according to each target rule score and a benchmark score corresponding to the target underlying evaluation indicator, including: if the benchmark score corresponding to the target underlying evaluation indicator is a first preset score, then after determining the first initial indicator score of the target underlying evaluation indicator according to the sum of each target rule score and the first preset score, the target indicator score of the target underlying evaluation indicator is determined according to the comparison result of the first initial indicator score and the first score threshold; if the benchmark score corresponding to the target underlying evaluation indicator is a second preset score, then determining the target indicator score of the target underlying evaluation indicator according to the sum of each target rule score and the second preset score; if the benchmark score corresponding to the target underlying evaluation indicator is a third preset score, then after determining the second initial indicator score of the target underlying evaluation indicator according to the sum of each target rule score and the third preset score, the target indicator score of the target underlying evaluation indicator is determined according to the comparison result of the second initial indicator score and the second score threshold; wherein the second preset score is greater than the first preset score and less than the third preset score.

[0071] Among them, the first preset score, the second preset score, the third preset score, the first score threshold and the second score threshold can be set according to user needs. For example, the first preset score can be set to 0 points, the first score threshold can be set to 100 points, the second preset score can be set to any value between 0 and 100, the third preset score can be set to 100, and the second score threshold can be set to 0 points.

[0072] In a specific embodiment, if the first initial indicator score is greater than or equal to the first score threshold, the target indicator score of the target underlying evaluation indicator is determined to be the first score threshold. If the second initial indicator score is less than or equal to the comparison result of the second score threshold, the target indicator score of the target underlying evaluation indicator is determined to be the second score threshold.

[0073] Continuing with the above example, if the benchmark score corresponding to the first underlying evaluation indicator is 0, the first score threshold is 100, the first rule score corresponding to the first calculation rule is 50, and the second rule score corresponding to the second calculation rule is 60, then the first initial indicator score corresponding to the first underlying evaluation indicator is determined to be 110. At this point, the first initial indicator score is greater than the first score threshold, and the target indicator score of the first underlying evaluation indicator is 100.

[0074] If the benchmark score corresponding to the first underlying evaluation indicator is any value between 0 and 100 points, the third rule score corresponding to the third calculation rule is -30, and the fourth rule score corresponding to the fourth calculation rule is 20 points, then it can be determined that the second underlying evaluation indicator is -10 points.

[0075] If the benchmark score corresponding to the third underlying evaluation indicator is 100 points, the second score threshold is 0 points, the fifth rule score corresponding to the fifth calculation rule is -50 points, and the sixth rule score corresponding to the sixth calculation rule is -60 points, then the second initial indicator score corresponding to the third underlying evaluation indicator is determined to be -10 points. At this time, the second initial indicator score is less than the first score threshold, and the target indicator score of the third underlying evaluation indicator is 0 points.

[0076] Optionally, before determining the target indicator score of the target underlying evaluation indicator based on the scores of each target rule and the benchmark score corresponding to the target underlying evaluation indicator, it also includes: judging whether the rule scores of each calculation rule corresponding to the target underlying evaluation indicator are all positive numbers; if so, setting the benchmark score corresponding to the target underlying evaluation indicator to the first preset score; if not, judging whether the rule scores of each calculation rule corresponding to the target underlying evaluation indicator are all negative numbers; if the rule scores of each calculation rule corresponding to the target underlying evaluation indicator are all negative numbers, setting the benchmark score corresponding to the target underlying evaluation indicator to the third preset score; if the rule scores of each calculation rule corresponding to the target underlying evaluation indicator are not all negative numbers, setting the benchmark score corresponding to the target underlying evaluation indicator to the second preset score.

[0077] In a specific implementation, if the rule scores of each calculation rule corresponding to the target underlying evaluation indicator are all positive numbers, the benchmark score corresponding to the target underlying evaluation indicator is set to a first preset score. If the rule scores of each calculation rule corresponding to the target underlying evaluation indicator are all negative numbers, the benchmark score corresponding to the target underlying evaluation indicator is set to a third preset score. If the rule scores of the calculation rules corresponding to the target underlying evaluation indicator include both positive numbers and negative numbers, the benchmark score corresponding to the target underlying evaluation indicator is set to a second preset score.

[0078] The advantage of this setting is that by determining the benchmark score corresponding to the target underlying evaluation indicator through the rule score of each calculation rule corresponding to the target underlying evaluation indicator, the accuracy of the target indicator score can be improved, thereby improving the accuracy of the shooting range task evaluation results.

[0079] S280, determining the evaluation score of the network range task according to the target indicator score of the target underlying evaluation indicator and the predefined upper-level evaluation indicator aggregation method.

[0080] The predefined upper layer evaluation indicator aggregation method includes at least one of the following: weighted calculation and selective use.

[0081] In this step, specifically, the evaluation score of the network range task can be determined based on the target indicator score of each target underlying evaluation indicator corresponding to the upper-level evaluation indicator, and the weight coefficient corresponding to each target indicator score. Alternatively, a first target underlying evaluation indicator can be determined among each target underlying evaluation indicator corresponding to the upper-level evaluation indicator, and the indicator score of the first target underlying evaluation indicator can be used as the evaluation score of the network range task. Alternatively, multiple second target underlying evaluation indicators can be determined among each target underlying evaluation indicator corresponding to the upper-level evaluation indicator, and the average value of each second target underlying evaluation indicator can be used as the evaluation score of the network range task.

[0082] Optionally, after obtaining the evaluation score of the network target range task by step-by-step calculation, it also includes: calculating a target difference obtained by subtracting the evaluation score from a preset standard score; if the target difference is greater than a preset failure threshold, returning to execute the operation of selecting a task measurement indicator from at least one type of data collected by each network target range device according to the task characteristics of the network target range task, until the target difference is less than or equal to the preset failure threshold.

[0083] The preset standard score may be determined by the user based on historical experience.

[0084] Optionally, after the target difference is less than or equal to the preset failure threshold, it also includes: determining whether the target difference is greater than the preset exit threshold; if so, returning to execute the operation of constructing the calculation rules corresponding to each task metric; if not, adding the calculation rules to the preset evaluation rule library, so as to directly determine the evaluation score of the network range task according to the calculation rules in the preset evaluation rule library

[0085] The advantage of this setting is that by reselecting the task measurement indicator when the target difference is greater than the preset failure threshold, and by redetermining the calculation rules when the target difference is less than or equal to the preset failure threshold but greater than the preset exit threshold, the strategy for evaluating the network target range task can be adjusted in time, thereby improving the accuracy of the network target range task evaluation results.

[0086] The technical solution of this embodiment obtains each target calculation rule corresponding to each alarm information, the target rule score corresponding to each target calculation rule, and the target underlying evaluation indicator bound to each target calculation rule, and determines the target indicator score of the target underlying evaluation indicator according to each target rule score and the benchmark score corresponding to the target underlying evaluation indicator; the technical means for determining the evaluation score of the network target range task according to the target indicator score of the target underlying evaluation indicator and the predefined upper-level evaluation indicator aggregation method solves the problem that the execution effect of the network target range task cannot be accurately determined by the large language model when the training data is insufficient, and can realize the step-by-step calculation of the task evaluation score of the network target range, thereby improving the accuracy of determining the execution effect of the network target range task and reducing the cost of determining the execution effect of the network target range task.

[0087] Figure 3 is a schematic diagram of the structure of a network range task evaluation system provided according to an embodiment of the present invention. Figure 4 It is a flowchart of a preferred method for evaluating a network range task provided according to an embodiment of the present invention.

[0088] Based on the above embodiments, the present invention also provides a logical architecture diagram of an evaluation system for network range tasks for implementing the above embodiments, such as Figure 3 As shown, the logical architecture of the network range task evaluation system includes a pre-built evaluation model, a rule converter, a time series database, an alarm processor and an evaluation result generator. Among them, the evaluation model is a tree structure composed of multiple evaluation indicators, and the score of each upper evaluation indicator is obtained by aggregating the score of at least one lower evaluation indicator. The leaf node is the bottom evaluation indicator, and the bottom evaluation indicator is bound to at least one calculation rule, and each calculation rule corresponds to a set rule score. The rule converter can be used to convert the calculation rule into an alarm rule and send the alarm rule to the time series database. The time series database can be used to determine whether the data collected under the scenario instance of the network range task meets the alarm rule, and send an alarm message to the alarm processor when the alarm rule is met. The alarm processor can be used to obtain each target calculation rule corresponding to each alarm information, the target rule score corresponding to each target calculation rule, and the target bottom evaluation indicator bound to each target calculation rule, and determine the target indicator score of the target bottom evaluation indicator according to each target rule score and the benchmark score corresponding to the target bottom evaluation indicator. The evaluation result generator can be used to calculate the evaluation score of the network range task step by step according to the target indicator score and the score aggregation method of the evaluation indicators at each level in the evaluation model. It is worth noting that the evaluation model can include the first type of upper evaluation indicators and the second type of upper evaluation indicators. In the evaluation model, the first type of upper evaluation indicators is located between the bottom evaluation indicators and the second type of upper evaluation indicators.

[0089] Based on Figure 3 The logical architecture shown in FIG. 1 is a preferred implementation scheme. Figure 4 As shown, first, a network range task, as well as instance devices and at least one type of data involved in the network range task, can be obtained, and the measurement name of at least one type of data in the time series database can be determined. According to the task characteristics of the network range task, a task measurement indicator is selected from each measurement name. Then, a calculation rule can be selected from the preset evaluation rule library, or a calculation rule corresponding to the task evaluation indicator can be formulated, and the calculation rule can be converted into an alarm rule through a rule converter, and the alarm rule can be sent to the time series database. When the network range task starts to execute, the real-time device collection data of the network range device is received through the time series database. If the real-time device collection data hits the alarm rule, an alarm message is generated through the time series database.

[0090] After receiving each alarm information, the alarm processor obtains each target calculation rule corresponding to each alarm information, the target rule score corresponding to each target calculation rule, and the target underlying evaluation index bound to each target calculation rule, and determines the target index score of the target underlying evaluation index according to each target rule score and the benchmark score corresponding to the target underlying evaluation index. The evaluation result generator determines the evaluation score of the network range task according to the target index score of the target underlying evaluation index and the upper evaluation index aggregation method predefined in the evaluation model.

[0091] After determining the evaluation score of the network range task, determine whether the target difference obtained by subtracting the evaluation score from the preset standard score is greater than the preset failure threshold; if so, return to execute the operation of selecting task metric indicators from each metric name according to the task characteristics of the network range task until the target difference is less than or equal to the preset failure threshold.

[0092] When the target difference is less than or equal to the preset failure threshold, determine whether the target difference is greater than the preset exit threshold. If the target difference is greater than the preset exit threshold, after archiving the calculation rules and results to the pre-built experience library to avoid the subsequent repeated use of the above calculation rules, return to execute the operation of formulating the calculation rules corresponding to the task evaluation indicators until the target difference is less than or equal to the preset exit threshold. If the target difference is less than or equal to the preset exit threshold, add the calculation rules to the preset evaluation rule library so that the evaluation score of the network range task can be determined directly according to the calculation rules in the preset evaluation rule library.

[0093] The advantage of such a setting is that, firstly, the technical solution of the present invention defines a set of evaluation indicators through the data collected by the network range equipment, so that the instructor only needs to consider the data that can be collected when formulating the plan to complete the formulation of the evaluation plan. Secondly, the technical solution of the present invention does not need to rely on the large language model and the maturity of the model training. When creating the plan, the instructor can formulate the evaluation indicators according to the effects and characteristics of the network range task when completing the task, and accurately predict the final evaluation score according to the completion of the network range task, thereby providing convenience for the instructor to create a targeted training plan.

[0094] Embodiment 3

[0095] Figure 5 is a schematic diagram of the structure of an evaluation device for a network range task provided according to Embodiment 3 of the present invention, such as Figure 5 As shown, the device includes: a shooting range task acquisition module 51, an evaluation model updating module 52, an alarm rule construction module 53, an alarm information generation module 54 and an evaluation score calculation module 55, wherein:

[0096] A range task acquisition module 51 is used to acquire a network range task, wherein the network range task includes at least one network range device, and each network range device is used to collect at least one type of data;

[0097] An evaluation model updating module 52 is used to update a pre-built evaluation model according to a network range task; wherein the evaluation model is a tree structure composed of multiple evaluation indicators, the score of each upper evaluation indicator is obtained by aggregating the score of at least one lower evaluation indicator, the leaf node is the bottom evaluation indicator, the bottom evaluation indicator is bound to at least one calculation rule, and each calculation rule corresponds to a set rule score;

[0098] An alarm rule construction module 53 is used to construct multiple alarm rules according to various calculation rules bound to the underlying evaluation indicators in the evaluation model;

[0099] The alarm information generation module 54 is used to execute the network range task and match the real-time device collection data of each network range device with each alarm rule, and generate alarm information after the match is successful;

[0100] The evaluation score calculation module 55 is used to calculate the evaluation score of the network range task step by step based on the score aggregation method of each alarm information and each level of evaluation indicators in the evaluation model.

[0101] The technical solution in this embodiment solves the problem that the execution effect of the network target range task cannot be accurately determined by the large language model when the training data is insufficient, through the mutual cooperation of the target range task acquisition module, the evaluation model update module, the alarm rule construction module, the alarm information generation module and the evaluation score calculation module. It can achieve accurate and reliable evaluation of the execution effect of the network target range task and reduce the evaluation cost of the network target range task.

[0102] Optionally, the evaluation model update module 52 is specifically used to: select task measurement indicators from at least one type of data collected by each network target range device according to the task characteristics of the network target range task; construct calculation rules corresponding to each task measurement indicator, and bind the determined calculation rules to the underlying evaluation indicators; and update the pre-constructed evaluation model according to the selected underlying evaluation indicators.

[0103] Optionally, the device also includes a metric reselection module, which is used to: calculate a target difference obtained by subtracting the evaluation score from a preset standard score; if the target difference is greater than a preset failure threshold, return to execute the operation of selecting the task metric from at least one type of data collected by each network target range device according to the task characteristics of the network target range task, until the target difference is less than or equal to the preset failure threshold.

[0104] Optionally, the device also includes a calculation rule redetermination module, which is used to: determine whether the target difference is greater than a preset exit threshold; if so, return to execute the operation of constructing calculation rules corresponding to each task metric; if not, add the calculation rules to a preset evaluation rule library, so as to directly determine the evaluation score of the network target range task based on the calculation rules in the preset evaluation rule library.

[0105] Optionally, the evaluation score calculation module 55 is specifically used to: obtain each target calculation rule corresponding to each alarm information, the target rule score corresponding to each target calculation rule, and the target underlying evaluation indicator bound to each target calculation rule, and determine the target indicator score of the target underlying evaluation indicator according to each target rule score and the benchmark score corresponding to the target underlying evaluation indicator; determine the evaluation score of the network target range task according to the target indicator score of the target underlying evaluation indicator and a predefined upper-level evaluation indicator aggregation method; wherein the predefined upper-level evaluation indicator aggregation method includes at least one of the following: weighted calculation and selective use.

[0106] Optionally, the evaluation score calculation module 55 is specifically used to: if the benchmark score corresponding to the target underlying evaluation indicator is a first preset score, then after determining the first initial indicator score of the target underlying evaluation indicator according to the sum of each target rule score and the first preset score, determine the target indicator score of the target underlying evaluation indicator according to the comparison result of the first initial indicator score and the first score threshold; if the benchmark score corresponding to the target underlying evaluation indicator is a second preset score, then determine the target indicator score of the target underlying evaluation indicator according to the sum of each target rule score and the second preset score; if the benchmark score corresponding to the target underlying evaluation indicator is a third preset score, then after determining the second initial indicator score of the target underlying evaluation indicator according to the sum of each target rule score and the third preset score, determine the target indicator score of the target underlying evaluation indicator according to the comparison result of the second initial indicator score and the second score threshold; wherein the second preset score is greater than the first preset score and less than the third preset score.

[0107] Optionally, the device also includes a benchmark score determination module, which is used to: determine whether the rule scores of each calculation rule corresponding to the target underlying evaluation indicator are all positive numbers; if so, set the benchmark score corresponding to the target underlying evaluation indicator to the first preset score; if not, determine whether the rule scores of each calculation rule corresponding to the target underlying evaluation indicator are all negative numbers; if the rule scores of each calculation rule corresponding to the target underlying evaluation indicator are all negative numbers, set the benchmark score corresponding to the target underlying evaluation indicator to the third preset score; if the rule scores of each calculation rule corresponding to the target underlying evaluation indicator are not all negative numbers, set the benchmark score corresponding to the target underlying evaluation indicator to the second preset score.

[0108] The network range task evaluation device provided in the embodiment of the present invention can execute the network range task evaluation method provided in any embodiment 1 of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. For the contents not described in detail in this embodiment, reference can be made to the description in any method embodiment of this application.

[0109] Embodiment 4

[0110] Figure 6 A schematic structural diagram of an electronic device 10 that can be used to implement an embodiment of the present invention is shown.

[0111] like Figure 6 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0112] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0113] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as the evaluation method of the network range task.

[0114] In some embodiments, the evaluation method of the network range task can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the evaluation method of the network range task described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the evaluation method of the network range task in any other appropriate manner (for example, by means of firmware).

[0115] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0116] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0117] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0118] To provide interaction with a client user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the client user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the client user can provide input to the electronic device. Other types of devices may also be used to provide interaction with a client user; for example, the feedback provided to the client user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the client user may be received in any form (including acoustic input, voice input, or tactile input).

[0119] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a client user computer with a graphical client user interface or a web browser through which a client user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0120] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0121] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0122] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for evaluating a network range task, characterized in that: The method comprises: Acquire a network range task, wherein the network range task includes at least one network range device, and each network range device is used to collect at least one type of data; The pre-built evaluation model is updated according to the network range task; wherein the evaluation model is a tree structure composed of multiple evaluation indicators, the score of each upper evaluation indicator is obtained by aggregating the score of at least one lower evaluation indicator, the leaf node is the bottom evaluation indicator, the bottom evaluation indicator is bound to at least one calculation rule, and each calculation rule corresponds to a set rule score; Construct multiple alarm rules based on the calculation rules bound to the underlying evaluation indicators in the evaluation model; Execute network range tasks, match the real-time device collection data of each network range device with each alarm rule, and generate alarm information after the match is successful; Based on the alarm information and the score aggregation method of each level of evaluation indicators in the evaluation model, the evaluation score of the network target range task is calculated step by step.

2. The method according to claim 1, characterized in that The evaluation model is constructed based on the network range tasks, including: According to the task characteristics of the cyber range task, select task metrics from at least one type of data collected by each cyber range device; Construct calculation rules corresponding to each task metric, and bind the determined calculation rules to the underlying evaluation indicators; According to the selected underlying evaluation indicators, the pre-built evaluation model is updated.

3. The method according to claim 2, characterized in that After the evaluation scores of the network range tasks are calculated step by step, it also includes: Calculate the target difference obtained by subtracting the evaluation score from the preset standard score. If the target difference is greater than the preset failure threshold, return to execute the operation of selecting the task measurement indicator from at least one type of data collected by each network target range device according to the task characteristics of the network target range task until the target difference is less than or equal to the preset failure threshold.

4. The method according to claim 3, characterized in that After the target difference is less than or equal to the preset failure threshold, the following also applies: Determine whether the target difference is greater than a preset exit threshold; If so, return to execute the operation of constructing the calculation rules corresponding to each task metric; If not, the calculation rules are added to the preset evaluation rule library so that the evaluation score of the network range task can be determined directly according to the calculation rules in the preset evaluation rule library.

5. The method according to claim 1, characterized in that Based on the alarm information and the aggregation method of evaluation indicators at each level in the evaluation model, the evaluation scores of the network range tasks are calculated step by step, including: Obtain each target calculation rule corresponding to each alarm information, the target rule score corresponding to each target calculation rule, and the target underlying evaluation indicator bound to each target calculation rule, and determine the target indicator score of the target underlying evaluation indicator according to each target rule score and the benchmark score corresponding to the target underlying evaluation indicator; Determine the evaluation score of the network range task based on the target indicator score of the target bottom-level evaluation indicator and the pre-defined upper-level evaluation indicator aggregation method; The predefined upper layer evaluation indicator aggregation method includes at least one of the following: weighted calculation and selective use.

6. The method according to claim 5, characterized in that According to the scores of each target rule and the benchmark scores corresponding to the target underlying evaluation indicators, the target indicator scores of the target underlying evaluation indicators are determined, including: If the benchmark score corresponding to the target underlying evaluation indicator is a first preset score, after determining a first initial indicator score for the target underlying evaluation indicator according to the sum of each target rule score and the first preset score, determine a target indicator score for the target underlying evaluation indicator according to a comparison result of the first initial indicator score and the first score threshold; If the benchmark score corresponding to the target underlying evaluation indicator is the second preset score, determining the target indicator score of the target underlying evaluation indicator according to the sum of each target rule score and the second preset score; If the benchmark score corresponding to the target underlying evaluation indicator is the third preset score, after determining the second initial indicator score of the target underlying evaluation indicator according to the sum of each target rule score and the third preset score, determine the target indicator score of the target underlying evaluation indicator according to the comparison result of the second initial indicator score and the second score threshold; The second preset score is greater than the first preset score and less than the third preset score.

7. The method according to claim 6, characterized in that Before determining the target indicator score of the target underlying evaluation indicator based on the scores of each target rule and the benchmark score corresponding to the target underlying evaluation indicator, it also includes: Determine whether the rule scores of each calculation rule corresponding to the target underlying evaluation indicator are all positive numbers; If yes, then setting the benchmark score corresponding to the target underlying evaluation indicator to the first preset score; If not, determine whether the rule scores of each calculation rule corresponding to the target underlying evaluation indicator are all negative; If the rule scores of each calculation rule corresponding to the target underlying evaluation indicator are negative, the benchmark score corresponding to the target underlying evaluation indicator is set to a third preset score; If the rule scores of each calculation rule corresponding to the target underlying evaluation indicator are not all negative numbers, the benchmark score corresponding to the target underlying evaluation indicator is set to the second preset score.

8. An evaluation device for a network range task, characterized in that: The device comprises: A range task acquisition module, used to acquire a network range task, wherein the network range task includes at least one network range device, and each network range device is used to collect at least one type of data; An evaluation model update module is used to update a pre-built evaluation model according to a network range task; wherein the evaluation model is a tree structure composed of multiple evaluation indicators, the score of each upper evaluation indicator is obtained by aggregating the score of at least one lower evaluation indicator, the leaf node is the bottom evaluation indicator, the bottom evaluation indicator is bound to at least one calculation rule, and each calculation rule corresponds to a set rule score; An alarm rule construction module is used to construct multiple alarm rules according to various calculation rules bound to the underlying evaluation indicators in the evaluation model; The alarm information generation module is used to execute the network range task and match the real-time device collection data of each network range device with each alarm rule, and generate alarm information after the match is successful; The evaluation score calculation module is used to calculate the evaluation score of the network target range task step by step based on the score aggregation method of each alarm information and the evaluation indicators of each level in the evaluation model.

9. An electronic device, characterized in that: The electronic device comprises: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the method for evaluating a network range task as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the evaluation method of a network range task according to any one of claims 1 to 7 when executed.

Images