Malware detection model training method and device based on itemized reward and operation index linkage

Abstract

The invention discloses a malicious software detection model training method and device based on item reward and operation index linkage, and belongs to the technical field of artificial intelligence and malicious software detection. The method comprises the following steps: constructing a reinforcement learning model containing a policy network; the policy network is configured to receive software feature input and output multi-modal actions including classification results, structured interpretation text and detection confidence; based on the training sample, generating a multi-modal action by utilizing a strategy network, and calculating a composite reward value for the generated multi-modal action by utilizing a preset subitem reward; the subitem rewards at least comprise a structure consistency reward, a key element hit reward and a confidence coefficient calibration reward; and calculating advantage estimation based on the composite reward value to construct an objective function of a near-end policy optimization algorithm by using the advantage estimation, and maximizing the objective function by updating parameters of the policy network. The method can improve the explanatory and confidence of model output, and improves the recognition precision.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and malware detection technology, and in particular to a method and apparatus for training a malware detection model based on the linkage of sub-rewards and operational indicators. Background Technology

[0002] Currently, malware detection models based on the linkage between itemized rewards and operational metrics are trained using reinforcement learning. However, reinforcement learning relies on global rewards during training, which can lead to unstable recognition, insufficient interpretability, and low confidence levels. Summary of the Invention

[0003] This invention provides a method and apparatus for training a malware detection model based on the linkage between itemized rewards and operational metrics. The technical solution is as follows: On the one hand, a method for training a malware detection model based on the linkage between sub-item rewards and operational metrics is provided, the method comprising: 100: Construct a reinforcement learning model that includes a policy network; the policy network is configured to receive software feature input and output multimodal actions including classification results, structured explanatory text, and detection confidence. 102: Based on the training samples, multimodal actions are generated using the policy network, and a composite reward value is calculated for the generated multimodal actions using pre-set sub-rewards; the sub-rewards include at least: a structural consistency reward for evaluating whether the generated structured explanatory text conforms to the set structural specifications, a key element hit reward for evaluating the key element matching degree, and a confidence calibration reward for calibrating the generated detection confidence to approximate the preset operational indicators. 104: Calculate the advantage estimate based on the composite reward value, and use the advantage estimate to construct the objective function of the near-end policy optimization algorithm. Maximize the objective function by updating the parameters of the policy network. 106: Repeat steps 102 to 104 to iteratively optimize the policy network until a trained malware detection model is obtained.

[0004] On the other hand, a malware detection model training device based on the linkage of sub-item rewards and operational indicators is provided, the device comprising: A building unit is used to build a reinforcement learning model containing a policy network; the policy network is configured to receive software feature input and output multimodal actions including classification results, structured explanatory text, and detection confidence. The computing unit is used to generate multimodal actions based on training samples using the policy network, and to calculate a composite reward value for the generated multimodal actions using pre-set sub-rewards; the sub-rewards include at least: a structural consistency reward for evaluating whether the generated structured explanatory text conforms to the set structural specifications, a key element hit reward for evaluating the key element matching degree, and a confidence calibration reward for calibrating the generated detection confidence to approximate the preset operational indicators. The update unit is used to calculate the advantage estimate based on the composite reward value, and to construct the objective function of the near-end policy optimization algorithm using the advantage estimate, and to maximize the objective function by updating the parameters of the policy network. An iterative unit is used to trigger the repeated execution of the computation unit and the update unit to iteratively optimize the policy network until a trained malware detection model is obtained.

[0005] On the other hand, a computer device is provided, the computer device including a memory and a processor, the memory for storing computer programs, and the processor for executing the computer programs stored in the memory to implement the steps of the above-described method for training a malware detection model based on the linkage of sub-item rewards and operational indicators.

[0006] On the other hand, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements the steps of the above-described method for training a malware detection model based on the linkage of sub-item rewards and operational indicators.

[0007] On the other hand, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the above-described method for training a malware detection model based on the linkage of sub-item rewards and operational indicators.

[0008] The technical solution provided by this invention can bring at least the following beneficial effects: When training a malware detection model using reinforcement learning, a component-based reward mechanism is introduced. This mechanism calculates a composite reward value based on the model's output, which is then used to adjust the model parameters. This allows the model to optimize for multi-dimensional objectives during training, and the structural consistency and confidence rewards enhance the interpretability and confidence of the model's output. Furthermore, operational metrics are directly integrated with the optimization objectives during training, ensuring that the model meets business requirements such as high accuracy. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a flowchart of a method for training a malware detection model based on the linkage between sub-item rewards and operational indicators, provided by an embodiment of the present invention. Figure 2 This is a structural diagram of a malware detection model training device based on the linkage of sub-item rewards and operational indicators, provided in an embodiment of the present invention. Figure 3 This is a hardware architecture diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0012] Analysis of existing malware detection systems reveals that their model training typically suffers from the following problems: 1. During the actual deployment phase, the model will focus more on operational metrics, but there is a disconnect between the training objectives and operational metrics of the existing model; 2. The model may give high confidence for incorrect predictions, and the operation side will try to alleviate false alarms by adjusting the threshold after the fact, which is a temporary solution; 3. Using classification results as rewards to supervise the training process, even if the model classifies correctly after training, there may be missing fields or disordered order in the structured interpretation.

[0013] Based on the problems identified in the above analysis, the inventive concept of this invention is to break down multiple key objectives in the malware detection task into calculable sub-rewards, and to link these sub-rewards with operational metrics during the training phase through reinforcement learning, thereby optimizing the training process.

[0014] The above-mentioned inventive concept will be explained below.

[0015] Please refer to Figure 1 This invention provides a method for training a malware detection model based on the linkage between sub-item rewards and operational metrics. The method includes: Step 100: Construct a reinforcement learning model that includes a policy network and a value network; the policy network is configured to receive software feature input and output multimodal actions including classification results, structured explanatory text, and detection confidence. Step 102: Based on the training samples, generate multimodal actions using the policy network, and calculate a composite reward value for the generated multimodal actions using pre-set sub-rewards; the sub-rewards include at least: a structural consistency reward for evaluating whether the generated structured explanatory text conforms to the set structural specifications, a key element hit reward for evaluating the key element matching degree, and a confidence calibration reward for calibrating the generated detection confidence to approximate the preset operational indicators. Step 104: Based on the composite reward value and advantage function, calculate the policy gradient, and use the near-end policy optimization algorithm to update the parameters of the policy network using the policy gradient; Step 106: Repeat steps 102 to 104 to iteratively optimize the policy network until a trained malware detection model is obtained.

[0016] In this embodiment of the invention, when training the malware detection model using reinforcement learning, a component reward mechanism is introduced. Component rewards are used to calculate a composite reward value for the model's output, which is then used to adjust the model parameters. This allows the model to optimize for multi-dimensional objectives during training, and the structural consistency reward and confidence reward enhance the interpretability and confidence of the model's output. Furthermore, operational metrics are directly integrated with the optimization objectives during training, ensuring that the model meets business requirements such as high accuracy.

[0017] The following description Figure 1 The execution method for each step is shown.

[0018] First, for step 100, a reinforcement learning model containing a policy network is constructed.

[0019] In this embodiment of the invention, a reinforcement learning method is used to refine the training of the malware detection model. The reinforcement learning method preferentially selects the proximal policy optimization (PPO) method, which enables the training process to have greater flexibility and adaptability. By finely optimizing for different types of malware samples, the overall performance of the model is improved.

[0020] The reinforcement learning model includes a policy network, which can be understood to also include a value network. This policy network is configured to receive software feature input and output multimodal actions including classification results, structured explanatory text, and detection confidence. Once the reinforcement learning model is trained, the policy network constitutes the malware detection model of this embodiment.

[0021] In this embodiment of the invention, before training the malware detection model using reinforcement learning, the malware detection model can be either pre-trained or untrained. Preferably, the malware detection model can be pre-trained; therefore, the policy network is initialized using the pre-trained malware detection model.

[0022] It should be noted that the methods for constructing and initializing the value network are existing technical solutions, and will not be described in detail in this embodiment of the invention.

[0023] In this embodiment of the invention, the input to the policy network is software features, and the multimodal output includes classification results, structured explanatory text, and detection confidence. The classification results can be binary, ternary, or fine-grained classification; the structured explanatory text explains the model's detection decisions in structured text form, providing a human-readable reasoning process, and may include family tags, attack behaviors, threat indicators, etc.; where family tags refer to the specific family or variant to which the malware belongs; and the detection confidence quantifies the model's confidence in its own predictions, reflecting the reliability of the predictions.

[0024] The input to the policy network is explained below.

[0025] In one embodiment of the present invention, the software features input to the policy network can be extracted based on the conversion of the software binary file into an coded image. Specifically, the software features are obtained as follows: A1: Determine the mapping relationship between binary bytes and image pixels; In this embodiment of the invention, the mapping relationship between binary bytes and image pixels can include single-channel mapping and multi-channel mapping. Single-channel mapping encodes one byte into one pixel; multi-channel mapping encodes multiple bytes (three bytes for RGB channels and four bytes for RGBA channels) into one pixel.

[0026] A2: Determine the encoding method for structural information; In one embodiment of the present invention, the structural information includes: segment information and / or high-entropy regions.

[0027] In this embodiment of the invention, the structure of a binary file typically includes multiple distinct parts, such as code segments and data segments, each referred to as a segment. The location and content of segments within the binary file are crucial for malware identification. In this embodiment of the invention, by preserving segment information and optimizing it through an attention mechanism, the malware detection system can better understand the structural characteristics of the binary file.

[0028] In one implementation, segment information includes segment boundaries. Segment boundaries are physical separation points between different segments in a binary file, used to distinguish different parts of the binary file. In another implementation, segment information also includes segment types. Segment types can include: .text segment, .data segment, .bss segment, .rodata segment, .stack segment, and .heap segment.

[0029] High-entropy regions typically contain complex or random content and are often associated with malicious behavior. In this embodiment of the invention, high-entropy regions of the binary file are determined by calculating the byte probability distribution of each window region in the binary file one by one. The attention mechanism is optimized by extracting entropy features to accurately identify key parts of malicious files.

[0030] It is understandable that the encoding methods for structural information include the encoding of segmental information and the encoding methods for high-entropy regions.

[0031] Specifically, the segment information is encoded in at least one of the following ways: each segment occupies a different area in the encoded image; different segments are distinguished by pixel intervals or line intervals; and different segment types are encoded using different colors.

[0032] Encoding high-entropy regions can be achieved by setting the pixel positions of the encoded image corresponding to the byte range of the high-entropy region in the binary file as specific feature markers. These feature markers include specific colors and / or specific shapes. This allows the malware detection model to focus more on these high-entropy regions during training, improving the accuracy of malware detection.

[0033] A3: Encode the byte sequence in the binary file into the image according to the mapping relationship and encoding method to obtain the encoded image; It should be noted that, in order to ensure the structural fidelity of the binary file content, in one embodiment of the present invention, the line width of the encoded image corresponding to different binary files is adaptively obtained based on the size and structure of the binary file. The line width of the encoded image is the minimum value that can carry the binary file, and the line width is within a preset line width range. Furthermore, the size of the encoded image satisfies a preset aspect ratio. In this way, the loss of important information caused by a fixed line width can be avoided.

[0034] A4: Use the image processing module to extract visual features from the coded image, and determine the software features based on the extracted visual features.

[0035] In this embodiment of the invention, the image processing module can be a convolutional neural network (CNN) or a visual transformer (ViT). The image processing module can extract visual features from the encoded image. Since the encoded image retains structural information, the image processing module pays attention to this structural information during the extraction of visual features.

[0036] This visual feature can be used as a software feature. Alternatively, it can be fused with textual features from a security context, and the fused feature can be used as a software feature.

[0037] Then, for step 102, based on the training samples, multimodal actions are generated using the policy network, and a composite reward value is calculated for the generated multimodal actions using pre-set sub-rewards.

[0038] In this embodiment of the invention, the training samples include software features as input and real multimodal actions as output. The training process is batch training, and each batch training process requires multiple training samples to participate in the training.

[0039] After inputting the software features of the training samples into the policy network, the policy network generates predicted multimodal actions. To make the output of the policy network more accurate, multiple sub-rewards are pre-set in this embodiment of the invention. Specifically, the sub-rewards include at least: structural consistency reward, key element hit reward, and confidence calibration reward.

[0040] The following is a detailed explanation of each of these three sub-rewards.

[0041] First, structural consistency reward In this embodiment of the invention, the structural consistency reward is used to evaluate whether the generated structured explanatory text conforms to the set structural specifications.

[0042] In one implementation, the structural consistency reward can at least be used to detect and establish field order and field integrity. Specifically, the structural consistency reward is calculated as follows: The structured explanatory text generated by the policy network is compared with the target text. An order consistency score is generated based on whether the field order is consistent. An integrity score is generated based on the coverage of the generated fields to all fields in the target text. The structural consistency reward is calculated based on the consistency score and the integrity score.

[0043] The target text is the structured explanatory text of the real multimodal actions output from the training samples.

[0044] For example, if the target text is {"category": "malware", "family": "ransomware", "confidence": 0.85}, and the generated structured interpretation text is {"family": "ransomware", "category":"malware", "confidence": 0.90}, then: Order consistency: By checking the order of fields, the order consistency score of the generated structured explanatory text can be 1 (consistent) or 0 (inconsistent). In the example above, the order consistency score is 0.

[0045] Completeness: Checks whether the generated fields cover all fields of the target text. If any field is missing in the generated structured explanation text, the completeness score is 0; otherwise, the completeness score is 1.

[0046] In one implementation, the structural consistency reward can be a weighted sum of the sequential consistency score and the integrity score.

[0047] Let the target field set be The generated field set is So, in one implementation: Integrity score It can be calculated using the following formula: Sequential consistency score It can be calculated using the following formula: The structural consistency reward is calculated using the sequence consistency score and integrity score: in, As a reward for structural consistency, These are the weighting coefficients.

[0048] The structural consistency reward is calculated using the above method, and this reward can directly constrain the explanation of structural quality.

[0049] Second, key element hit reward In this embodiment of the invention, the key element hit reward is used to evaluate the key element matching degree. In one implementation, the key elements include at least malware family tags and attack behaviors.

[0050] Specifically, the calculation method for the key element hit reward is as follows: Identify and extract malware family tags and attack behaviors contained in the structured explanatory text generated by the policy network; The extracted results are matched with the real target, and the key element hit reward is calculated using the precision and recall of the matching.

[0051] In one implementation method, let the set of elements of the real target be... The extracted result set of elements is ,So: Precision and recall are calculated using the following formulas: in, For accuracy; This refers to the recall rate.

[0052] In one implementation, the key element hit reward can be a weighted sum of precision and recall.

[0053] Right now: in, For key element hit rewards, These are the weighting coefficients.

[0054] By introducing this key element of hit reward, the accuracy of critical information such as malware families and attack behaviors in the model output can be ensured.

[0055] Third, confidence calibration reward In this embodiment of the invention, the confidence calibration reward is used to calibrate the generated detection confidence level to approximate a preset operational metric. In one implementation, the operational metric includes the false positive rate (FPR) and the true positive rate (TPR).

[0056] The confidence calibration reward is calculated as follows: Determine the target confidence threshold based on preset operational indicators; The confidence calibration reward is calculated based on the deviation between the generated detection confidence score and the target confidence score threshold.

[0057] In this embodiment of the invention, once the operational indicators are determined, for example, if the operational target is FPR@TPR(95%) ≤ 2%, that is, to primarily ensure that the TPR (true positive rate) is not lower than 95%, and under this premise, to pursue the FPR (false positive rate) as low as possible (target ≤ 2%), then the target confidence threshold can be determined by the sorting search method or the PPO dynamic calibration method. Specifically, it can be ensured that... Under the premise of minimizing FPR, the confidence level that minimizes FPR is selected as the target confidence threshold.

[0058] Furthermore, to make the confidence level predicted by the policy network smoother and more consistent with the deviation from the actual results, temperature scaling can be used to calibrate the predicted confidence level. Specifically, before calculating the confidence level calibration reward, the following steps are included: calibrating the generated detection confidence level using a temperature scaling factor to obtain a calibrated confidence level, and then calculating the confidence level calibration reward using the deviation between the calibrated confidence level and the target confidence level threshold.

[0059] The calibration method is as follows: calculate the quotient of the generated detection confidence level and the temperature scaling factor, and use the softmax function to normalize the quotient to obtain the calibrated confidence level.

[0060] In one implementation, the absolute value of the deviation between the calibrated confidence level and the target confidence level threshold is determined as the confidence level calibration reward.

[0061] For example, the confidence calibration reward can be calculated using the following formula: in, As a confidence level calibration reward, The calibrated confidence level. The target confidence threshold, This is the temperature scaling factor.

[0062] In summary, we can obtain rewards for structural consistency, key element hits, and confidence calibration.

[0063] After obtaining the structural consistency reward, key element hit reward, and confidence calibration reward, the composite reward value can be calculated using the structural consistency reward, key element hit reward, and confidence calibration reward and their weight values.

[0064] In one implementation, the compound reward value is calculated using the following formula: Among them, R total R is the composite reward value. structure For structural consistency rewards, R key As a critical element hit reward, R confidence The confidence calibration reward is represented by w1, w2, and w3, which are the structural consistency reward, key element hit reward, and confidence calibration reward, respectively.

[0065] Next, for step 104, the advantage estimate is calculated based on the composite reward value, and the objective function of the near-end policy optimization algorithm is constructed using the advantage estimate. The parameters of the policy network are updated to maximize the objective function.

[0066] In this embodiment of the invention, after obtaining the composite reward value, the composite reward value is not directly used for gradient descent. Instead, the composite reward value is transformed into a unified advantage signal that can measure the quality of the action. Then, the network is updated safely and effectively by constructing a proxy objective function.

[0067] In this embodiment of the invention, calculating the advantage estimate based on the composite reward value may specifically include: using the composite reward value as the reward value for time step t in the time difference error calculation to calculate the advantage estimate.

[0068] Specifically, the advantage estimate is calculated as follows: Step 1040: Calculate the time difference error using the following formula. δ t ; in, r t For time step t The compound reward value; γ As a discount factor, give higher weight to recent rewards; V ( s (For value network state) s The valuation represents the expected cumulative return starting from that state.

[0069] In this embodiment of the invention, r t It is not a single-category reward, but is jointly determined by structural consistency rewards, key element hit rewards, confidence calibration rewards, and operational objectives.

[0070] Step 1042: Perform an exponentially weighted summation of all future time difference errors to obtain the dominance estimate.

[0071] After obtaining the advantage estimate, the objective function of the proximal policy optimization algorithm (PPO) can be constructed using the advantage estimate. This objective function can be: in, r t ( θ E is the probability ratio of the new strategy to the old strategy. t As expected, A t For the sake of advantage estimation, This is a hyperparameter.

[0072] In the PPO algorithm, model parameters are updated using the policy gradient method. Specifically, the model continuously adjusts its network weights based on the objective function calculated above, thereby optimizing the policy and improving the model's performance in malware detection tasks.

[0073] The update of the model parameters includes at least one of the following: sub-item reward weights, temperature scaling factor, target confidence threshold, and PPO constraint parameters.

[0074] In one embodiment of the present invention, in order to enable the model to converge quickly, dynamic weights can be used, specifically: In the early stages of training, the accuracy reward of the classification result is given a higher weight; in the middle stages of training, the weight of the structural consistency reward and the key element hit reward is increased; in the later stages of training, the weight of the confidence calibration reward is increased to finely optimize the operational metrics.

[0075] After the model is trained, the trained policy network is used as the malware detection model in this embodiment of the invention to identify the software to be processed.

[0076] In the embodiments of the present invention, at least the following beneficial effects are achieved: By introducing structural consistency rewards, key element hit rewards, and confidence calibration rewards, the malware detection model not only possesses high-precision classification capabilities but also provides clear and logically consistent explanations, enhancing its interpretability. By linking operational metrics with the model training process, optimizations closely aligned with actual business needs are achieved. This linked optimization ensures the model's practical application effectiveness, thereby reducing false positives and false negatives and improving detection reliability. The introduction of reinforcement learning algorithms enables the model to adaptively adjust to different types of malware, improving its flexibility and adaptability, especially when handling new and unseen samples.

[0077] Please refer to Figure 2 This invention provides a training device for a malware detection model based on the linkage of sub-item rewards and operational indicators. The device includes: The construction unit 200 is used to construct a reinforcement learning model containing a policy network; the policy network is configured to receive software feature input and output multimodal actions including classification results, structured explanatory text, and detection confidence. The computing unit 202 is used to generate multimodal actions based on training samples using the policy network, and to calculate a composite reward value for the generated multimodal actions using pre-set sub-rewards; the sub-rewards include at least: a structural consistency reward for evaluating whether the generated structured explanatory text conforms to the set structural specifications, a key element hit reward for evaluating the key element matching degree, and a confidence calibration reward for calibrating the generated detection confidence to approximate the preset operational indicators. The update unit 204 is used to calculate the advantage estimate based on the composite reward value, and to construct the objective function of the near-end policy optimization algorithm using the advantage estimate, and to maximize the objective function by updating the parameters of the policy network. The iteration unit 206 is used to trigger the repeated execution of the computation unit and the update unit to iteratively optimize the policy network until a trained malware detection model is obtained.

[0078] In one embodiment of the present invention, the step of calculating the composite reward value for the generated multimodal action using pre-set sub-rewards includes: calculating the structural consistency reward, key element hit reward, and confidence calibration reward based on the multimodal action, and calculating the composite reward value using the structural consistency reward, key element hit reward, and confidence calibration reward and their weight values.

[0079] In one embodiment of the present invention, the structural consistency reward is calculated as follows: the structured explanatory text generated by the policy network is compared with the target text, an order consistency score is generated based on whether the field order is consistent, and an integrity score is generated based on the coverage of the generated fields to all fields in the target text; the structural consistency reward is calculated based on the order consistency score and the integrity score.

[0080] In one embodiment of the present invention, the key element hit reward is calculated as follows: identifying and extracting malware family tags and attack behaviors contained in the structured explanatory text generated by the policy network; matching the extraction results with real targets; and calculating the key element hit reward using the precision and recall of the matching.

[0081] In one embodiment of the present invention, the confidence calibration reward is calculated as follows: a target confidence threshold is determined based on preset operational indicators; the operational indicators include false positive rate and true positive rate; and a confidence calibration reward is calculated based on the deviation between the generated detection confidence and the target confidence threshold.

[0082] In one embodiment of the present invention, the policy network is initialized using a pre-trained malware detection model.

[0083] It should be noted that the malware detection model training device based on the linkage of sub-item rewards and operational indicators provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the malware detection model training device based on the linkage of sub-item rewards and operational indicators provided in the above embodiments belongs to the same concept as the malware detection model training method embodiment based on the linkage of sub-item rewards and operational indicators. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0084] Embodiments of this application also provide a computer device, please refer to... Figure 3 The computer device includes a processor and a memory, the memory storing at least one instruction, at least one program, code set, or instruction set, wherein at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor to implement the malware detection model training method based on the linkage of sub-item rewards and operational indicators provided in the above-described method embodiments.

[0085] The embodiments of this application also provide a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the malware detection model training method based on the linkage of sub-item rewards and operational indicators provided in the above-described method embodiments.

[0086] Embodiments of this application also provide a computer program product, which includes a computer program. A processor of a computer device reads the computer program from a computer-readable storage medium and executes the computer program, causing the computer device to perform any of the malware detection model training methods based on the linkage of sub-item rewards and operational indicators described in the above embodiments.

[0087] For ease of description, the above systems or devices are described separately as various modules or units based on their functions. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware components.

[0088] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0089] Finally, it should be noted that in this document, relational terms such as first, second, third, and fourth are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0090] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for training a malware detection model based on the linkage between sub-item rewards and operational indicators, characterized in that, The method includes: 100: Construct a reinforcement learning model that includes a policy network; the policy network is configured to receive software feature input and output multimodal actions including classification results, structured explanatory text, and detection confidence. 102: Based on the training samples, multimodal actions are generated using the policy network, and a composite reward value is calculated for the generated multimodal actions using pre-set sub-rewards; the sub-rewards include at least: a structural consistency reward for evaluating whether the generated structured explanatory text conforms to the set structural specifications, a key element hit reward for evaluating the key element matching degree, and a confidence calibration reward for calibrating the generated detection confidence to approximate the preset operational indicators. 104: Calculate the advantage estimate based on the composite reward value, and use the advantage estimate to construct the objective function of the near-end policy optimization algorithm. Maximize the objective function by updating the parameters of the policy network. 106: Repeat steps 102 to 104 to iteratively optimize the policy network until a trained malware detection model is obtained.

2. The method according to claim 1, characterized in that, The calculation of the composite reward value for the generated multimodal action using pre-set sub-rewards includes: The structural consistency reward, key element hit reward, and confidence calibration reward are calculated based on multimodal actions, and the composite reward value is obtained by using the structural consistency reward, key element hit reward, and confidence calibration reward and their weight values.

3. The method according to claim 2, characterized in that, The structural consistency reward is calculated as follows: The structured explanatory text generated by the policy network is compared with the target text. An order consistency score is generated based on whether the field order is consistent. An integrity score is generated based on the coverage of the generated fields to all fields in the target text. The structural consistency reward is calculated based on the sequence consistency score and the integrity score.

4. The method according to claim 2, characterized in that, The calculation method for the key element hit reward is as follows: Identify and extract malware family tags and attack behaviors contained in the structured explanatory text generated by the policy network; The extracted results are matched with the real target, and the key element hit reward is calculated using the precision and recall of the matching.

5. The method according to claim 2, characterized in that, The confidence calibration reward is calculated as follows: The target confidence threshold is determined based on preset operational indicators; these operational indicators include the false positive rate and the true positive rate. The confidence calibration reward is calculated based on the deviation between the generated detection confidence score and the target confidence score threshold.

6. The method according to any one of claims 1-5, characterized in that, The policy network is initialized using a pre-trained malware detection model.

7. A training device for a malware detection model based on the linkage of sub-item rewards and operational indicators, characterized in that, The device includes: A building unit is used to build a reinforcement learning model containing a policy network; the policy network is configured to receive software feature input and output multimodal actions including classification results, structured explanatory text, and detection confidence. The computing unit is used to generate multimodal actions based on training samples using the policy network, and to calculate a composite reward value for the generated multimodal actions using pre-set sub-rewards; the sub-rewards include at least: a structural consistency reward for evaluating whether the generated structured explanatory text conforms to the set structural specifications, a key element hit reward for evaluating the key element matching degree, and a confidence calibration reward for calibrating the generated detection confidence to approximate the preset operational indicators. The update unit is used to calculate the advantage estimate based on the composite reward value, and to construct the objective function of the near-end policy optimization algorithm using the advantage estimate, and to maximize the objective function by updating the parameters of the policy network. An iterative unit is used to trigger the repeated execution of the computation unit and the update unit to iteratively optimize the policy network until a trained malware detection model is obtained.

8. A computer device, characterized in that, The computer device includes a memory and a processor. The memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory to implement the steps of the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the method described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1-6.

Images