Quality control of malware detection

Abstract

To provide a method and system that continuously develop an internal threat scan engine based on an iterative quality assessment.SOLUTION: A method iteratively implements a dynamic assessment of a quality of threat detection with a frequency defined for each of a plurality of objects in an object collection, and includes: iteratively implementing a dynamic assessment including an internal scan result of the plurality of objects and external scan result thereof, and a consistency verdict of these results; changing a frequency of a scan iteration of the plurality of objects on the basis of the consistency verdict; classifying the plurality of objects on the basis of a result of the dynamic assessment; and, creating a development task including the internal and external scan results of the plurality of objects, meta-data on the plurality of objects and a plurality of automated test results. to provide a plurality of detailed contents for developing a software to fix inconsistency of the internal and external scan results of the plurality of objects.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to computer security including a quality management system and method for a virus countermeasure engine.

Background Art

[0002] An important indicator of the quality of security products is the level of threat detection, and conventionally, depending on the type of detection engine, it is evaluated using a specific test collection of objects such as files and hyperlinks.

[0003] The test is part of the release cycle of the security application and is performed at a specific stage of the release of the application. After the application is released, the product is tested by a test institution to perform a benchmark test or to check for errors when releasing a new version of the detection database so that there are no false detections or errors that affect the performance of the product.

[0004] This approach manages the quality of the product only at a specific point in time and cannot monitor the quality of detection in real time. In addition to the test procedures of the product, the quality of detection is also affected by the source, URL, and other objects of the files used to add to the databases of threat files and clean files. The quality of these collections (i.e., the reliability of the classification of the objects in the collection, the size of the collection, and the number of new files that were not previously known in the collection) directly determines the level of false detection, the level of detection, and the system performance by optimizing the processing of the filtering techniques of the detector that can reduce the time-consuming processing for the dynamic analysis of the objects on the user's end machine.

[0005] Traditional methods and systems test the quality of threat detection by the engine without evaluating the dynamics of quality metrics compared to third-party solutions and services, making it impossible to perform a relative assessment of detection quality in the future and identify areas for improvement.

[0006] Such systems and methods require manually retrieving various collections of objects and running tests on them, but they do not necessarily provide objective evaluations and results that can improve the product without additional analysis.

[0007] Therefore, in order to manage the process of adding to the detection database and to implement product updates accompanying modifications to the detection mechanism, it is necessary to address the issue of dynamic analysis of multiple detection metrics across various versions of the product, decoding engine, and file collection source. [Overview of the Initiative]

[0008] An exemplary feature of the present invention is to provide an iterative dynamic assessment of the quality of detection by periodically scanning multiple objects from multiple collections of various sources with each of the supported products and / or scan engines, and comparing the multiple verdicts with the multiple verdicts of multiple third-party scan services.

[0009] Based on multiple evaluation results, the confidence levels of multiple sources across multiple collections are determined, multiple classes are assigned to multiple objects detected by multiple products and multiple engines with lower quality levels than the supporters of multiple services, and tasks are created to align multiple rules across multiple detection engines and multiple products. This process is repeated periodically to achieve a specified quality of threat detection.

[0010] The exemplary invention provides a method for continuously developing an internal threat scanning engine based on iterative quality evaluations. The method comprises iteratively performing a dynamic assessment of the quality of threat detection for each of several objects in a collection of objects at a defined frequency, wherein the results of the dynamic assessment include internal and external scan results for several objects, and a determination of the consistency between the internal and external scan results for several objects; iteratively performing a dynamic assessment, which includes the results of internal and external scan results for several objects, and a determination of the consistency between the internal and external scan results for several objects; changing the frequency of iterative scanning of several objects based on the determination of the consistency between the external and internal scan results for several objects; classifying several objects based on the results of the dynamic assessment; creating development tasks which include internal and external scan results for several objects, metadata for several objects, and the results of several automated tests, in order to provide several details for developing software that resolves discrepancies between the internal and external scan results for several objects; controlling the dynamic assessment in accordance with the dynamic execution of the development tasks; and maintaining the quality of threat detection at a given level based on the controlled dynamic assessment and the priority of the development tasks.

[0011] The exemplary invention also provides a system for continuously developing an internal threat scanning engine based on iterative quality assessments. This system includes a processor coupled to memory that stores multiple instructions. The processor is configured to iteratively perform a dynamic evaluation of the quality of threat detection at a defined frequency for each of multiple objects in a collection of objects, the results of which include internal and external scan results for multiple objects, and a determination of the consistency between the internal and external scan results for multiple objects; to change the frequency of iterative scanning of multiple objects based on the determination of the consistency between the external and internal scan results for multiple objects; to classify multiple objects based on the results of the dynamic evaluation; to create development tasks that include internal and external scan results for multiple objects, metadata for multiple objects, and the results of multiple automated tests, in order to provide multiple details for developing software to resolve discrepancies between the internal and external scan results for multiple objects; to control the dynamic evaluation in accordance with the dynamic execution of the development tasks; and to maintain the quality of threat detection at a given level based on the priority of the controlled dynamic evaluation and development tasks.

[0012] The exemplary system can increase the speed and prioritize the classification of multiple objects detected at a lower quality level, and the iterative improvement of that quality. Furthermore, it is possible to test the detection database and engine by taking into account the confidence levels for multiple collections of multiple objects from multiple separate sources and for multiple separate classes of multiple objects. [Brief explanation of the drawing]

[0013] Exemplary aspects of the present invention will be better understood from the following detailed description of exemplary embodiments of the present invention with reference to the drawings. [Figure 1] Figure 1 is a flowchart illustrating an exemplary procedure for an iterative detection quality evaluation process according to an exemplary embodiment. [Figure 2]Figure 2 shows an exemplary system for an iterative detection quality evaluation process according to an exemplary embodiment. [Figure 3] Figure 3 shows an exemplary entity relationship diagram according to an exemplary embodiment. [Figure 4] Figure 4 shows a schematic diagram of an exemplary database according to an exemplary embodiment. [Modes for carrying out the invention]

[0014] Next, several exemplary embodiments of the present invention will be described with reference to several drawings. As illustrated in Figure 1, process 101 for the iteration of detection quality assessment includes receiving objects from the object collection, as shown in step 102. The objects are scanned by the internal scan engine in step 104 and by the external scan engine in step 103.

[0015] In step 105, the external scan results and object metadata, including the time of at least the first time object detection, are obtained, and in step 106, the internal scan results and object metadata are obtained.

[0016] Metadata, for example, is related to VirusTotal and includes a complete list of the engines used and a list of existing privileges. Because metadata is included in the scan results, either fully or partially, it can be used for object classification.

[0017] In step 107, a composite verdict is obtained, which includes the internal and external scan results from the internal and external scan engines, along with object metadata.

[0018] Step 108 involves updating object history information using a composite iteration determination, which is evaluated in step 109. If the combined determination is found to be a match in step 110, in step 112, task "A" is scheduled for the next iteration; in step 118, a list of object sources that provided the object is obtained; and in step 119, the confidence level of the object sources is updated based on the completed iterations.

[0019] A composite verdict is considered a mismatch if several pairs of verdict attributes are not equal. For example, this could be due to different scan verdicts (e.g., malicious, black, suspicious, grey, and white), different security ratings (numerical values), different threat classes (Trojan, virus, ransomware), or the inclusion of additional data in the metadata.

[0020] If the composite determination is found to be inconsistent in step 110, in step 111 the object is classified with other objects in the collection based on the composite determination, in step 113 several automated tests are run to verify the result inconsistency regarding the object's class, and in step 114 the object history information is updated with the results of the multiple automated tests.

[0021] If it is determined in step 115 that the internal determination is correct, the process proceeds to step 112, and task "B" is scheduled for the next iteration. The scheduled time, the scope of the external engine and the internal engine, and the object collection are set separately for tasks "A" and "B".

[0022] If it is determined in step 115 that the internal determination is inaccurate, in step 116, a scan engine development task is created based on the composite determination and multiple automated test results, and in step 118, by obtaining a list of object sources that provided the objects, the task is scheduled for the next iteration in step 117.

[0023] Figure 2 shows an exemplary system for iterative inspection quality evaluation. As shown in Figure 2, the composite verdict analyzer 205 receives an object 209 and exchanges information with an iteration scheduler 206, an external scan manager 203, an internal scan manager 204, and an object history 207 by generating a task 208 provided to a development operation system 210.

[0024] The iteration scheduler 206 interacts with the external scan manager 203, the internal scan manager 204, and a plurality of automated tests 219.

[0025] More specifically, the iteration scheduler 206 controls the flow of scans during the iteration and schedules the next iteration based on the composite determination. The internal scan manager 204 communicates with security applications 216, 217 and the internal scan engine 218 by sending multiple commands or objects to multiple internal scan engines or products using an API, command line, or other interface to retrieve multiple current scan results.

[0026] The external scan manager 203 communicates with the external scan engines 201 and 202 by sending multiple commands or objects to multiple third-party tools and engines using an API, command line, or other interface to retrieve multiple current scan results.

[0027] Object 209 is supplied by multiple object collections 211, including object collections 212 and 213 that communicate with object sources 214 and 215. Object 209 may include multiple files (e.g., scripts, executable files, documents, web pages, etc.), URLs, IP addresses, domain names, etc.

[0028] In an exemplary embodiment, the system shown in Figure 2 includes a computer processor connected to memory that stores multiple instructions. The processor is configured to repeatedly perform a dynamic assessment of the quality of threat detection at a defined frequency for each of multiple objects 209 within multiple object collections 211.

[0029] The results of the dynamic evaluation include the internal and external scan results of multiple objects by internal scan engine 1 and external scan engine 2, respectively, and consistency verdict between the internal and external scan results of multiple objects 209.

[0030] The system changes the frequency of iterations of scanning multiple objects 209 based on the agreement between the external and internal scan results of multiple objects evaluated by the combined judgment analyzer 205.

[0031] The system further classifies multiple objects 209 based on the results of dynamic evaluation and creates a development task 208 that includes internal and external scan results for object 209, metadata for multiple objects, and results of multiple automated tests, in order to provide multiple details for developing software to fix discrepancies between the internal and external scan results of multiple objects.

[0032] The system also controls dynamic evaluation in accordance with the dynamic implementation of development task 208, and maintains a given level of threat detection quality based on the controlled dynamic evaluation and the prioritization of development task 208.

[0033] Figure 3 shows an exemplary entity-relationship diagram of an exemplary malicious code detection quality control system for an antivirus engine.

[0034] In an exemplary embodiment of the present invention, the system automatically receives information about malicious and clean files from various sources (e.g., VirusTotal feed, MalaShare, workstations, etc.). The system automatically classifies multiple files into multiple categories, such as black, white, and gray, and enables obtaining a confusion matrix and its time dependency for multiple verdict providers across different datasets.

[0035] A set of file appearances and last scan dates, a dataset, and any combination of these parameters are available, providing a collection and storage of all historical information (e.g., as scan results). This system enables metric visualization and operates in real time (i.e., with only a delay of the time required for classification).

[0036] In the process described above, as illustrated in Figure 3, the data source preferably includes a resource or object that enables the reception of information about malicious and clean files, including multiple new files. The dataset is obtained from a specific data source, and the determination provider includes an antivirus engine installed on VirusTotal, BitDefender, CleanSet, or another source that provides useful information about the files. This task includes a program that performs several processes, including receiving new files, scanning them, receiving determinations, and saving the results to a database, with detection quality including the values ​​of the confusion matrix.

[0037] The multiple entities shown in the illustrative Figure 3 include a data source, a file, a decision provider, and a decision. A data source has two attributes: "id" and "name". It can generate multiple files (for example, using only hashes) and fill in all required fields. A single data source can generate many files. Simultaneously, a single file can be generated by multiple different data sources (i.e., with different IDs).

[0038] The file has multiple properties. All required fields are populated by the data source, and optional fields can be updated by decisions. The file can be scanned by multiple different decision providers.

[0039] The determination provider has two attributes, "id" and "name," and provides determinations. Considering the temporal dependency of determinations, a particular file may be scanned multiple times to obtain multiple determinations.

[0040] The determination has four attributes, including "id", "file_id", "scan_date (date of scan)", and "result (result)", which are entered by the determination provider. The task controls the entire system. First, multiple new files are generated by multiple data sources. Next, multiple files of interest are selected from all the files (for example, multiple recently received new files). Then, the selected files are rescanned by multiple decision providers, and multiple new decisions are received.

[0041] The entity relationship diagram shown in Figure 3 provides a clearer illustration of the database schematic diagram in Figure 4. Figure 4 shows an existing table and its fields, which can provide the following forms of data sources, files, decision providers, and decisions.

[0042] Data source Includes attributes of the data source.

[0043] [Table 1] file Includes attributes of the data source.

[0044] [Table 2] Decision provider This includes the attributes of the decision provider.

[0045] [Table 3] judgement The judgment attribute is included.

[0046] [Table 4] In an exemplary embodiment, when the database is created, four empty tables are created. The judgment table is created with three columns ("id", "file_id", and "scan_date"). When a record is added to the judgment provider table, a column for the corresponding name is created in the judgment table.

[0047] Regarding the component model, MDQCS includes the following components: 1. Data Sources 2. VerdictProviders 3. DetectDB (Detection Database) 4. Tasks.

[0048] The following tables describe the role of each component.

[0049] [Table 5] DataSources

[0050] [Table 6] VerdictProviders

[0051] [Table 7] DetectDB

[0052] [Table 8] Regarding metrics, the appearance time can be considered as both the time of reception by the system (i.e., the first transmission) and the time of appearance in the VT system (i.e., the first appearance).

[0053] All metrics are calculated as follows: For various datasets {dataset='dataset_name'} For the average of all datasets {dataset='average'} The weighted mean of all datasets {dataset='weighted'}.

[0054] Regarding the time dependency of detection quality, detection quality is calculated up to a set time limit, based on the day the file appeared, the following day, and so on. Regarding averaging previous time-series metrics over a specified period, one option is to average detection quality across zero-day files, next-day files, and so on.

[0055] The average detection quality for multiple files is received over a specified period. The number of files from a particular source is recognized by other sources, and how often they appear depends on the following: Local (white) files and BD cleanset; MalShare and VT; and Local files and VT.

[0056] The descriptions of various exemplary embodiments of the present invention are presented for illustrative purposes only and are not intended to exhaust or limit the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used herein have been selected to best describe the principles of the embodiments, their practical applications to marketable technologies or technical improvements, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0057] Furthermore, the applicant's intent is to include equivalents of all elements of the claims, and any amendment to any claim of this application should not be construed as a waiver of any interest or right to equivalents of any element or feature of the amended claim.

Claims

1. A method for continuously developing an internal threat scanning engine based on iterative quality assessments, The process of iteratively performing a dynamic evaluation of the quality of threat detection for each of a plurality of objects in an object collection at a defined frequency, wherein the results of the dynamic evaluation include internal scan results and external scan results for the plurality of objects, and a determination of the consistency between the internal scan results and the external scan results for the plurality of objects. Based on the matching determination between the external scan results and the internal scan results of the plurality of objects, the frequency of repetition of scanning the plurality of objects is changed. Classifying the multiple objects based on the results of the dynamic evaluation, To provide several details for developing software that resolves discrepancies between the internal scan results and external scan results of the aforementioned multiple objects, a development task is created that includes the internal scan results and external scan results of the aforementioned multiple objects, the metadata of the aforementioned multiple objects, and the results of several automated tests. Controlling the dynamic evaluation in accordance with the dynamic execution of the development task, A method comprising maintaining the quality of threat detection at a given level based on the controlled dynamic evaluation and the prioritization of the development tasks.

2. The method according to claim 1, wherein if the plurality of objects include multiple new files, the frequency of iterations of scanning the plurality of objects is changed.

3. The method according to claim 2, wherein if the internal scan results and the external scan results of the plurality of objects differ, the frequency of iterations of scanning the plurality of objects is further changed.

4. The method according to claim 1, wherein when changing the frequency of the iteration of the scan of the plurality of objects, if the plurality of objects are multiple new files and the internal scan result and the external scan result of the plurality of objects are different, the frequency of the iteration of the scan of the plurality of objects is increased.

5. The method according to claim 1, wherein, if the internal scan results and external scan results of the plurality of objects differ, the frequency of repetition of the scan of the plurality of objects is greater than the frequency of repetition of the scan of the plurality of objects when the internal scan results and external scan results of the plurality of objects match.

6. The method according to claim 1, further comprising receiving information about the plurality of objects regarding malicious and clean files from a plurality of sources before changing the frequency of the iteration of scanning the plurality of objects.

7. The method according to claim 1, wherein the internal scan results and external scan results of the plurality of objects include file occurrence, last scan date, dataset, and history information.

8. The method according to claim 1, wherein the plurality of objects are from multiple collections of various sources by supported products and a scan engine.

9. Classifying the aforementioned multiple objects is The method according to claim 1, further comprising comparing the determination of the internal scan results and external scan results of the plurality of objects with information from a plurality of third-party scan services relating to the plurality of objects.

10. The method according to claim 1, wherein the iterative performance of the dynamic evaluation is continued repeatedly to achieve a predetermined value for the quality of the threat detection.

11. A system for continuously developing an internal threat scanning engine based on iterative quality evaluations, A processor coupled to a memory that stores multiple instructions, wherein the processor is The process of iteratively performing a dynamic evaluation of the quality of threat detection for each of a plurality of objects in an object collection at a defined frequency, wherein the results of the dynamic evaluation include internal scan results and external scan results for the plurality of objects, and a determination of the consistency between the internal scan results and the external scan results for the plurality of objects. Based on the matching determination between the external scan results and the internal scan results of the plurality of objects, the frequency of repetition of scanning the plurality of objects is changed. Classifying the multiple objects based on the results of the dynamic evaluation, To provide several details for developing software that resolves discrepancies between the internal scan results and external scan results of the aforementioned multiple objects, a development task is created that includes the internal scan results and external scan results of the aforementioned multiple objects, the metadata of the aforementioned multiple objects, and the results of several automated tests. Controlling the dynamic evaluation in accordance with the dynamic execution of the development task, A system configured to maintain a given level of threat detection quality based on controlled dynamic evaluation and the prioritization of development tasks.

12. If the plurality of objects include a plurality of new files, the processor changes the frequency of the iteration of scanning the plurality of objects, according to claim 11.

13. The system according to claim 12, wherein if the internal scan results and external scan results of the plurality of objects differ, the processor further changes the frequency of iterations of scanning the plurality of objects.

14. The system according to claim 11, wherein when changing the frequency of the iteration of the scan of the plurality of objects, if the plurality of objects are multiple new files and the internal scan result and the external scan result of the plurality of objects are different, the processor increases the frequency of the iteration of the scan of the plurality of objects.

15. The system according to claim 11, wherein if the internal scan results and external scan results of the plurality of objects differ, the frequency of iteration of the scan of the plurality of objects is greater than the frequency of iteration of the scan of the plurality of objects when the internal scan results and external scan results of the plurality of objects match.

16. The system according to claim 11, wherein before changing the frequency of the iteration of the scan of the plurality of objects, the processor receives information about the plurality of objects regarding malicious files and clean files from the plurality of sources.

17. The system according to claim 11, wherein the internal scan results and external scan results of the plurality of objects include file occurrence, last scan date, dataset, and historical information.

18. The system according to claim 11, wherein the plurality of objects are from multiple collections of various sources by supported products and a scan engine.

19. Classifying the aforementioned multiple objects is The system according to claim 11, further comprising comparing the determination of the internal scan results and external scan results of the plurality of objects with information from a plurality of third-party scan services relating to the plurality of objects.

20. The system according to claim 11, wherein the processor iteratively continues the dynamic evaluation to achieve a preset value for the quality of threat detection.

Images