Method and apparatus for evaluating antivirus engine

Abstract

A method and apparatus for evaluating an antivirus engine may provide a method for evaluating an antivirus engine comprising: a step of generating an adversarial example by applying at least one perturbation to malicious code; a step of obtaining a virus detection result by inputting a normal file, the malicious code, and the adversarial example into a plurality of antivirus engines; and a step of evaluating the detection instability of each of the plurality of antivirus engines based on the virus detection result.

Description

Technology Field

[0001] The present invention relates to a method and apparatus for evaluating an antivirus engine, and more specifically, to a method and apparatus for generating an adversarial example using a semantic preserving perturbation and evaluating the detection instability of an antivirus engine based thereon. Background Technology

[0002] Although antivirus (AV) engines have evolved over the years, they exhibit detection instability, with detection results easily fluctuating in response to obfuscation, conditional execution, packing, and perturbations that manipulate malware to generate new forms. In particular, perturbations that maintain functionality—such as malware section rearrangement, instruction modification, and feature shifting—can destabilize the detection of commercial AVs and pose a threat where detection results can easily vary depending on the type of perturb, timing, and environment.

[0003] Therefore, existing disturbances are not suitable for evaluating or improving the detection instability of antivirus engines, and there is also a lack of evaluation tools that reflect the actual threat environment.

[0004] The aforementioned background technology is technical information that the inventor possessed for the derivation of the present invention or acquired during the process of deriving the present invention, and it cannot be considered as prior art disclosed to the general public prior to the filing of the present invention. The problem to be solved

[0005] The object of the present disclosure is to provide a method and apparatus for evaluating an antivirus engine. The problems to be solved by the present disclosure are not limited to the technical problems mentioned above, and other unmentioned technical problems will be clearly understood by those skilled in the art from the description of the present invention and will be more clearly understood by the embodiments of the present disclosure. Furthermore, it will be understood that the problems and advantages to be solved by the present disclosure can be realized by the means and combinations thereof set forth in the claims. means of solving the problem

[0006] As a means for solving the aforementioned technical problem, the first aspect of the present disclosure may provide a method for evaluating an antivirus engine, comprising: a step of generating an adversarial example by applying at least one perturbation to malicious code; a step of inputting a normal file, the malicious code, and the adversarial example into a plurality of antivirus engines to obtain a virus detection result; and a step of evaluating the detection instability of each of the plurality of antivirus engines based on the virus detection result.

[0007] A second aspect of the present disclosure may provide an apparatus for evaluating antivirus engines, comprising: a memory in which at least one program is stored; and a processor that performs operations by executing the at least one program, wherein the processor generates an adversarial example by applying at least one disturbance to malicious code, inputs a normal file, the malicious code, and the adversarial example into a plurality of antivirus engines to obtain a virus detection result, and evaluates the detection instability of each of the plurality of antivirus engines based on the virus detection result.

[0008] A third aspect of the present disclosure may provide a computer-readable recording medium that records a program for executing the method of the first aspect on a computer.

[0009] In addition to this, other methods for implementing the present invention, other devices, and computer-readable recording media recording a program for executing said method may be further provided.

[0010] Other aspects, features, and advantages other than those described above will become clear from the following drawings, claims, and detailed description of the invention. Effects of the invention

[0011] According to the means for solving the problem of the present disclosure described above, the detection instability of an antivirus engine can be precisely measured and compared.

[0012] In addition, according to the means for solving the problem of the present disclosure, detection instability of the antivirus engine can be improved by reflecting disturbances to the actual detection area and variations over time.

[0013] The effects of the embodiments are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the description of the present invention. Brief explanation of the drawing

[0014] FIG. 1 is a schematic diagram showing a system for evaluating an antivirus engine according to one embodiment. FIG. 2 is a diagram showing a part of the structure of an executable file to which a disturbance according to one embodiment can be applied. FIG. 3 is a diagram illustrating a method for evaluating detection instability of an antivirus engine according to one embodiment. FIG. 4 is an illustrative diagram showing the type of single disturbance according to one embodiment. FIG. 5 is a flowchart of a method for evaluating an antivirus engine according to one embodiment. FIG. 6 is an example diagram of the configuration of a device for evaluating an antivirus engine according to one embodiment. Specific details for implementing the invention

[0015] The terms used in the embodiments have been selected to be as widely used as possible, taking into account their functions in the present disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant explanatory section. Therefore, terms used in the present disclosure should be defined not merely by their names, but based on their meanings and the overall content of the present disclosure.

[0016] When a part of a specification is described as "comprising" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, terms such as "...part" or "...module" as used in the specification refer to a unit that processes at least one function or operation, and this may be implemented in hardware or software, or as a combination of hardware and software.

[0017] The expression "at least one of a, b, and c" described throughout the specification may include 'a alone', 'b alone', 'c alone', 'a and b', 'a and c', 'b and c', or 'a, b, and c all'.

[0018] The "terminal" mentioned below may be implemented as a computer or portable terminal capable of connecting to a server or other terminal via a network. Here, the computer includes, for example, a notebook, desktop, or laptop equipped with a web browser, and the portable terminal may include, for example, a wireless communication device that ensures portability and mobility, and may include all types of handheld-based wireless communication devices such as communication-based terminals like IMT (International Mobile Telecommunication), CDMA (Code Division Multiple Access), W-CDMA (W-Code Division Multiple Access), and LTE (Long Term Evolution), smartphones, tablet PCs, etc.

[0019] Embodiments of the present disclosure are described below with reference to the attached drawings so that those skilled in the art can easily implement them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein.

[0020] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0021] In describing the embodiments, technical details that are well known in the technical field to which the present invention belongs and are not directly related to the present invention are omitted. This is intended to convey the essence of the present invention more clearly without obscuring it by omitting unnecessary explanations.

[0022] For the same reason, some components in the attached drawings have been exaggerated, omitted, or schematically depicted. Additionally, the size of each component does not entirely reflect its actual dimensions. Identical or corresponding components in each drawing have been assigned the same reference numbers.

[0023] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but can be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. Throughout the specification, the same reference numerals refer to the same components.

[0024] At this time, it will be understood that each block of the process flow diagrams and combinations of the flow diagrams can be executed by computer program instructions. Since these computer program instructions can be loaded into the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing equipment, the instructions executed through the processor of the computer or other programmable data processing equipment create means to perform the functions described in the flow diagram block(s). Since these computer program instructions can also be stored in computer-available or computer-readable memory that can be directed toward the computer or other programmable data processing equipment to implement the function in a specific manner, the instructions stored in such computer-available or computer-readable memory can also produce a manufactured item containing instruction means to perform the function described in the flow diagram block(s). Since computer program instructions can be loaded onto a computer or other programmable data processing equipment, instructions that perform a series of operation steps on the computer or other programmable data processing equipment to create a process executed by the computer can also provide steps for executing the functions described in the flowchart block(s).

[0025] Additionally, each block may represent a module, segment, or part of code containing one or more executable instructions for executing a specified logical function(s). It should also be noted that in some alternative execution examples, the functions mentioned in the blocks may occur out of order. For instance, two blocks described in succession may actually be executed substantially simultaneously, or the blocks may be executed in reverse order according to their corresponding functions.

[0026] FIG. 1 is a schematic diagram showing a system for evaluating an antivirus engine according to one embodiment.

[0027] According to various embodiments, a system (100) for evaluating an antivirus engine includes an electronic device (110). According to an embodiment, the system (100) for evaluating an antivirus engine may further include an antivirus engine (120). Although only one antivirus engine (120) is shown in FIG. 1, there may be multiple antivirus engines (120). According to an embodiment, the system (100) for evaluating an antivirus engine may further include a network that supports the transmission and reception of information between the electronic device (110) and at least some of the one or more antivirus engines (120).

[0028] Each of the electronic device (110) and one or more antivirus engines (120) may include a transceiver, memory, and a processor. Additionally, each of the electronic device (110) and one or more antivirus engines (120) represents a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software. Meanwhile, throughout the embodiments, each of the electronic device (110) and one or more antivirus engines (120) is referred to as a separate device or server, but this may be a logically divided structure, and at least some of them may be implemented by functions separated from a single device or server.

[0029] According to one embodiment, the electronic device (110) and one or more antivirus engines (120) may include a number of computer systems or computer software implemented as network servers. For example, at least some of the electronic device (110) and one or more antivirus engines (120) may refer to computer systems and computer software that are connected to a sub-device capable of communicating with other network servers via a computer network such as an intranet or the Internet, receive requests for task execution, perform tasks thereon, and provide results. In addition, at least some of the electronic device (110) and one or more antivirus engines (120) may be understood in a broad sense as including a series of applications capable of operating on a network server and various databases built on internal or connected other nodes. For example, at least some of the electronic device (110) and one or more antivirus engines (120) may be implemented using network server programs provided in various ways depending on an operating system such as DOS, Windows, Linux, UNIX, or MacOS.

[0030] The electronic device (110) is a device that acquires and provides various information. The electronic device (110) can perform various tasks to evaluate the detection instability of one or more antivirus engines (120). Specifically, the electronic device (110) can generate an adversarial example by applying at least one disturbance to malicious code, input a normal file, malicious code, and the adversarial example into one or more antivirus engines (120) to obtain a virus detection result, and evaluate the detection instability of each of the one or more antivirus engines (120) based on the virus detection result.

[0031] According to one embodiment, the electronic device (110) may generate an adversarial example from malicious code and input the adversarial example together with a normal file and the original malicious code into one or more antivirus engines (120), and receive a virus detection result from one or more antivirus engines (120), but the scope of the present disclosure is not limited thereto.

[0032] According to one embodiment, the electronic device (110) can generate an adversarial example by applying at least one disturbance to malicious code.

[0033] FIG. 2 is a diagram showing a part of the structure of an executable file to which a disturbance according to one embodiment can be applied.

[0034] Antivirus engines may possess detection instability, where virus detection results fluctuate easily in response to perturbations that manipulate malicious code to generate new forms of malware, i.e., adversarial examples. In particular, perturbations that maintain the functionality of the malicious code—such as section rearrangement, instruction modification, or feature shifting—can destabilize the antivirus engine's detection and pose a risk that virus detection results may easily vary depending on the type, timing, and environment of the perturbation.

[0035] Referring to FIG. 2, a portion of the structural specifications of an executable file, such as a normal file, malicious code, or hostile example, according to the present specification is illustrated in an operating system (OS) according to one embodiment. This may include resources such as icons, fonts, and images required while the executable file is executed, and code representing the actual operation of the executable file, and may be composed of headers and corresponding sections.

[0036] The structure of the executable file may include not only headers such as the DOS header, rich header, Optional header, and COFF header shown in FIG. 2, but also sections such as an overlay, .text section, and .rsrc section.

[0037] In this case, applying disturbances to superficial areas that commercial antivirus engines typically do not detect—such as modifying certain fields that do not affect header execution, injecting noise into overlays, adding noise to padding between empty spaces and sections of the malicious code, changing section names, or adding meaningless sections—has the advantage that the malicious code can still be executed after application; however, since actual virus variants can emerge in new forms by altering the commands themselves or implementing the same behavior using different command structures, rather than just these superficial disturbances, this approach may not be suitable for evaluating the instability of the antivirus system.

[0038] Meanwhile, when a section-targeting disturbance is applied to malicious code, an adversarial example that bypasses the antivirus engine can be generated, and consequently, the detection instability of the antivirus engine can be analyzed and evaluated. In addition, since a section-targeting disturbance can also preserve the behavior of the original malicious code, an adversarial example generated by applying a single disturbance or a combination of at least two of multiple single disturbances can also perform the same behavior as the original malicious code.

[0039] FIG. 3 is a diagram illustrating a method for evaluating detection instability of an antivirus engine according to one embodiment.

[0040] According to one embodiment, the electronic device (110) can generate an adversarial example by applying at least one disturbance (320) to the malicious code (310). In this case, the at least one disturbance (320) may include a disturbance to a non-core area of ​​the malicious code (310) and a disturbance to a core area of ​​the malicious code (310).

[0041] According to one embodiment, a non-core area of ​​the malicious code (310) may include at least one of the header and metadata of the malicious code (310), and a core area may include at least one of the code section, resource section, and overlay of the malicious code (310). In this case, the code section (e.g., .text section) is a section containing the actual executable code of the malicious code (e.g., CPU instructions), the resource section (e.g., .rsrc section) is a section containing the icon, menu, version information, etc. of the malicious code, and the overlay may be an additional data section added to the last area of ​​the malicious code.

[0042] According to one embodiment, at least one disturbance (320) may include at least one of a header disturbance that applies a disturbance to a header, a section disturbance that applies a disturbance to a code section and / or resource section, and an overlay disturbance that applies a disturbance to an overlay. In this case, the disturbance to a non-core area of ​​the malicious code (310) may include a header disturbance, and the disturbance to a core area of ​​the malicious code (310) may include at least one of a section disturbance and an overlay disturbance.

[0043] Additionally, according to one embodiment, as described above, at least one disturbance (320) may be any one of a plurality of single disturbances or a combination of two or more of a plurality of single disturbances.

[0044] FIG. 4 is an illustrative diagram showing the type of single disturbance according to one embodiment.

[0045] Although only 14 single disturbances P1 to P14 are illustrated in FIG. 4, these are merely examples of disturbances for malicious code, and other single disturbances or combinations of two or more single disturbances may be used in addition to the single disturbances illustrated in FIG. 4 to generate an adversarial example of the present disclosure.

[0046] The single disturbances of P12 to P14 according to one embodiment are single disturbances that reflect the analysis results of a signature-based antivirus engine, and are disturbances designed based on the fact that most of the antivirus engine's overall detection relies on byte pattern signatures of content areas such as code sections, resource sections, and overlays.

[0047] According to one embodiment, single disturbances are applied to malicious code having a PE format and may be semantic-preserving disturbances that do not impair the behavior of the original malicious code. Using the disturbance according to one embodiment of the present disclosure, adversarial examples can be generated faster than methods based on reinforcement learning or adding simple noise, and various adversarial examples can be generated by combining multiple single disturbances. Furthermore, since the disturbance according to one embodiment of the present disclosure directly disturbs the area primarily detected by the antivirus engine, the detection instability of the antivirus engine can be realistically evaluated.

[0048] Returning to FIG. 3, according to one embodiment, an electronic device (110) can input a normal file (330), malicious code (310), and hostile examples into a plurality of antivirus engines (e.g., AV1, AV2 of FIG. 3) to obtain a virus detection result (340). As an example, the electronic device (110) can input the normal file (330), malicious code (310), and hostile examples into a malicious code detection platform, such as VirusTotal, in which a plurality of antivirus engines participate, to obtain detection results (340) from a plurality of antivirus engines. The detection result (340) may include the detection result of each of the plurality of antivirus engines (e.g., an analysis report for each file).

[0049] According to one embodiment, the electronic device (110) can quantitatively evaluate the detection instability of a plurality of antivirus engines by comparing and analyzing the detection results (340) of a plurality of antivirus engines, combining the virus detection effect of at least one disturbance (320) and the change in detection results over time.

[0050] According to one embodiment, the electronic device (110) can evaluate the detection instability (350) of each of a plurality of antivirus engines based on the virus detection result (340).

[0051] According to one embodiment, the electronic device (110) can calculate a first indicator (e.g., Attack Success Rate, ASR) representing the ratio of antivirus engines that determine an adversarial example to be normal among a plurality of antivirus engines from a virus detection result (340). The first indicator is the ratio of the generated adversarial example to the virus detection of the antivirus engine, and may be an indicator representing the detection evasion performance of at least one disturbance (320) applied to the adversarial example. For example, the first indicator (ASR) may be calculated according to the following mathematical formula 1.

[0052]

[0053] According to one embodiment, the electronic device (110) can calculate a second indicator (e.g., Label Change Rate, LCR) from a virus detection result (340) that represents the ratio of antivirus engines that have derived different judgments regarding malicious code (310) and hostile examples among a plurality of antivirus engines. The second indicator may be an indicator representing the detection consistency of the antivirus engine, which is the ratio of cases where a signature label different from that of malicious code (310) is assigned even when a hostile example is detected. For example, the second indicator (LCR) can be calculated according to the following mathematical formula 2.

[0054]

[0055] According to one embodiment, the electronic device (110) can calculate a third indicator (e.g., True Positive Rate, TPR) representing the ratio of antivirus engines that determine malicious code (310) as malicious among a plurality of antivirus engines from the virus detection result (340). The third indicator is the ratio of accurately detecting malicious code and may be an indicator representing the detection accuracy of the antivirus engine. For example, the third indicator (TPR) can be calculated according to the following mathematical formula 3.

[0056]

[0057] According to one embodiment, the electronic device (110) can calculate a fourth indicator (False Negative Rate, FPR) from a virus detection result (340), which represents the ratio of antivirus engines that determine a normal file (330) as malicious among a plurality of antivirus engines. The fourth indicator may be an indicator representing the false positive tendency of the antivirus engines or the conservatism of the detection policy. For example, the fourth indicator (FPR) may be calculated according to the following mathematical formula 4.

[0058]

[0059] In the above mathematical formulas 1 and 2, is malicious code (310), is an adversarial example, in mathematical equation 3 above is a malicious code (310) dataset ( Sample of ), from mathematical formula 4 above is a normal file (330) dataset ( Sample of ), in mathematical formulas 1 to 4 is the number of cases where the antivirus engine determines the input file to be malicious, is a label for the result (malicious or normal) detected by the antivirus engine for the input file.

[0060] According to one embodiment, the electronic device (110) can determine a single score or a combination of two or more of the first to fourth indicators as the evaluation result of the antivirus engine. By using the first to fourth indicators to repeatedly evaluate adversarial examples and malicious code over a long period, the electronic device (110) can reflect disturbances and variations over time in the actual detection area of ​​multiple antivirus engines, and can be utilized to improve detection instability.

[0061] FIG. 5 is a flowchart of a method for evaluating an antivirus engine according to one embodiment.

[0062] In step 510, according to one embodiment, the electronic device (110) can generate an adversarial example by applying at least one disturbance to malicious code.

[0063] According to one embodiment, at least one disturbance may be any one of a plurality of single disturbances or a combination of two or more of a plurality of single disturbances.

[0064] According to one embodiment, a plurality of single disturbances include a disturbance to a non-core area of ​​malicious code and a disturbance to a core area of ​​malicious code, wherein the non-core area includes at least one of a header and metadata of malicious code, and the core area may include at least one of a code section, a resource section, and an overlay of malicious code.

[0065] In step 520, according to one embodiment, the electronic device (110) can input normal files, malicious code, and hostile examples into a plurality of antivirus engines to obtain virus detection results.

[0066] In step 530, according to one embodiment, the electronic device (110) can evaluate the detection instability of each of a plurality of antivirus engines based on the virus detection results.

[0067] According to one embodiment, the electronic device (110) can calculate a first indicator representing the ratio of antivirus engines that determine an adversarial example as normal among a plurality of antivirus engines from the virus detection results.

[0068] According to one embodiment, the electronic device (110) can calculate a second indicator representing the ratio of antivirus engines that have derived different judgments regarding malicious code and hostile examples among a plurality of antivirus engines from the virus detection results.

[0069] According to one embodiment, the electronic device (110) can calculate a third indicator representing the ratio of antivirus engines that determined malicious code to be malicious among a plurality of antivirus engines from the virus detection results.

[0070] According to one embodiment, the electronic device (110) can calculate a fourth indicator representing the ratio of antivirus engines that determined a normal file to be malicious among a plurality of antivirus engines from the virus detection results.

[0071] FIG. 6 is an example diagram of the configuration of a device for evaluating an antivirus engine according to one embodiment.

[0072] Referring to FIG. 6, the electronic device (110) includes a processor (620) and a memory (630) and may further include a transceiver (610). The electronic device (110) can be connected to an antivirus engine (120) and other external devices, etc., through the transceiver (610) and exchange data.

[0073] The processor (620) may include at least one device described through FIGS. 1 to 5 or perform at least one method described through FIGS. 1 to 5. The memory (630) may store information for performing at least one method described through FIGS. 1 to 5. The memory (630) may be a volatile memory or a non-volatile memory.

[0074] The processor (620) can execute a program and control the electronic device (110). The code of the program executed by the processor (620) can be stored in memory (630).

[0075] Additionally, the electronic device (110) according to one embodiment may further include an interface capable of providing information to a user. In this case, the information may be a score determined by the first to fourth indicators calculated as described above and / or a combination thereof.

[0076] Meanwhile, the present specification and drawings disclose preferred embodiments of the present invention. Although specific terms have been used, they are used merely in a general sense to facilitate the explanation of the technical content of the present invention and to aid in understanding the invention, and are not intended to limit the scope of the present invention. It is obvious to those skilled in the art that, in addition to the embodiments disclosed herein, other variations based on the technical concept of the present invention are possible.

[0077] The electronic device or terminal according to the embodiments described above may include a processor, memory for storing and executing program data, permanent storage such as a disk drive, a communication port for communicating with an external device, and user interface devices such as a touch panel, a key, a button, etc. Methods implemented as software modules or algorithms may be stored on a computer-readable recording medium as computer-readable code or program instructions executable on the processor. Here, computer-readable recording media include magnetic storage media (e.g., ROM (read-only memory), RAM (random-access memory), floppy disks, hard disks, etc.) and optical reading media (e.g., CD-ROM, DVD (Digital Versatile Disc)). Computer-readable recording media may be distributed across networked computer systems, allowing computer-readable code to be stored and executed in a distributed manner. The medium may be readable by a computer, stored in memory, and executed by a processor.

[0078] The present embodiment may be represented by functional block configurations and various processing steps. These functional blocks may be implemented by various numbers of hardware and / or software configurations that execute specific functions. For example, the embodiment may employ integrated circuit configurations such as memory, processing, logic, look-up tables, etc., capable of executing various functions by the control of one or more microprocessors or other control devices. Similar to how components may be implemented as software programming or software elements, the present embodiment may be implemented in programming or scripting languages ​​such as C, C++, Java, assembler, Python, etc., including various algorithms implemented as combinations of data structures, processes, routines, or other programming configurations. Functional aspects may be implemented as algorithms executed on one or more processors. Additionally, the present embodiment may employ prior art for electronic configuration, signal processing, and / or data processing. Terms such as "mechanism," "element," "means," and "configuration" may be used broadly and are not limited to mechanical and physical configurations. The above terms may include the meaning of a series of software processes (routines) in conjunction with processors, etc.

[0079] The aforementioned embodiments are merely examples, and other embodiments may be implemented within the scope of the claims set forth below.

Claims

Claim 1 A method for evaluating antivirus engines, comprising: a step of generating an adversarial example by applying at least one perturbation to malicious code; a step of inputting a normal file, the malicious code, and the adversarial example into a plurality of antivirus engines to obtain a virus detection result; and a step of evaluating the detection instability of each of the plurality of antivirus engines based on the virus detection result, wherein the at least one perturbation is any one of a plurality of single perturbations or a combination of two or more of the plurality of single perturbations, and the plurality of single perturbations include a perturbation to a non-core region of the malicious code and a perturbation to a core region of the malicious code, wherein the non-core region includes at least one of a header and metadata of the malicious code, and the core region includes at least one of a code section, a resource section, and an overlay of the malicious code. Claim 2 delete Claim 3 delete Claim 4 A method for evaluating an antivirus engine according to claim 1, wherein the step of evaluating detection instability comprises the step of calculating a first indicator representing the ratio of antivirus engines that determine the adversarial example as normal among the plurality of antivirus engines from the virus detection result. Claim 5 A method for evaluating an antivirus engine according to claim 1, wherein the step of evaluating detection instability includes the step of calculating a second indicator representing the ratio of antivirus engines that derive different judgments regarding the malicious code and the adversarial example among the plurality of antivirus engines from the virus detection results. Claim 6 A method for evaluating an antivirus engine according to claim 1, wherein the step of evaluating detection instability includes the step of calculating a third indicator representing the ratio of antivirus engines that determined the malicious code to be malicious among the plurality of antivirus engines from the virus detection result. Claim 7 A method for evaluating an antivirus engine according to claim 1, wherein the step of evaluating detection instability includes the step of calculating a fourth indicator representing the ratio of antivirus engines that determined the normal file to be malicious among the plurality of antivirus engines from the virus detection result. Claim 8 An apparatus for evaluating antivirus engines, comprising: a memory in which at least one program is stored; and a processor that performs operations by executing the at least one program, wherein the processor generates an adversarial example by applying at least one disturbance to malicious code, inputs a normal file, the malicious code, and the adversarial example into a plurality of antivirus engines to obtain a virus detection result, and evaluates the detection instability of each of the plurality of antivirus engines based on the virus detection result, wherein the at least one disturbance is any one of a plurality of single disturbances or a combination of two or more of the plurality of single disturbances, and the plurality of single disturbances include a disturbance to a non-core area of ​​the malicious code and a disturbance to a core area of ​​the malicious code, wherein the non-core area includes at least one of a header and metadata of the malicious code, and the core area includes at least one of a code section, a resource section, and an overlay of the malicious code. Claim 9 A computer-readable recording medium storing a program for executing the method of claim 1 on a computer.

Images