Web vulnerability scanning method, device, electronic device and computer-readable storage medium

Abstract

The present application discloses a Web vulnerability scanning method, device, electronic device and computer-readable storage medium. The method comprises: obtaining the traffic to be scanned and writing the traffic to be scanned into the traffic message queue of a distributed publish-subscribe message system, wherein the traffic to be scanned includes Http traffic and Https traffic for external access to the target network area; normalizing the traffic to be scanned in the traffic message queue to obtain normalized traffic; deduplicating the normalized traffic according to the normalized reference traffic in the deduplication database, and writing the deduplicated traffic into the scanning message queue of the distributed publish-subscribe message system and the deduplication database respectively; performing Web vulnerability scanning according to the deduplicated traffic in the scanning message queue. The technical solution of the present application can improve scanning efficiency and reduce vulnerability scanning costs in traffic scenarios of a higher order of magnitude.

Description

Technical Field

[0001] The present application relates to the field of network security technology, and in particular to a Web vulnerability scanning method, device, electronic device, and computer-readable storage medium. Background Art

[0002] With the rapid development of information technology, cyberspace security issues have become increasingly prominent, and enterprises are facing various security threats from both outside and inside. Among them, vulnerabilities, as one of the main risk points of network security, pose a serious threat to the information systems of enterprises. Therefore, timely discovery and repair of vulnerabilities to ensure the safe and stable operation of information systems has become an urgent problem for enterprises.

[0003] The existing technology mainly implements fine-grained vulnerability scanning of URL parameters through triggered crawlers after users input IP and domain names. Such vulnerability scanning solutions have at least the following defects:

[0004] 1) Users need to have complete enterprise domain name assets, but the cost of collecting assets is high and the asset coverage rate is low;

[0005] 2) In large-scale website scenarios, triggered crawlers are needed to obtain URLs and parameters. The URL and parameter coverage is low, and isolated URLs cannot be obtained. At the same time, the server cost and time cost required by the crawler are high;

[0006] 3) In large-scale website scenarios, the amount of data to be scanned by triggered crawlers is huge. The accuracy of traffic deduplication based on statistics is low and the rate of false deduplication is high. The computational cost of scanning large amounts of data is high. Summary of the invention

[0007] The embodiments of the present application provide a Web vulnerability scanning method, device, electronic device, and computer-readable storage medium to improve scanning efficiency and reduce vulnerability scanning costs in a traffic scenario with a higher order of magnitude.

[0008] The present application embodiment adopts the following technical solutions:

[0009] In a first aspect, an embodiment of the present application provides a Web vulnerability scanning method, the Web vulnerability scanning method is executed by a vulnerability scanning server, and the method includes:

[0010] Obtaining the traffic to be scanned and writing the traffic to be scanned into the traffic message queue of the distributed publish-subscribe message system, wherein the traffic to be scanned includes Http traffic and Https traffic for external access to the target network area;

[0011] Normalizing the traffic to be scanned in the traffic message queue to obtain normalized traffic;

[0012] Performing traffic deduplication processing on the normalized traffic according to the normalized reference traffic in the deduplication database, and writing the deduplication traffic into the scanning message queue of the distributed publish-subscribe message system and the deduplication database respectively;

[0013] Web vulnerability scanning is performed according to the deduplicated traffic in the scanning message queue.

[0014] In a second aspect, an embodiment of the present application further provides a Web vulnerability scanning device, which is applied to a vulnerability scanning server, and the device includes:

[0015] A traffic acquisition unit, used for acquiring the traffic to be scanned and writing the traffic to be scanned into the traffic message queue of the distributed publish-subscribe message system, wherein the traffic to be scanned includes Http traffic and Https traffic for external access to the target network area;

[0016] A normalization unit, used for normalizing the traffic to be scanned in the traffic message queue to obtain normalized traffic;

[0017] A traffic deduplication unit, configured to perform traffic deduplication processing on the normalized traffic according to the normalized reference traffic in the deduplication database, and write the deduplication traffic into the scanning message queue of the distributed publish-subscribe message system and the deduplication database respectively;

[0018] A vulnerability scanning unit performs Web vulnerability scanning according to the deduplicated traffic in the scanning message queue.

[0019] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0020] A processor; and a memory arranged to store computer executable instructions, which, when executed, cause the processor to perform a Web vulnerability scanning method.

[0021] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple application programs, the electronic device performs a Web vulnerability scanning method.

[0022] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0023] The embodiment of the present application writes the traffic to be scanned into the traffic message queue of the distributed publish-subscribe message system, consumes the traffic to be scanned in the traffic message queue and normalizes it, compares the normalized traffic with the normalized reference traffic in the deduplication database, and completes the rapid deduplication of the traffic to be scanned. In a traffic scenario with a higher order of magnitude, it can improve the scanning efficiency and reduce the cost of vulnerability scanning. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0025] Figure 1 A schematic diagram of a process flow of a Web vulnerability scanning method in an embodiment of the present application;

[0026] Figure 2 This is a schematic diagram of a method for obtaining Http traffic in an embodiment of the present application;

[0027] Figure 3 This is a schematic diagram of traffic deduplication in an embodiment of the present application;

[0028] Figure 4 This is a structural block diagram of a Web vulnerability scanning device in an embodiment of the present application;

[0029] Figure 5 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0031] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0032] The present application embodiment provides a Web vulnerability scanning method, such as Figure 1 As shown, a flow chart of a Web vulnerability scanning method in an embodiment of the present application is provided. The Web vulnerability scanning method is performed by a vulnerability scanning server, and the vulnerability scanning server includes multiple traffic analysis servers. The Web vulnerability scanning method is specifically performed by each traffic analysis server. The method at least includes the following steps S110 to S140:

[0033] Step S110, obtaining the traffic to be scanned and writing the traffic to be scanned into the traffic message queue of the distributed publish-subscribe message system, wherein the traffic to be scanned includes Http traffic and Https traffic for external access to the target network area.

[0034] The distributed publish-subscribe messaging system in the embodiments of the present application can be a distributed publish-subscribe messaging system in the prior art, such as the Kafka system, or it can be a self-built distributed publish-subscribe messaging system, which can be used to cope with large data volumes and high concurrency scenarios.

[0035] It can be understood that the Http traffic in this embodiment refers to the data traffic transmitted through Http (Hyper Text Transfer Protocol), and the Https (encrypted Http) traffic refers to the data traffic transmitted through SSL (Secure Socket Layer) / TLS (Transport Layer Security) encrypted transmission. The target network area in this embodiment includes the Internet Data Center (IDC).

[0036] Step S120, normalizing the traffic to be scanned in the traffic message queue to obtain normalized traffic.

[0037] The normalization processing in the embodiment of the present application refers to standardizing the traffic to be scanned within a common range, so that the comparison between traffic data is more intuitive and accurate. The embodiment of the present application normalizes the traffic to be scanned through a multi-dimensional normalization model to obtain multiple hash values ​​of the traffic to be scanned in different dimensions, so as to provide a data basis for traffic deduplication with high performance, high deduplication rate and low false deduplication rate.

[0038] Among them, the multi-dimensional normalization model includes URL (Uniform Resource Locator) semantic rule model, compound URL semantic rule model, load configuration model, business content rule model, regular algorithm model, and Bloom algorithm model. Various models will be described in detail later.

[0039] Step S130, performing traffic deduplication processing on the normalized traffic according to the normalized reference traffic in the deduplication database, and writing the deduplication traffic into the scanning message queue of the distributed publish-subscribe message system and the deduplication database respectively.

[0040] The deduplication database in the embodiment of the present application is used to provide a normalized reference flow for traffic deduplication. In the initial stage, the deduplication database is empty, and the first normalized flow produced will be directly saved in the deduplication database as the first normalized reference flow. Thereafter, the second normalized flow produced is compared with the first normalized reference flow in the deduplication database. If the similarity between the second normalized flow and the first normalized reference flow is greater than a set value, the second normalized flow is removed. Otherwise, the second normalized flow is written into the scanning message queue and the deduplication database respectively, and the normalized flow produced next is deduplicated accordingly.

[0041] Step S140: Perform Web vulnerability scanning according to the deduplicated traffic in the scanning message queue.

[0042] This embodiment uses an extensible distributed scanning tool to consume the traffic in the scan message queue, and calls the scanning plug-in according to the task parameters to complete the vulnerability scan. For example: the scanning of XSS (Cross-Site Scripting) vulnerabilities will pollute the parameters of each field at risk in the Http request data packet, add the XSS attack Payload, and then determine whether the polluted XSS Payload is parsed in the Response of the Http request data packet to determine whether the current Http request data packet has a vulnerability. Those skilled in the art can refer to the existing solutions to perform Web vulnerability scanning, and this embodiment of the application will not be described in detail.

[0043] like Figure 1 It can be seen from the Web vulnerability scanning method shown that the embodiment of the present application writes the traffic to be scanned into the traffic message queue of the distributed publish-subscribe message system, consumes the traffic to be scanned in the traffic message queue and normalizes it, compares the normalized traffic with the normalized reference traffic in the preset database, and completes the rapid deduplication of the traffic to be scanned. In a traffic scenario with a higher order of magnitude, it can improve the scanning efficiency and reduce the vulnerability scanning cost.

[0044] In some embodiments of the present application, obtaining the traffic to be scanned in the above step S110 includes:

[0045] By performing traffic mirroring on the ingress switch of the target network area, the Http traffic of external access to the target network area is obtained;

[0046] And obtain the Https traffic for external access to the target network area through the Nginx proxy service and parse it, and obtain the parsed Https traffic for external access to the target network area.

[0047] by Figure 2Taking a certain computer room network scenario shown in the figure as an example, traffic mirroring is performed on the core switches at each entrance of the computer room, and the mirrored traffic port is connected to the splitter switch port using optical fiber, and all southbound traffic is sent to the splitter switch. On the splitter switch, the mirrored traffic is distributed to multiple traffic analysis servers on the vulnerability scanning server. The traffic analysis server uses the network card as the computing hardware and the DPDK (Data Plane Development Kit) program as the software to parse the Http and Https traffic in the traffic and write it into the traffic message queue of the Kafka system. In this way, in a large traffic scenario, the traffic to be scanned can be quickly obtained through streaming computing.

[0048] Among them, north-south traffic and east-west traffic are a network traffic description mode in a network data center environment. North-south traffic refers to the traffic between clients outside the network data center and servers inside the network data center. In most scenarios, north-south traffic is also called Server-Client traffic. When drawing typical network diagrams, core network components are usually drawn at the top (North), clients are drawn at the bottom (South), and different servers in the network data center are drawn horizontally (East-West). Therefore, the southbound traffic in this embodiment refers to the traffic from the client to the server.

[0049] In some embodiments of the present application, Figure 1 The methods include:

[0050] The traffic to be scanned in the traffic message queue is filtered by static resource files, where the static resource files include, for example, CSS (Cascading Style Sheets), JS (Java-script) text, ZIP files, and the like.

[0051] Correspondingly, normalizing the traffic to be scanned in the traffic message queue specifically refers to normalizing the filtered traffic to be scanned.

[0052] In some embodiments of the present application, the above step S120 normalizes the traffic to be scanned in the traffic message queue, specifically including:

[0053] Performing normalization processing of different dimensions on the traffic to be scanned in the traffic message queue through a multi-dimensional normalization model to obtain a normalization processing result of the traffic to be scanned;

[0054] A hash calculation is performed on the normalization processing result of the traffic to be scanned to obtain a hash value corresponding to the traffic to be scanned and each normalization model.

[0055] Among them, the multi-dimensional normalization model includes a URL semantic rule model, a compound URL semantic rule model, a load configuration model, a business content rule model, a regular algorithm model, and a Bloom algorithm model.

[0056] In a high-traffic scenario, the volume of data to be scanned is generally high, for example, it may reach the level of tens of billions of entries. At this time, the common URL similarity comparison solution in the industry cannot take into account both high deduplication rate and high efficiency. To address this problem, this embodiment is based on six models, including URL semantics, load configuration, business content, URL with parameters, regular expression, and Bloom algorithm, to perform high-performance, high deduplication rate normalization processing on the scanned traffic, among which:

[0057] The URL semantic rule model automatically generates normalization rules for category data by integrating the category services within the target network. In some application scenarios, the algorithm model analyzes the semantics of the URL and generates normalization rules related to the URL semantics. For example, the path (Path) at the same position of the URL: "fuwuyuan" in "www.abc.com / fuwuyuan" will be marked as an occupational label, and "siji" in "www.abc.com / siji" will also be marked as an occupational label. After the two flows are normalized by the URL semantic rule model, the normalization results of the two flows are "www.abc.com / %zhiye%". Since the normalization results of the two flows are the same, the corresponding Hash values ​​are also the same. In the flow deduplication stage, the two flows can be determined to be duplicate flows based on the Hash, and one of the flows can be deleted.

[0058] The composite URL semantic rule model refers to the analysis of not only the URL of the traffic, but also other parameters such as the parameter name and parameter value of the traffic, to ensure that scenarios where different parameter values ​​trigger different code logics are not mistakenly removed by deduplication. For example, the value of each parameter is counted to ensure that the top 5 values ​​of each parameter are retained and not removed by deduplication.

[0059] The load configuration model uses the Nginx configuration extraction tool to extract the configurations of different routes configured in the Nginx Conf configuration file inside the target network pointing to the same system, and automatically generates normalization rules related to the load configuration.

[0060] The business content rule model uses code analysis tools to analyze the codes in all code repositories of the target network, extract the routing configurations in all codes, and automatically generate normalized rules related to the business content.

[0061] The regular algorithm model obtains historical data that should be deduplicated but has not been deduplicated, and designs regular expressions based on relevant rules for such historical data, thereby forming normalization rules related to the regular expressions.

[0062] After the normalization rules of the above five models normalize each flow, a corresponding Hash value will be generated. In the prior art, the code syntax is generally used directly to compare the similarity of the Hash values. In the application embodiment, in order to cope with large-volume and high-concurrency scenarios, the Bloom algorithm is used in the Hash value similarity comparison function for auxiliary judgment, which improves the efficiency of the similarity comparison work by an order of magnitude.

[0063] After the traffic to be scanned in the traffic message queue is normalized at different latitudes by the above-mentioned multi-dimensional normalization model of the embodiment of the present application, each traffic to be scanned will generate a corresponding Hash value, and each traffic to be scanned includes the original data content and the corresponding Hash value. In other words, the normalized traffic in the preset database includes the original data content of the traffic and the Hash value generated based on the multi-dimensional normalization model.

[0064] In some embodiments of the present application, the above step S130 performs traffic deduplication processing on the normalized traffic according to the normalized reference traffic in the preset database, specifically including:

[0065] Performing statistics on the normalized reference flow in the preset database, and obtaining historical thermal data according to the statistical results;

[0066] Cache the historical hot data in the memory of the vulnerability scanning server;

[0067] The normalized traffic is deduplicated according to the historical hot data in the memory.

[0068] After each traffic to be scanned is normalized by the above-mentioned multi-dimensional normalization model, its normalized traffic has six Hash values, and each Hash value corresponds to the normalization model of the above-mentioned six dimensions. The number of Hash values ​​corresponding to the normalization model of each dimension is not fixed, and can be 1 or more. For example, the number of Hash values ​​obtained based on the URL semantic rule model is not limited to 1, and is related to the specific rules of the URL semantic rule model.

[0069] In actual applications, when deduplication is performed on normalized traffic, if at least one hash value of the current normalized traffic is the same as the hash value of a historical hot data in the memory, the current normalized traffic is determined to be duplicate traffic, and the current normalized traffic and its original traffic should be removed. Of course, the specific traffic deduplication strategy can be formulated according to demand and is not limited to the duplicate traffic determination scheme based on the hash value comparison result in this example.

[0070] In some possible implementation schemes of this embodiment, the statistical result includes a usage rate statistical result of the normalized reference flow, and obtaining historical thermal data according to the statistical result includes:

[0071] The normalized reference flow rate whose utilization rate is greater than the set value is used as the historical thermal data.

[0072] It is understandable that in other possible implementation schemes of the present embodiment, the normalized reference flow in the deduplication database can also be statistically analyzed in combination with other indicators of the normalized reference flow (such as the time information of the last use, etc.) to obtain statistical results related to each indicator, and historical thermal data can be obtained based on the statistical results related to each indicator. For example, the normalized reference flow with a usage rate greater than a set value and the last use time within one week is used as the historical thermal data.

[0073] The vulnerability scanning server of the embodiment of the present application includes multiple traffic analysis servers, each of which is deployed with a stream computing engine, such as Figure 3 The Flink streaming computing engine shown in the figure consumes the traffic in the Kafka traffic message queue through the streaming computing engine to normalize the consumed traffic; in addition, the traffic analysis server also uses a certain synchronization strategy (such as an event-based trigger strategy and a time-based trigger strategy) to obtain the deduplication database (i.e. Figure 3 The historical hot data is synchronized in the Mysql database shown in the figure and the synchronized historical hot data is cached in the memory of the traffic analysis server, and the newly generated normalized traffic is compared with the historical hot data in the memory to deduplicate the newly generated normalized traffic.

[0074] The study found that when the newly generated normalized traffic is directly compared with the normalized reference traffic in the MySQL database to implement traffic deduplication, there are problems such as large data volume and high concurrency in the deduplication calculation.

[0075] In order to improve the deduplication efficiency, this embodiment uses a hot data caching method to count the normalized reference traffic read from the deduplication database, and caches the data with higher usage rate into the memory of each traffic analysis server of the vulnerability scanning server. The traffic analysis server can obtain the benchmark data required for traffic deduplication from its memory, thereby greatly reducing the query volume to the deduplication database and reducing the traffic deduplication delay.

[0076] In some applications, the embodiments of the present application preload the historical hot data into the memory of the vulnerability scanning server during the startup phase of the vulnerability scanning server to avoid deduplication delays caused by high concurrency and large number of queries to the deduplication database at startup.

[0077] In practical applications, this embodiment also manages the deduplication database based on certain strategies, such as regularly cleaning up the data in the deduplication database based on the usage of the normalized reference flow in the deduplication database, such as deleting the normalized reference flow that has not been used for a set time (half a month, a month, or other time lengths). Of course, in practical applications, those skilled in the art can also manage the deduplication database based on other strategies.

[0078] Compared with the existing scanning scheme, the Web vulnerability scanning method of the above embodiment of the present application uses network traffic analysis technology, big data streaming computing deduplication technology, and distributed scanning task management technology to complete the quasi-real-time Web vulnerability scanning task in a large traffic scenario at a low cost. The embodiment of the present application can ensure that the coverage of the scanned assets is not lower than or even much higher than that of the attacker, and the scanning cost is much lower than that of the traditional Web vulnerability scanner, which solves the dilemma that the traditional Web vulnerability scanner cannot highly cover the scanned assets and a large number of assets cannot be scanned at a low cost, and is easy to deploy and has high practicality.

[0079] The embodiment of the present application also provides a Web vulnerability scanning device 400, which is applied to a vulnerability scanning server, such as Figure 4 As shown, a schematic diagram of the structure of a Web vulnerability scanning device in an embodiment of the present application is provided, wherein the device 400 includes: a traffic acquisition unit 410, a normalization unit 420, a traffic deduplication unit 430 and a vulnerability scanning unit 440, wherein:

[0080] A traffic acquisition unit 410 is used to acquire the traffic to be scanned and write the traffic to be scanned into a traffic message queue of a distributed publish-subscribe message system, wherein the traffic to be scanned includes Http traffic and Https traffic for external access to a target network area;

[0081] A normalization unit 420, configured to perform normalization processing on the traffic to be scanned in the traffic message queue to obtain normalized traffic;

[0082] A traffic deduplication unit 430, configured to perform traffic deduplication processing on the normalized traffic according to the normalized reference traffic in the deduplication database, and write the deduplication traffic into the scanning message queue of the distributed publish-subscribe message system and the deduplication database respectively;

[0083] The vulnerability scanning unit 440 performs Web vulnerability scanning according to the deduplicated traffic in the scanning message queue.

[0084] In some embodiments of the present application, the traffic deduplication unit 430 is specifically used to perform statistics on the normalized reference traffic in the deduplication database, obtain historical hot data based on the statistical results; cache the historical hot data in the memory of the vulnerability scanning server; and perform traffic deduplication processing on the normalized traffic based on the historical hot data in the memory.

[0085] In some embodiments of the present application, the statistical results include usage statistics of the normalized reference flow, and the flow deduplication unit 430 is specifically used to use the normalized reference flow with a usage rate greater than a set value as the historical thermal data.

[0086] In some embodiments of the present application, the Web vulnerability scanning device 400 further includes a preloading unit, and the preloading unit is used to preload the historical hot data into the memory of the vulnerability scanning server during the startup phase of the vulnerability scanning server.

[0087] In some embodiments of the present application, the Web vulnerability scanning device 400 further includes a preprocessing unit, the preprocessing unit being configured to perform filtering processing of static resource files on the traffic to be scanned in the traffic message queue;

[0088] The normalization unit 420 is specifically configured to perform normalization processing on the filtered traffic to be scanned.

[0089] In some embodiments of the present application, the traffic acquisition unit 410 is specifically used to obtain the Http traffic for external access to the target network area by performing traffic mirroring on the ingress switch of the target network area; and to obtain the Https traffic for external access to the target network area through the Nginx proxy service and parse it to obtain the parsed Https traffic for external access to the target network area.

[0090] In some embodiments of the present application, the normalization unit 420 is specifically used to perform normalization processing on the traffic to be scanned in the traffic message queue at different dimensions through a multi-dimensional normalization model. The multi-dimensional normalization model includes a URL semantic rule model, a compound URL semantic rule model, a load configuration model, a business content rule model, a regular algorithm model, and a Bloom algorithm model.

[0091] It can be understood that the above-mentioned Web vulnerability scanning generation device can implement each step of the Web vulnerability scanning method provided in the above-mentioned embodiment, and the relevant explanations about the Web vulnerability scanning method are applicable to the Web vulnerability scanning device, which will not be repeated here.

[0092] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 5 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0093] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0094] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0095] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a Web vulnerability scanning device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0096] Obtaining the traffic to be scanned and writing the traffic to be scanned into the traffic message queue of the distributed publish-subscribe message system, wherein the traffic to be scanned includes Http traffic and Https traffic for external access to the target network area;

[0097] Normalizing the traffic to be scanned in the traffic message queue to obtain normalized traffic;

[0098] Performing traffic deduplication processing on the normalized traffic according to the normalized reference traffic in the deduplication database, and writing the deduplication traffic into the scanning message queue of the distributed publish-subscribe message system and the deduplication database respectively;

[0099] Web vulnerability scanning is performed according to the deduplicated traffic in the scanning message queue.

[0100] The above application Figure 1 The method performed by the Web vulnerability scanning device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in software form. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the fraud pattern recognition model generation method in combination with its hardware.

[0101] The electronic device may also perform Figure 1 A method for executing a Web vulnerability scanning device in a Figure 1 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0102] The present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by an electronic device including multiple application programs, enable the electronic device to execute Figure 1 The method performed by the Web vulnerability scanning device in the illustrated embodiment is specifically used to perform:

[0103] Obtaining the traffic to be scanned and writing the traffic to be scanned into the traffic message queue of the distributed publish-subscribe message system, wherein the traffic to be scanned includes Http traffic and Https traffic for external access to the target network area;

[0104] Normalizing the traffic to be scanned in the traffic message queue to obtain normalized traffic;

[0105] Performing traffic deduplication processing on the normalized traffic according to the normalized reference traffic in the deduplication database, and writing the deduplication traffic into the scanning message queue of the distributed publish-subscribe message system and the deduplication database respectively;

[0106] Web vulnerability scanning is performed according to the deduplicated traffic in the scanning message queue.

[0107] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0108] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a deduplication computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0109] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0110] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0111] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0112] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0113] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0114] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0115] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0116] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A Web vulnerability scanning method, characterized in that: The Web vulnerability scanning method is executed by a vulnerability scanning server, and the method includes: Obtaining the traffic to be scanned and writing the traffic to be scanned into the traffic message queue of the distributed publish-subscribe message system, wherein the traffic to be scanned includes Http traffic and Https traffic for external access to the target network area; Normalizing the traffic to be scanned in the traffic message queue to obtain normalized traffic; specifically, normalizing the traffic to be scanned in the traffic message queue in different dimensions through a multi-dimensional normalization model, wherein the multi-dimensional normalization model includes a URL semantic rule model, a composite URL semantic rule model, a load configuration model, a business content rule model, a regular algorithm model, and a Bloom algorithm model; Performing traffic deduplication processing on the normalized traffic according to the normalized reference traffic in the deduplication database, and writing the deduplication traffic into the scanning message queue of the distributed publish-subscribe message system and the deduplication database respectively; Performing Web vulnerability scanning according to the deduplicated traffic in the scanning message queue; The performing traffic deduplication processing on the normalized traffic according to the normalized reference traffic in the deduplication database includes: Performing statistics on the normalized reference traffic in the deduplication database, and obtaining historical hot data according to the statistical results; caching the historical hot data in the memory of the vulnerability scanning server; and performing traffic deduplication processing on the normalized traffic according to the historical hot data in the memory; The statistical result includes a usage rate statistical result of the normalized reference flow, and the acquiring of historical thermal data according to the statistical result includes: The normalized reference flow rate whose utilization rate is greater than the set value is used as the historical thermal data.

2. The Web vulnerability scanning method according to claim 1, characterized in that: The method further comprises: During the startup phase of the vulnerability scanning server, the historical thermal data is preloaded into the memory of the vulnerability scanning server.

3. The Web vulnerability scanning method according to claim 1, characterized in that: The method further comprises: Performing static resource file filtering processing on the traffic to be scanned in the traffic message queue; The normalizing the traffic to be scanned in the traffic message queue includes: The filtered traffic to be scanned is normalized.

4. The Web vulnerability scanning method according to claim 1, characterized in that: The obtaining of the traffic to be scanned includes: By performing traffic mirroring on the ingress switch of the target network area, the Http traffic of external access to the target network area is obtained; And obtain the Https traffic for external access to the target network area through the Nginx proxy service and parse it, and obtain the parsed Https traffic for external access to the target network area.

5. A Web vulnerability scanning device, characterized in that: The Web vulnerability scanning device is applied to a vulnerability scanning server, and the device comprises: A traffic acquisition unit, used for acquiring the traffic to be scanned and writing the traffic to be scanned into the traffic message queue of the distributed publish-subscribe message system, wherein the traffic to be scanned includes Http traffic and Https traffic for external access to the target network area; A normalization unit, used to normalize the traffic to be scanned in the traffic message queue to obtain normalized traffic; specifically, normalizing the traffic to be scanned in the traffic message queue in different dimensions through a multi-dimensional normalization model, wherein the multi-dimensional normalization model includes a URL semantic rule model, a composite URL semantic rule model, a load configuration model, a business content rule model, a regular algorithm model, and a Bloom algorithm model; A traffic deduplication unit is used to perform traffic deduplication processing on the normalized traffic according to the normalized reference traffic in the deduplication database, and write the deduplication traffic into the scanning message queue of the distributed publish-subscribe message system and the deduplication database respectively; specifically, statistics are performed on the normalized reference traffic in the deduplication database, and historical hot data is obtained according to the statistical results; the historical hot data is cached in the memory of the vulnerability scanning server; the normalized traffic is deduplicated according to the historical hot data in the memory; wherein the statistical results include the usage rate statistics of the normalized reference traffic, and the traffic deduplication unit uses the normalized reference traffic whose usage rate is greater than the set value as the historical hot data; A vulnerability scanning unit performs Web vulnerability scanning according to the deduplicated traffic in the scanning message queue.

6. An electronic device comprising: processor; as well as A memory arranged to store computer executable instructions, wherein when the executable instructions are executed, the processor executes the Web vulnerability scanning method according to any one of claims 1 to 4.

7. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of application programs, enables the electronic device to execute the Web vulnerability scanning method according to any one of claims 1 to 4.

Images